A topic-enhanced text sentiment classification method based on VAE and attention

Abstract

The application relates to a topic-enhanced text sentiment classification method based on VAE and Attention, and belongs to the field of natural language processing, and comprises the following steps: S1, converting preprocessed text data into word vectors; S2, using a variational autoencoder VAE to reconstruct a BoW input to learn latent topics and keywords; S3, extracting sentence feature information through a CNN-Bi-LSTM; and S4, using an attention mechanism Attention to calculate the weight of a vector of a last time sequence output of a CNN-Bi-LSTM layer and a latent topic vector, and performing weighted summation to obtain a feature vector, and then performing softmax classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and relates to a topic-enhanced text sentiment classification method based on VAE and attention. BACKGROUND

[0002] In recent years, with the continuous development of Internet technology, more and more people express their feelings on the Internet through various platforms, and the number of network comments also increases explosively. More and more people are willing to express their attitudes and emotions on the network rather than simply browsing and accepting. A large amount of network comment data is often accompanied by the emotional information of the commenters, such as 'happy', 'angry', 'depressed', etc. The nodes in the social network are all human beings. Emotion is one of the important characteristics of human beings. The emotion of a user plays an important role in understanding individual activities. Individual emotion helps to explore social networks and their applications, such as recommendation and information dissemination. Individual emotion will deeply affect the behavior and decision of the user. Individual emotional information can help us understand how to affect the preference of the user and make accurate recommendations. The state of individual emotion will not only affect their behavior and decision, but also affect the fluctuation of the economy. Therefore, how to extract the emotion of the user and make prediction is very important. The existing sentiment analysis technology still has many deficiencies. Most of the work mainly classifies multiple emotions according to the text feature information by using supervised learning, and ignores the influence of the latent topic of the text on the emotion. The conventional topic generation method such as LDA cannot be directly integrated into the neural network in a joint manner. SUMMARY

[0003] Therefore, the purpose of the application is to provide a method for classifying text emotion by capturing topic information based on VAE (variational autoencoder) and attention mechanism, which can obtain the latent topic vector of the text through VAE, and combine the sentence structure information obtained by CNN-Bi-LSTM processing through the attention mechanism, fully utilize the latent topic information of the text, and improve the accuracy of emotion classification.

[0004] To achieve the above purpose, the application provides the following technical scheme:

[0005] A topic-enhanced text sentiment classification method based on VAE and attention, comprising the following steps:

[0006] S1: converting the preprocessed text data into a word vector;

[0007] S2: using a variational autoencoder VAE to reconstruct the BoW input to learn the latent topic and the keyword;

[0008] S3: extracting sentence feature information through CNN-Bi-LSTM;

[0009] S4: using attention mechanism Attention to calculate the weighted sum of the vector of the last time output of CNN-Bi-LSTM layer and the latent topic vector as the feature vector, and then performing softmax classification.

[0010] Further, the preprocessing in step S1 includes removing punctuation and stop words from the original text data, then performing word segmentation processing, and finally performing word2vec word embedding processing.

[0011] Further, the VAE in step S2 is composed of an encoder and a decoder, wherein:

[0012] The input of the encoder is the bag-of-words feature of the sentence, and the given sentence S={s1,....s L}L represents the length of the sentence, and each word vector is represented as a BoW feature x BoW ∈R V , and V represents the size of the vocabulary;

[0013] The decoder approximates the posterior distribution Z through variational inference, and reconstructs Z into

[0014]

[0015] where M∈R T×V is the kernel of f d (·);

[0016] M is normalized following each topic t=(0,....T):

[0017]

[0018] where M t represents the tth topic through the keyword distribution on the vocabulary;

[0019] Model learning: learn the parameters of the VAE by maximizing the variational lower bound of the feature marginal log-likelihood:

[0020]

[0021] where φ and θ are the parameters of the encoder and the decoder, respectively, and the KL-divergence term ensures that q φ (Z|x BoW ) is close to the prior distribution p(Z), and p(Z) is a standard normal distribution

[0022] Further, the encoder f e (.) is composed of multiple nonlinear hidden layers, which takes x BowTransforming into the prior parameters mu and sigma:

[0023]

[0024]

[0025] Define latent variable Z = mu + sigma * epsilon, epsilon is a Gaussian noise variable sampled from Z e R T , T is the topic number, normalized by the softmax function, Z reflects the distribution of the latent topic T.

[0026] Further, the encoder adopts KL cost annealing and word breaking to weaken the autoregressive decoder and anneal the KL divergence term.

[0027] Further, the CNN-Bi-LSTM in step S3 comprises a CNN layer and a Bi-LSTM layer, wherein:

[0028] The processing steps of the CNN layer are: extracting features with different sizes of convolution kernels, then performing maximum pooling, and finally splicing the features extracted by different sizes of convolution kernels together as the feature vector input into the Bi-LSTM;

[0029] The Bi-LSTM layer is used to obtain bidirectional feature information of the sentence.

[0030] Further, the processing steps of the CNN layer specifically include:

[0031] Let x i e R k be the k-dimensional word vector corresponding to the i-th word in the sentence, R k is the matrix representation of the sentence, and the length of the sentence is n. The sentence is an n*k matrix, which represents the n words of a sentence, and each word is a k-dimensional vector. Assuming that the number of words in the longest sentence in the document is n, then the sentence less than n is filled with m; The final word vector matrix is n*k, which is represented as:

[0032]

[0033] Where is a concatenation operator that makes x i：n a concatenation of words x i ;

[0034] The convolution operation includes a filter Represents the convolution kernel, which is used for the window of word h to generate a new feature, and the feature c i is generated by the window of word x i：i+h-1 :

[0035] c i = f(w · x i：i+h-1 +b)

[0036] h is the window, that is, the height of the filter, b is a bias term, w is a weight, and f is a nonlinear function; the filter is applied to the sentence {X 1：h , X 2：h+1 , …, X n-h+1：n} to generate a feature map:

[0037] c = [c1, c2, …, c n-h+1 ]

[0038] Then a max-over-time pooling operation is applied on the feature map, and the maximum value is taken as the corresponding feature special filter;

[0039] A plurality of filters with different window sizes are used to obtain a plurality of features, and the features are formed into a penultimate layer and transmitted to a fully connected Bi-LSTM layer;

[0040] Dropout is used in the penultimate layer, and three kinds of convolution kernels are used to correspond to 3-gram, 4-gram and 5-gram.

[0041] Further, the Bi-LSTM layer includes a forget gate, an input gate, and an output gate;

[0042] Forget gate: controls whether to forget the hidden cell state of the previous layer with a certain probability, represented as:

[0043] f t = σ(W f · [h t-1 , x t ]+b f )

[0044] where h t-1 represents historical information, x t represents new information in the current inflow cell, W f represents a weight, and b f represents a bias.

[0045] Input gate: responsible for processing the input of the current sequence position, determines what kind of new information is stored in the cell state, including a sigmoid layer for deciding what value to update; a tanh layer for creating a new candidate value vector (current cell state at the current time) and adding it to the state, the substructure is as follows:

[0046] i t = σ(W i · [ht-1 , x t ]+b i )

[0047]

[0048] where i t denotes the information to be updated, σ denotes a sigmoid activation function, W i denotes an update information weight, b i denotes an update information bias, W C denotes a current cell weight, and b c denotes a current cell bias.

[0049] The current cell state, C t-1 , is updated to C t :

[0050]

[0051] The output gate has the following substructure:

[0052] o t = σ(W o [h t-1 , x t ]+b o )

[0053] h t = o t *tanh(C t )

[0054] The functions corresponding to the input gate, the forget gate, and the output gate are all sigmoid functions, and the output result is [0, 1]. When it is 0, the gate is completely closed, and when it is 1, the gate is completely open.

[0055] Further, step S4 specifically includes the following steps:

[0056] S41: First, the last time sequence vector output by the CNN-Bi-LSTM layer and the latent topic vector extracted in the VAE are taken as input vectors of the Attention layer, and the query vector, the key vector, and the value vector are obtained by multiplying the word vector with three matrices W Q , w K , and W V , respectively;

[0057] S42: Calculate the attention score, calculate the similarity between Q and K, represented by f(Q, K i ), and divide each score by where d k is the dimension of the Key vector:

[0058] f(Q, K i ) = Q T K i

[0059] S43: the scores are subjected to a Softmax operation, the Softmax normalizes the scores so that they are all positive numbers and the sum is 1, and the weight a is calculated i :

[0060]

[0061] S44: for the calculated weight, all values in V are weighted and summed to obtain an Attention vector, and then a classification result is output through a fully connected layer:

[0062]

[0063] wherein a i represents the calculated sum, V i represents the i-th vector of the W v matrix.

[0064] The beneficial effects of the present application are that the latent topics and keywords are learned by the VAE, the topic information of the text is fully utilized, the influence of the topic on the emotion is added to the emotion classification model by combining the structural information of the text through the attention mechanism, and the accuracy of the emotion classification is improved.

[0065] Other advantages, objects, and features of the present application will be in part apparent and in part pointed out hereinafter in the specification, and will be learned from practice of the present application. The objects and other advantages of the present application can be realized and attained by the specification below. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:

[0067] Figure 1 for an embodiment of the present application, a flow chart of a topic-enhanced text emotion classification method based on VAE and attention is shown;

[0068] Figure 2 for an embodiment of the present application, a flow chart of a text data preprocessing process in step S1 is shown;

[0069] Figure 3 for an embodiment of the present application, a VAE model structure diagram in step S2 is shown;

[0070] Figure 4 Figure 3 is a model structure diagram of the CNN-Bi-LSTM model in step S3 of the embodiment of the present application;

[0071] Figure 5 Figure 4 is a model structure diagram of the Attention layer in step S4 of the embodiment of the present application;

[0072] Figure 6 Figure 5 is a distribution diagram of the public English dataset SemEval 18;

[0073] Figure 7 Figure 6 is a distribution diagram of the microblog Chinese dataset;

[0074] Figure 8 Figure 7 is a diagram of the hyperparameter setting in the experiment;

[0075] Figure 9 Figure 8 is a comparison diagram of the experimental results of the present method and the traditional sentiment classification model on the public English dataset SemEval 18;

[0076] Figure 10 Figure 9 is a comparison diagram of the experimental results of the present method and the traditional sentiment classification model on the microblog Chinese dataset. DETAILED DESCRIPTION

[0077] The present application is described in detail below by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0078] The accompanying drawings are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0079] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0080] Figure 1 A flow chart of a subject-enhanced text sentiment classification method based on VAE and Attention according to an embodiment of the present application is shown in Figure 1 , which comprises:

[0081] S1, converting the preprocessed text data into word vectors;

[0082] S2, using VAE to reconstruct the BoW input to learn latent topics and keywords;

[0083] S3, extracting sentence feature information through CNN-Bi-LSTM;

[0084] S4, calculating the weight of the vector of the last time output of the CNN-Bi-LSTM layer and the latent topic vector through Attention, and performing weighted summation as a feature vector, and then performing softmax classification;

[0085] Figure 2 A flow chart of data preprocessing is shown, which comprises removing punctuation, removing stop words, tokenizing, and performing word2vec word embedding processing.

[0086] The VAE model according to the present application is shown in Figure 2 , which comprises an encoder and a decoder.

[0087] Encoder: The input of the encoder is the bag-of-words feature of the sentence. Given a sentence S={s1,....s L}(L represents the length of the sentence), each word vector is represented as a BoW feature X BoW ∈R V ,(V represents the size of the vocabulary).

[0088] In particular, the encoder f e (.) is composed of multiple nonlinear hidden layers, which converts x Bow into prior parameters mu and sigma:

[0089]

[0090]

[0091] Define the latent variable Z = μ + σ · ∈, ∈ is a Gaussian noise variable sampled from Z ∈ R T (T is the topic number) is normalized by the softmax function, which reflects the distribution of latent topics T.

[0092] Decoder: We approximate the posterior distribution Z by variational inference, a feed-forward neural network is used to reconstruct Z into

[0093]

[0094] Here M ∈ R T×V is the kernel of f d (·) that enters the attention layer together with the embedding of the rich topic keyword. M is normalized following each topic t = (0, …, T) :

[0095]

[0096] Where M t is the representation of the t-th topic by the keyword distribution over the vocabulary.

[0097] Model learning: The parameters of the VAE are learned by maximizing the variational lower bound of the log marginal likelihood of the features:

[0098]

[0099] Where φ and θ are the parameters of the encoder and decoder, respectively, and the KL-divergence term ensures that q φ (Z|x BoW ) approaches the prior distribution p(Z), which is a standard normal distribution

[0100] Since the training objective of the decoder is to reconstruct the input, the source features can be directly accessed. When the decoder is trained, it is assumed that q(Z|x BoW ) = q(Z) = p(Z), which means that the KL loss is 0, which makes the latent variable Z unable to capture useful information, the reason behind this is that the decoder is too powerful, in order to solve this problem, we use KL cost annealing and word breaking for the encoder. For example, randomly replace some input words (e.g. 5%) with "UNK" tokens to weaken the autoregressive decoder and anneal the KL divergence term.

[0101] The CNN-Bi-LSTM model described in the present application is as shown in Figure 4 specifically as follows:

[0102] CNN layer: Let x i ∈R k be the k-dimensional word vector corresponding to the i-th word in the sentence. R k is the matrix representation of the sentence. A sentence of length n (padded if necessary) is represented as a n*k matrix, where the left side represents the n words of the sentence, each word being a k-dimensional vector. If the number of words in the longest sentence in the document is n, then sentences with less than n words are padded to m. The resulting word vector matrix is n*k. It is represented as:

[0103]

[0104] where is the concatenation operator. Let Xi:n be the concatenation of the words xi: (from the i-th word to the n-th word, concatenate them), the convolution operation contains a filter which is applied to a window of words h to produce a new feature. For example, the feature c i is generated by the window of words x i：i+h-1 :

[0105] c i = f(w·x i：i+h-1 +b) (6)

[0106] h is the window, that is, the height of the filter, here b e R is a bias term, w is the weight, and f is a nonlinear function. This filter is applied to the sentence {X 1：h , X 2：h+1 , …, X n-h+1：n} to generate a feature map:

[0107] c = [c1, c2,..., c n-h+1 ] (7)

[0108] Then apply the max-over-time pooling operation on the feature map and take the maximum value as the feature corresponding to this special filter. The idea is to capture the most important feature for each feature map, that is, the feature with the highest value. This pooling scheme naturally handles variable sentence length. This is the process of extracting one feature from one filter. The model uses multiple filters (with different window sizes) to obtain multiple features. These features form the penultimate layer and are passed to the fully connected Bi-LSTM layer.

[0109] The penultimate layer uses dropout, (to solve the problem of overfitting, which occurs when the model parameters are too many and the training samples are too few.)

[0110] Using three feature size of convolution kernel can be considered to correspond to 3-gram, 4-gram and 5-gram. The overall model structure is as follows, first using different size (3, 4, 5) of convolution kernel to extract features, and then maximum pooling, finally the features extracted by different size of convolution kernel are spliced together as the feature vector input into Bi-LSTM (bidirectional long short-term memory network).

[0111] Where LSTM contains 3 modules, which are forget gate, input gate and output gate.

[0112] The forget gate controls whether to forget the hidden cell state of the previous layer with a certain probability, which can be expressed as:

[0113] f t =σ(W f ·[h t-1 ,x t ]+b f ) (8)

[0114] Here h t-1 represents historical information, x t represents new information flowing into the cell. x t Here the role is to decide which historical information to forget according to the new information input.

[0115] The input gate (input gate) is responsible for processing the input of the current sequence position, and determines what new information is stored in the cell state. Here contains two parts. First, the sigmoid layer is called "input gate layer" to determine what value we will update. Then a tanh layer creates a new candidate value vector will be added to the state. Its substructure is as follows:

[0116] i t =σ(W i ·[h t-1 ,x t ]+b i ) (9)

[0117]

[0118] We want to add the gender of the new subject to the cell state to replace the old subject that needs to be forgotten. Therefore, when updating the new cell state, the two main things to do are to decide which historical information should flow into the current cell (forget gate control) and which new information should flow into the cell (input gate control).

[0119] After obtaining the input gate and forget gate coefficients, the current cell state C t-1 is updated to Ct .

[0120]

[0121] Output gate: after getting the new hidden cell state C t , we start to output the result, the substructure of output gate is as follows:

[0122] o t = sigma (W o [h t-1 , x t ]+b o ) (12)

[0123] h t = o t *tanh (C t ) (13)

[0124] The update of hidden state h t is composed of two parts, and the output gate is still determined by the historical information h t-1 and the new information x t . The functions corresponding to the input gate, the forget gate and the output gate are all sigmoid functions (because the output value range of the sigmoid function is 0-1, which is equivalent to controlling the percentage of the filter gate), so the output result is [0, 1], when it is 0, the gate is completely closed, when it is 1, the gate is completely opened. The input gate controls how much information of the current input value flows into the current calculation, the forget gate controls how much information of the historical information flows into the current calculation, and the output gate controls how much information of the output value flows into the hidden layer. We obtain the bidirectional feature information of the sentence by combining the outputs of the forward and backward LSTMs.

[0125] The Attention layer in the application is as shown in Figure 5 , the Attention calculates the weighted sum of the vector of the last time sequence output of the CNN-Bi-LSTM layer and the latent topic vector as a feature vector, and then performs softmax classification, as follows:

[0126] (1) first, the last time sequence vector output by the CNN-Bi-LSTM layer and the latent topic vector extracted in the VAE are taken as input vectors of the Attention layer, and three vectors are created for them, which are Query vector, key vector and value vector. They are obtained by multiplying the word vector and three matrices W Q , w K , W V , respectively, and the three matrices are obtained by training.

[0127] (2) Compute attention scores. Compute the similarity between Q and K, denoted by f. And divide each score by (dk is the dimension of Key vector). The purpose is to make the gradient more stable when backpropagating

[0128] f(Q, K i ) = Q T K i (14)

[0129] (3) Perform Softmax operation on these scores. Softmax normalizes the scores so that they are all positive and sum to 1:

[0130]

[0131] (4) For the calculated weights, perform weighted sum on all values in V to get the Attention vector, and then output the classification result through the fully connected layer.

[0132]

[0133] Comparative experiment:

[0134] Experiments were conducted on the public English dataset SemEval 18 and the Chinese microblog dataset. The dataset distributions are shown in Figure 6 and Figure 7 .

[0135] Experimental setup:

[0136] The tensorflow framework was used to implement our model, and the parameters were updated by stochastic gradient descent. Since multiple evaluation indicators were used for this task, the F macro was used as the first evaluation indicator during training. The experimental hyperparameters are shown in Figure 8 .

[0137] Evaluation indicators:

[0138] AP, F micro , F macro , and hamming loss were used as evaluation indicators.

[0139] AP represents the average precision, which represents the average proportion of relevant labels ranked higher than the given label. The formula is as follows:

[0140]

[0141] Hamming loss is used to see the sample misclassification based on a single label, for example, when the gold label does not appear in the expected label set or the wrong label exists. The smaller the value of this evaluation indicator, the better the performance of the system. The formula is as follows:

[0142]

[0143] Here, △ represents the symmetric difference between two sets.

[0144] F macro Usually used to mitigate the impact of the main category of quantity imbalance, the indicator assigns equal weights to the categories in the evaluation, which can be calculated by the following formula:

[0145]

[0146]

[0147] Here |c| indicates the number of categories, F i represents the F1 value of category i, P i and R i represent the accuracy and recall of category i.

[0148] In contrast, F micro (micro-average indicator) ignores the impact of data imbalance:

[0149]

[0150]

[0151]

[0152] P micro represents the micro-average accuracy, R micro represents the micro-average recall. TP i represents the number of samples whose actual label is category i and whose predicted label is also category i, FP i represents the number of samples whose actual label is category i, but whose predicted label is not category i. FN i represents the number of samples whose actual label is not category i, but whose predicted label is category i.

[0153] Experimental results analysis:

[0154] Figure 9 and Figure 10 are the experimental results of the overall model described in this method and the traditional sentiment classification model on the public English data set SemEval 18 and the micro-blog Chinese data set, respectively. It can be seen that in multiple indicators, the method is better than the traditional classification model, especially in AP and Fmacro Great progress has been made. The reason is that the model is more understandable to potential emotional information through topic enhancement, but the model performs better in the English dataset, which is due to the small amount of data in the Chinese dataset and the large difference in the number of "like" and other emotions, resulting in poor results. Among them, F micro The reason for the large change in hamming loss is that some sentences in the Chinese dataset are ambiguous, which affects the model's judgment of the topic. Future work will further solve the semantic disambiguation problem.

[0155] Finally, it should be explained that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A method for topic-enhanced text sentiment classification based on VAE and Attention, characterized in that: The method comprises the following steps: S1: converting the preprocessed text data into word vectors; S2: using a variational autoencoder VAE to learn latent topics and keywords by reconstructing the BoW input; S3: extracting sentence feature information through a CNN-Bi-LSTM; S4: calculating the weighted sum of the vector of the last time output of the CNN-Bi-LSTM layer and the latent topic vector as a feature vector by using an attention mechanism Attention, and then performing softmax classification; The VAE of step S2 is composed of an encoder and a decoder, wherein: The input of the encoder is the bag-of-words features of a sentence, given a sentence L represents the length of the sentence, and each word vector is represented as a BoW feature V represents the size of the vocabulary; The decoder approximates the posterior distribution Z by variational inference and reconstructs Z into : = wherein is the core of M normalizes each topic t = (0, ……, T): wherein, representing the t-th topic by a distribution of keywords over a vocabulary; Model learning: learn the parameters of the VAE by maximizing the variational lower bound of the feature marginal log-likelihood: where and are parameters of the encoder and decoder, respectively, and the KL-divergence term ensures proximity to the prior distribution p(Z), which is a standard normal distribution ; Encoder consisting of multiple non-linear hidden layers, transformed into a prior parameter and : Defining latent variables , is a Gaussian noise variable sampled from , , T is the topic number, normalized by a softmax function; Use KL cost annealing and word breaking for the encoder to weaken the autoregressive decoder and anneal the KL divergence term.

2. The VAE and Attention based topic augmented text sentiment classification method of claim 1, wherein: The preprocessing in step S1 includes removing punctuation and stop words from the original text data, performing word segmentation, and finally performing word2vec word embedding processing.

3. The VAE and Attention based topic augmented text sentiment classification method of claim 1, wherein: The CNN-Bi-LSTM of step S3 includes a CNN layer and a Bi-LSTM layer, wherein: The processing steps of the CNN layer are: extracting features with different size convolution kernels, then performing max pooling, and finally concatenating the features extracted by different size convolution kernels as the feature vector input into the Bi-LSTM; The Bi-LSTM layer is used to obtain bidirectional feature information of the sentence.

4. The VAE and Attention based topic augmented text sentiment classification method of claim 3, wherein: The processing steps of the CNN layer specifically include: Let x i ∈ R k be a k-dimensional word vector corresponding to the i-th word in a sentence, R k is the matrix representation of the sentence, a sentence of length n is an n*k matrix, representing the n words of a sentence, each word is a k-dimensional vector, assuming that the number of words in the longest sentence in the document is n, then the sentence less than n is filled with m; the final word vector matrix is n*k, represented as: where ⊕ is the concatenation operator, letting become the concatenation of the words ​ The convolution operation contains a filter w e , represents a convolution kernel, used on a window of words h to produce a new feature, feature generated by a window of words ​ h is the window, i.e. the height of the filter, b e R is a bias term, w is the weight, f is a non-linear function; the filter is applied to the sentence {X 1:h ,X 2:h+1 ,…,X n-h+1:n} to generate the feature map: Then apply a max-over-time pooling operation on the feature maps and take the maximum value =max{c} as the feature-specific filter corresponding to this. Use multiple filters of different window sizes to obtain multiple features, form the second-to-last layer with these features, and pass them to the fully connected Bi-LSTM layer; Use dropout in the second-to-last layer, and use three feature size convolution kernels corresponding to 3-gram, 4-gram and 5-gram.

5. The VAE and Attention based topic augmented text sentiment classification method of claim 4, wherein: The Bi-LSTM layer includes a forget gate, an input gate, and an output gate; Forget gate: control whether to forget the hidden cell state of the previous layer with a certain probability, represented as: wherein represents information to be forgotten, represents historical information, represents new information currently flowing into the cell, represents a weight, represents a bias amount; Input gate: responsible for processing the input at the current sequence position, determining what new information is stored in the cell state, including a sigmoid layer to decide what values to update, and a tanh layer to create a new candidate value vector and added to the state, denotes the cell state at the current time step, and the substructure is as follows: wherein represents information to be updated, represents a sigmoid activation function, represents an update information weight, represents an update information bias, represents a weight of the current cell, represents a bias amount of the current cell; update the current cell state, to : Output gate: the substructure of the output gate is as follows: The functions corresponding to the input gate, forget gate, and output gate are all sigmoid functions, and the output result is [0, 1]. When it is 0, the gate is completely closed, and when it is 1, the gate is completely open.

6. The VAE and Attention based topic augmented text sentiment classification method of claim 1, wherein: Step S4 specifically includes the following steps: S41: First, the last time vector output by the CNN-Bi-LSTM layer and the latent topic vector extracted in the VAE are taken as the input vectors of the Attention layer, and the word vector is multiplied by three matrices to obtain the Query vector, the key vector, and the value vector, respectively. S42: Compute attention scores, compute similarity of Q and K, denoted by and divide each score by where is the dimension of the Key vector: S43: perform Softmax operation on the scores, which normalizes the scores to positive numbers and sums to 1: S44: calculate the weighted sum of all values in V according to the calculated weights to obtain an Attention vector, and then output the classification result through a fully connected layer: wherein represents the calculated weight, represents the i-th vector of the matrix.

Images