A text sentiment analysis method based on attention mechanism, multi-channel convolution and gate recurrent unit

Abstract

The application discloses a text sentiment analysis method based on a multi-channel convolution and a gated recurrent unit of an attention mechanism, and comprises the following steps: step 1, preprocessing obtained text data, and taking the preprocessed data as the input of a model; step 2, using sliding windows of different sizes to extract shallow text information, and using a fusion Concatenate operation to combine text feature information; step 3, using a residual unit and a gated recurrent unit respectively to extract abstract features of the text, and using a connection operation to combine the extracted feature information; and step 4, introducing an attention mechanism, giving different weights to words in a sentence, and finally completing classification by using batch normalization processing and a full connection layer. The application uses one-dimensional convolution combined with a residual unit and a gated recurrent unit connection and an attention mechanism to complete text classification, and improves the text classification precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text classification, and in particular to a text sentiment analysis method based on multi-channel convolution and gated recurrent unit of attention mechanism. BACKGROUND

[0002] Sentiment analysis is the analysis of texts published by people to understand their emotions or opinions. Sentiment analysis has long been an active research field in the field of natural language processing, which can identify positive, negative or neutral sentiment tendencies in text. Sentiment analysis can be roughly divided into news review analysis, product review analysis, movie review analysis, etc. Among them, the analysis of news reviews can understand the public's attitude towards hot events and help the government to respond quickly to hot events; the review and analysis of products can understand the attitude of consumers towards products and help businesses to adjust products in time; the analysis of film reviews can understand the satisfaction of moviegoers towards movies. Therefore, sentiment analysis can not only identify emotions or attitudes from texts expressing opinions, but also be of great help in public opinion analysis, product recommendation, market prediction, etc.

[0003] With the rapid development of social media platforms such as Twitter and Facebook, more and more people like to express their opinions on the Internet. The number of comments on the Internet is increasing rapidly, and it is difficult to analyze quickly and manually using traditional methods. In the era of big data, it is of great social significance to mine the sentiment tendencies of network platforms through artificial intelligence technology.

[0004] Currently, text sentiment analysis is mainly based on sentiment dictionary, machine learning and deep learning sentiment analysis methods. Traditional sentiment analysis methods usually combine sentiment dictionary and machine learning, but require manual annotation and consume a lot of manpower and material resources. Deep learning methods have been widely used in text sentiment classification. Convolutional neural networks and recurrent neural networks have become a research hotspot in text classification. Deep learning methods greatly reduce the cost of manual work, avoid the tediousness of feature extraction, and can achieve good performance. However, the extraction of text features is usually relatively simple. Therefore, we try to use various methods to extract sentiment features to obtain better classification results. SUMMARY

[0005] The present application proposes a text sentiment analysis method based on multi-channel convolution and gated recurrent unit of attention mechanism to solve the problem of low text classification accuracy caused by traditional classification algorithms. One-dimensional convolution is combined with residual unit and gated recurrent unit connection and attention mechanism to complete text classification, which improves the text classification accuracy.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] A text sentiment analysis method based on attention mechanism, multi-channel convolution and gated recurrent unit, comprising the following steps:

[0008] Step 1: Preprocess the obtained text data, and use the preprocessed data as the input of the model; the purpose of preprocessing the text data is to clean the text, and then convert the text data into word vector data that can be trained;

[0009] Step 2: Use different size sliding windows to extract shallow text information respectively, and use the fusion Concatenate operation to combine the text feature information;

[0010] Step 3: Use residual units and gated recurrent units respectively to extract abstract features of the text, and use the connection operation to combine the extracted feature information;

[0011] Step 4: Introduce the attention mechanism, give different weights to the words in the sentence, and finally complete the classification using batch normalization processing and full connection layer;

[0012] Further, the preprocessing in step 1 specifically includes the following steps:

[0013] Step 1.1: Normalize the labeled sample of the input text, which is to clean the data to obtain the labeled text data, and delete @user, links, punctuation, numbers and special characters by using regular expressions. These are meaningless data and will not affect the analysis of the model later. In addition, when the length of the word is less than 3, for example: 'a', 'oh', 'hmm', …, these words have no influence on the overall semantic tendency of the sentence, so they are deleted;

[0014] Step 1.2: Use NLTK in the Python library to extract the stem of the word, which is a process of deleting the suffix from the word based on rules;

[0015] Step 1.3: Stop words do not affect the overall sentiment of the sentence, so stop words are deleted;

[0016] Step 1.4: Use the BERT model to train the corresponding word vector, which can dynamically convert words into word vectors with high-quality features.

[0017] Further, we use the NLTK package to obtain the English open source sentiwordnet sentiment dictionary, and give each word in the sentence a corresponding weight w i ; wherein the sentiment dictionary contains sentiment polarity values, and the sentiment polarity ranges from [-1, 1], wherein -1 and +1 represent extremely negative and extremely positive respectively; the sentiment weight formula is as follows:

[0018]

[0019] where w i represents a word, v i represents the word w i the weight in the sentiment dictionary, and WD represents the sentiment dictionary.

[0020] Further, in step 2, one-dimensional convolution filters with sizes of 2, 3, 4, 5, 6, and 7 are used to extract feature values, respectively. Because the features obtained by different sizes of convolution kernels have different sizes, the dimensions of the features are made the same by maximum pooling, and then the features are connected to obtain a new feature vector.

[0021] Further, in step 3, the specific execution order of the residual unit structure is BN→ReLU→Conv→BN→ReLU→Conv.

[0022] Further, the residual unit uses a feature learning method of 3×3 convolution to extract text information.

[0023] Further, in step 3, a gate recurrent unit (GRU) is added, which uses an update gate and a reset gate to determine what information can be passed to the output. The gradient problem caused by long-term dependence of the recurrent neural network can be alleviated.

[0024] Further, the gate recurrent unit in step 3 extracts text information in the following manner:

[0025] The update gate u t is responsible for controlling the amount of memory information that can be retained to the current time, and the calculation of the update gate is shown in formula (2):

[0026] u t =σ(W u ·[h t-1 ,x t ]) (2)

[0027] where x t represents the input data at time t, h t-1 represents the vector of the data at the previous time, W u represents the weight matrix of the update gate, and σ represents the sigmod activation function.

[0028] The reset gate r t is responsible for determining the data to be discarded, and the calculation of the reset gate is shown in formula (3):

[0029] r t =σ(Wr ·[h t-1 ,x t ]) (3)

[0030] wherein, x t represents the input data at time t, h t-1 represents the vector of data at the previous time, W r represents the weight matrix of the reset gate, and σ represents the sigmod activation function.

[0031] The information stored in the current unit The stored information is shown in formula (4):

[0032]

[0033] wherein, W represents the weight matrix, r t represents the reset gate, and tanh is the activation function.

[0034] The vector h t is calculated, which transmits the information stored in the current unit to the next unit, and this process is shown in formula (5):

[0035]

[0036] Further, the step 4 comprises:

[0037] Step 4.1: Calculate the similarity by the output of the fully connected layer and the learning parameters in the context to obtain the attention score s i , and the attention score s i is shown in formula (6):

[0038]

[0039] wherein, z i represents the output of the fully connected layer, z w represents the learning parameters in the context, and f is the cosine similarity function.

[0040] Step 4.2: Numerical conversion of the attention score by using the softmax function. The weight a i is shown in formula (7):

[0041]

[0042] Step 4.3: Weighted summation according to the weight coefficient. The weighted summation of the weight a i and the hidden layer output t i is performed, and x represents the feature vector of the output sentence after the weighted summation:

[0043] x = ∑ ia i ·t i (8)

[0044] Further, the method further comprises:

[0045] Based on the training set, the model parameters are trained by using a stochastic gradient descent algorithm;

[0046] Input text data, and complete classification by using the best trained model.

[0047] Compared with the prior art, the present application has the beneficial effects:

[0048] The present application realizes a text sentiment analysis model based on multi-channel convolution and gated recurrent unit of attention mechanism for the obtained text data in the network, such as English comments. One-dimensional convolution is used in the network to extract shallow text features, and different sizes of sliding windows are used to extract shallow text information. Compared with the traditional one-dimensional convolution for extracting text, multiple different sliding windows can obtain more useful text information and achieve better results. Abstract features in the text are extracted by using residual units and gated recurrent units. The text information is extracted by using 3*3 convolution, and residual connection is used to fuse the text features to relieve overfitting in the network. Meanwhile, maximum pooling operation is used to reduce the dimension of the text. The attention mechanism is introduced to give different weights to the words in the sentence. Batch normalization and full connection layer are used to finally complete classification, thereby improving the classification accuracy of the text. The present application can fully extract text features and achieve better sentiment classification results. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A basic flowchart of a text sentiment analysis method based on multi-channel convolution and gated recurrent unit of attention mechanism according to an embodiment of the present application;

[0050] Figure 2 A structure diagram of a multi-channel convolution feature extraction module according to an embodiment of the present application;

[0051] Figure 3 A model structure diagram of a text sentiment analysis method based on multi-channel convolution and gated recurrent unit of attention mechanism according to an embodiment of the present application;

[0052] Figure 4 A columnar diagram of classification accuracy of different models on a data set. DETAILED DESCRIPTION

[0053] The present application will be further explained in combination with the drawings and specific embodiments:

[0054] As Figure 1As shown, a text sentiment analysis method based on attention mechanism multi-channel convolution and gated recurrent unit includes the following steps:

[0055] Step S1: Preprocess the obtained text data, and use the preprocessed data as the input of the model. The purpose of text data preprocessing is to clean the text, and then convert the text data into word vector data that can be trained.

[0056] Further, the preprocessing specifically includes the following steps:

[0057] The step S11: input the labeled sample of the text to perform normalization operation. Normalization is to clean the data to obtain the labeled text data, and regular expressions are used to delete @user, links, punctuation, numbers and special characters. These are meaningless data and will not affect the analysis of the model later. In addition, when the length of the word is less than 3, for example: 'a', 'oh', 'hmm', …, these words have no influence on the overall semantic tendency of the sentence, so they are deleted;

[0058] The step S12: use NLTK in the Python library to extract the stem of the word, which is a process of deleting the suffix from the word based on rules;

[0059] The step S13: stop words do not affect the overall sentiment of the sentence, so the stop words are deleted;

[0060] The step S14: use the BERT model to train the corresponding word vector. Through this method, the word can be dynamically converted into a word vector with high-quality features.

[0061] Step S2: Use sliding windows with sizes of 2, 3, 4, 5, 6, and 7 to extract shallow text information, respectively, and use the fusion Concatenate operation to merge the text feature information.

[0062] One-dimensional convolution with filter sizes of 2, 3, 4, 5, 6, and 7 is used to extract feature values, respectively. Since the feature sizes obtained by different size convolution kernels are different, the dimensions are made the same through max pooling, and then they are connected to obtain a new feature vector.

[0063] Step S3: Use residual units and gated recurrent units to extract abstract features of the text, respectively, and use the connection operation to merge the extracted feature information.

[0064] Further, the specific execution order of the residual unit structure is BN→ReLU→Conv→BN→ReLU→Conv.

[0065] Further, the residual unit extracts text information using a feature learning method of 3x3 convolution.

[0066] Specifically, the gated recurrent unit (GRU) determines what information can be passed to the output by using an update gate and a reset gate. The gradient problem caused by long-term dependence of the recurrent neural network can be alleviated.

[0067] Further, the gated recurrent unit extracts text information in the following manner:

[0068] The update gate u t is responsible for controlling the amount of memory information that can be retained to the current time, and the calculation of the update gate is shown in formula (2):

[0069] u t = σ(W u · [h t-1 , x t ]) (2)

[0070] where x t represents the input data at time t, h t-1 represents the vector of data at the previous time, W u represents the weight matrix of the update gate, and σ represents the sigmod activation function.

[0071] The reset gate r t is responsible for determining the data to be discarded, and the calculation of the reset gate is shown in formula (3):

[0072] r t = σ(W r · [h t-1 , x t ]) (3)

[0073] where x t represents the input data at time t, h t-1 represents the vector of data at the previous time, W r represents the weight matrix of the reset gate, and σ represents the sigmod activation function.

[0074] The information stored in the current unit The stored information is shown in formula (4):

[0075]

[0076] where W represents the weight matrix, r t represents the reset gate, and tanh is the activation function.

[0077] The calculation vector h tThe vector transmits the information stored in the current cell to the next cell, and the process is shown in formula (5):

[0078]

[0079] Step S4: Introduce attention mechanism, give different weights to the words in the sentence, and finally complete classification using batch normalization processing and full connection layer.

[0080] Further, step S4 includes:

[0081] Step S41: Calculate the similarity by the output of the full connection layer and the learning parameters in the context, and get the attention score s i , the attention score s i As shown in formula (6):

[0082]

[0083] Where z i represents the output of the full connection layer, z w represents the learning parameters in the context, and f is the cosine similarity function.

[0084] Step S42: Numerical conversion of the attention score by using the softmax function. The weight a i As shown in formula (7):

[0085]

[0086] Step S43: Weighted summation according to the weight coefficient. By weighted summation of the weight a i and the hidden layer output t i , x represents the feature vector of the output sentence after weighted summation.

[0087] x=∑ i a i ·t i (8)

[0088] Further, after step S4, it also includes:

[0089] Based on the training set, the model parameters are trained by using the stochastic gradient descent algorithm;

[0090] Define the model with the minimum loss rate on the validation set as the best model, and save it;

[0091] Input text data and use the trained best model to complete classification.

[0092] It is worth mentioning that the present application is mainly applied to the field of text classification in the future. The present application can realize real-time and efficient classification work for English text in social networks. As a specific implementable manner, English text in Twitter is selected as a data set to verify the effectiveness of the model, and the specific process is as follows:

[0093] Step A: Obtain part of the text data with emotional color in Twitter, and divide it into an 8:2 ratio of training set and validation set.

[0094] Firstly, the obtained original data is normalized, the stem is extracted, and the stop word is removed, and then the text length is supplemented with zero value, and the text length is too long to be cut off, and finally the text is converted into a corresponding word vector to obtain a 30*768 vector matrix.

[0095] Step B: The vector matrix is processed by multi-channel one-dimensional convolution to extract the shallow features of the text, and the generated vector matrix is transmitted into the network.

[0096] Step C: The abstract features of the text are further extracted by the residual unit and the gated recurrent unit.

[0097] Step D: Introduce the attention mechanism into the text sentiment analysis model based on the multi-channel convolution and the gated recurrent unit of the attention mechanism for training, and further improve the performance of the model.

[0098] Step E: Finally, the classification results of multiple categories of the model input text are obtained.

[0099] The present application improves the classification accuracy of the text, extracts the shallow layer of the text by multi-channel one-dimensional convolution, further extracts the abstract features of the text by the residual unit and the gated recurrent unit, introduces the attention mechanism module, and improves the classification performance of the text. The present application is applied to text classification, compared with the traditional single text feature extraction method, the present application has the advantages of fully extracting text features and being more efficient in classifying text.

[0100] In order to prove the effect of the present application, the following experiments are carried out:

[0101] 1. Experimental environment

[0102] Hardware equipment: CPU is Intel(R) Core(TM) i7-7700HQ CPU@2.80GHz, memory size is 16GB, and GPU model is NVIDIA GeForce RTX1050;

[0103] Software platform: Python version is 3.6.2, CUDA version is 11.4, and the deep learning framework of PyTorch version 1.2.0 is used for model structure building.

[0104] 2. Experimental data set

[0105] In order to measure the final effect of the application, 6401 data on Twitter are selected as the data set of our experiment, of which 1364 data are positive, 4217 data are neutral, and 890 data are negative.

[0106] 3. Experimental setup

[0107] The Batch Size size is set to 32, the length of the word in the sentence is set to 30, the length of the sentence less than 30 is supplemented with 0, and the part exceeding 30 of the sentence length is cut off. The size of the dictionary is set to 80000, and each experiment is executed for 30 Epochs.

[0108] The random gradient descent algorithm (SGD) is used to optimize the training parameters, wherein the value of the momentum is set to 0.9, and the weight decay is set to 5e-4. The initial learning rate is set to 0.03, and as the training proceeds, if the learning rate is greater than 1e-5, the learning rate is correspondingly reduced every round to make it better to train the model. In random sampling, the proportion of the training set is 80%, and the proportion of the validation set is 20%. In order to evaluate the performance of the proposed method, accuracy (Accuracy) and Micro-F1 are used to evaluate the model, and the higher the value of the index indicates the better classification result.

[0109] 4. Comparison of different classification models

[0110] The current several better classification method models: CNN, LSTM, GRU, ATT-CNN, Multi-channel-CNN, DPCNN, and the model of the application (referred to as TADA) are compared in detail. Among these models, CNN and ATT-CNN are based on convolutional neural network method; LSTM, GRU is based on gated recurrent unit method; ATT-CNN is based on attention mechanism method; DPCNN and the application proposed is a residual unit method.

[0111] Among them, the final classification results of different models on the data set are shown in Table 1.

[0112] On the data set, only the convolutional neural network (CNN) shows lower accuracy, while other models based on long short-term memory network (LSTM) and gated recurrent unit (GRU) are superior to the accuracy of only using the convolutional neural network; in addition, the model based on attention mechanism combined with the convolutional neural network shows excellent accuracy, because CNN has the inherent feature extraction capability, and the attention mechanism can more effectively extract text information; in addition, the model of multi-channel convolutional neural network (Multi-channel-CNN) also shows relatively excellent accuracy, because multi-channel can more fully extract text feature information; finally, DPCNN shows relatively excellent accuracy by using the residual unit, which can more effectively extract the abstract features of the text. It can be seen that the classification accuracy of the model of the application is the highest, and compared with other models, the accuracy of TADA is improved by 4.22%. In addition, the accuracy of ATT-CNN, Multi-channel-CNN and DPCNN all reaches more than 79%, and these models show relatively excellent effect.

[0113] It is worth noting that TADA is superior to other models in Micro-F1 on the data set, and the balance of each class of the model is higher than that of other models, which shows that the designed model is more stable. Overall, the model of the application has obvious advantages in classification accuracy.

[0114] Table 1 Classification accuracy of different models on the data set

[0115]

[0116] Further, Figure 4 The classification accuracy of different models on the data set is shown in the following table.

[0117] In the above comparative experiment, a fixed number of samples are randomly selected and input into the model, which is also the way most researchers take. The single convolutional neural network has a good effect on shallow text feature extraction, but it ignores abstract text features and semantic relationships in the context. The residual unit has a good effect on deep text semantic information extraction, and the gated recurrent unit has a good effect on context semantic information extraction, but both have limited effect on shallow text feature extraction. Therefore, the application combines them to fully extract the features of the text. In addition, by introducing the attention mechanism, different weights are given to the words in the sentence, which more efficiently extracts the text features, so as to achieve better text classification effect.

[0118] Therefore, the method adopts a multi-channel one-dimensional convolution to construct a shallow feature extraction module, and a residual unit and a gated recurrent unit to construct an abstract feature extraction module, learns shallow features and abstract features of text data, additionally introduces an attention mechanism to optimize the learning of the model, thereby constructing a text sentiment analysis network model based on the multi-channel convolution and the gated recurrent unit of the attention mechanism, and the model only shows strong feature extraction capability;

[0119] In summary, the text classification framework proposed in the application has higher classification accuracy in text classification, and the model can fully and efficiently extract text features, thereby achieving higher classification accuracy in sample data.

[0120] The above only shows the preferred embodiments of the application, and it should be noted that for those skilled in the art, without departing from the principles of the application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the application.

Claims

1. A text sentiment analysis method based on attention mechanism multi-channel convolution and gated recurrent unit, characterized in that, The application relates to a method for classifying text data, and belongs to the technical field of artificial intelligence. Step 1: preprocessing the obtained text data, and taking the preprocessed data as the input of the model; Step 2: using different sizes of sliding windows to extract shallow text information, and using a fusion Concatenate operation to combine the text feature information; Step 3: respectively using a residual unit and a gated recurrent unit to extract abstract features of the text, and using a connection operation to combine the extracted feature information; Step 4: introducing an attention mechanism, giving different weights to the words in the sentence, and finally completing classification by using batch normalization processing and a full connection layer.

2. The text sentiment analysis method based on the attention mechanism, multi-channel convolution and gated recurrent unit according to claim 1, characterized in that, The step 1 comprises the following steps. Step 1.1: deleting @user, links, punctuation marks, numbers and special characters by using a regular expression, and deleting a word when the length of the word is less than 3; Step 1.2: extracting the stem of the word by using an NLTK in a Python library; Step 1.3: deleting stop words; Step 1.4: using a BERT model to train to obtain corresponding word vectors.

3. The text sentiment analysis method based on the attention mechanism multi-channel convolution and the gated recurrent unit according to claim 2, characterized in that, The step 1.2 comprises the following steps. An English open-source sentiwordnet sentiment dictionary is obtained by using an NLTK package, and each word in the sentence is given a corresponding sentiment weight; The sentiment weight formula is as follows: where w i represents a word, v i represents the word w i a weight in the sentiment dictionary, WD represents the sentiment dictionary.

4. The text sentiment analysis method based on the attention mechanism, multi-channel convolution and gated recurrent unit according to claim 1, characterized in that, The step 2 comprises the following steps. One-dimensional convolution with filter sizes of 2, 3, 4, 5, 6 and 7 is used to extract feature values.

5. The text sentiment analysis method based on the attention mechanism, multi-channel convolution and gated recurrent unit according to claim 1, characterized in that, In the step 3, the specific execution sequence of the residual unit structure is BN->ReLU->Conv->BN->ReLU->Conv.

6. The text sentiment analysis method based on the attention mechanism, multi-channel convolution and gated recurrent unit according to claim 1, characterized in that, The residual unit uses a 3*3 convolution feature learning method to extract text information.

7. The text sentiment analysis method based on the attention mechanism, multi-channel convolution and gated recurrent unit according to claim 1, characterized in that, The gated recurrent unit decides the information delivered to the output by using an update gate and a reset gate.

8. The text sentiment analysis method based on the attention mechanism multi-channel convolution and the gated recurrent unit according to claim 7, characterized in that, The gated recurrent unit extracts text information in the following manner: The update gate u t The data amount responsible for controlling the memory information capable of continuing to be retained to the current time is calculated as the update gate u shown in Equation (2). u t = σ(W u · [h t-1 , x t ]) (2) Wherein, x t represents the input data at time t, h t-1 represents the vector of the data at the previous time, W u represents the weight matrix of the update gate, and σ represents the sigmod activation function. Reset gate r t The reset gate r is responsible for determining which data needs to be discarded and is calculated as shown in equation (3): r t = σ(W r · [h t-1 , x t ]) (3) Wherein, x t represents the input data at time t, h t-1 represents the vector of the data at the previous time, W r represents the weight matrix of the reset gate, and σ represents the sigmod activation function. information stored in the current cell In this case, the stored information is given by equation (4): where W represents a weight matrix, r t denotes a reset gate, and tanh is an activation function. The vector h is computed t This vector transfers the information stored in the current cell to the next one, as shown by equation (5):

9. The text sentiment analysis method based on the attention mechanism, multi-channel convolution and gated recurrent unit according to claim 1, characterized in that, The step 4 comprises the following steps. Step 4.1: Compute the similarity by the output of the fully connected layer and the learned parameters of the context, and get the attention score s i , the attention score s i As shown in equation (6): where z i represents the output of the fully connected layer, z w represents the learning parameters in the context, and f is a cosine similarity function. Step 4.2: Numerical conversion of attention scores with softmax function, weight a i As shown in equation (7): Step 4.3: Weighted sum according to weight coefficients, by weighting the outputs of the hidden layers t i and the hidden layer outputs t i perform a weighted sum: x = ∑ i a i • t i (8) Wherein x represents the feature vector of the output sentence after weighted summation.

10. The text sentiment analysis method based on the attention mechanism, multi-channel convolution and gated recurrent unit according to claim 1, characterized in that, The application further comprises the following steps. Based on the training set, a random gradient descent algorithm is used to train the model parameters; Input text data, and use the best model after training to complete classification.

Images