A text sentiment analysis method based on deep learning

Through the Self-Attention model and BERT-Bi-LSTM structure of deep learning, combined with the Attention-GRU network, the problems of user individual differences and insufficient accuracy of sentiment analysis in the existing technology are solved, and the high accuracy of personalized sentiment analysis is achieved Sex and stability are superior to traditional machine learning models.

CN114595693BActive Publication Date: 2024-12-03MARKETING SERVICE CENT OF STATE GRID LIAONING ELECTRIC POWER CO LTD +2
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202011428365.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-07
Publication Date
2024-12-03
Estimated Expiration
2040-12-07

AI Technical Summary

Technical Problem

Existing text sentiment analysis methods are difficult to accurately reflect the user's true emotional expression, especially ignoring the individual differences of users, resulting in low quality of analysis results and lack of personalized sentiment analysis tools. There are challenges in dynamically capturing users' personalized emotional preferences.

Method used

Using the Self-Attention model based on deep learning, combined with BERT and Bi-LSTM layers, text sentiment analysis is performed through the Attention-GRU network model, user characteristics and product function influence are introduced, and word importance is calculated using the Attention mechanism, and network structure is optimized to improve Emotional classification accuracy.

Benefits of technology

Achieve more accurate personalized sentiment analysis, improve the accuracy and recall of sentiment classification, reduce information redundancy and loss, and provide a general deep learning solution suitable for sentiment analysis of multiple data sets, and is excellent in traditional machine learning models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114595693B_ABST
    Figure CN114595693B_ABST
Patent Text Reader

Abstract

The present invention relates to a text sentiment analysis method based on deep learning, comprising: step one: preprocessing text sample data, and manually pre-marking sentiment rating levels; step two: constructing a Self-Attention deep learning model for online text sentiment analysis, and training the model with training set data; calculating a loss function each time, calculating the gradient of output layer neurons, and updating the network parameter value of each layer by forward and backward propagation, until a cutoff condition is reached, and then obtaining an optimized Self-Attention deep learning model and each network parameter; step three: collecting actual text corpus data, processing the data with the optimized Self-Attention deep learning model, and obtaining an online text sentiment analysis result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and text mining, and specifically relates to a text sentiment analysis method based on deep learning. Background Art

[0002] In online interactive platforms, text analysis has greatly changed people's communication and thinking patterns, and promoted the explosive growth of user-generated information. In recent years, a large amount of text generated by users has become one of the most representative data sources of big data. Mining and analyzing user-generated information has become an important part of social development research. As an emerging information processing technology for analyzing, processing, summarizing and reasoning subjective texts with emotions, sentiment analysis of social media texts has received widespread attention in academia and industry in recent years and has been widely used in many fields of the Internet. Even in life, it has a wide range of applications, such as in the user interaction field of service robots in power business halls. Traditional sentiment analysis research mainly focuses on analyzing text emotions, but ignores the individual differences of users in emotional expression, thereby affecting the quality of the analysis results. To solve these problems, the present invention aims to solve the problem of personalized sentiment analysis. Considering the wide application of BP neural network technology in social media processing, the present invention proposes a variety of BP neural network-based models to apply social media text personalized sentiment analysis methods to online product reviews.

[0003] Sentiment analysis (SA) is the process of analyzing, processing, summarizing and reasoning about subjective texts with emotional expressions (e.g., microblogs, online comments and online news, etc.). The history of sentiment analysis research is not long. It began to receive widespread attention around 2000 and developed rapidly, and then gradually became a hot topic in the field of natural language processing and text mining. Sentiment analysis also has many aliases and similar technologies, such as opinion mining, sentiment mining, subjectivity analysis, etc., all of which can be studied under sentiment analysis. For example, for movie reviews, it can identify and analyze users' evaluations of movies, or analyze product reviews of digital cameras, such as users' emotional trends on indicators such as "price", "size", and "zoom". At present, sentiment analysis has become an interdisciplinary comprehensive research field such as natural language processing, information retrieval, computational linguistics, machine learning and artificial intelligence. Existing text sentiment analysis algorithms are mainly about users' views and opinions on text. Due to the lack of interpretation of user characteristics of text, these algorithms are difficult to fully and accurately reflect the user's true emotional expression. In order to overcome the shortcomings of existing methods, the present invention introduces the influence of users and even product functions to propose a personalized text sentiment analysis method. Although the sentiment analysis of text has been widely studied and has achieved outstanding performance in many public evaluation tasks. However, there are few studies on truly usable text sentiment analysis tools, especially personalized text sentiment analysis tools, and they have been neglected by scholars and the industry to some extent. At the same time, how to dynamically capture the personalized emotional preferences of users also poses new challenges to the problem of sentiment analysis. The present invention first introduces the idea of ​​Attention Model. The Attention Model can calculate the importance of the current part, that is, the attention model can track the classification target, focus on and find the classification target of interest. At the same time, the LSTM deep neural network model based on the Attention model can effectively solve the problem of text information redundancy and long-term dependence problems such as information loss. In order to design a more lightweight network structure, the present invention improves on the basis of the Attention-LSTM network model to obtain the Attention-GRU network model to classify personalized emotions. Summary of the invention

[0004] In order to solve the above-mentioned deficiencies in the prior art, the technical solution adopted by the present invention is:

[0005] A text sentiment analysis method based on deep learning, comprising:

[0006] Step 1: Preprocess the text sample data and manually label the sentiment rating level in advance;

[0007] Step 2: Build a Self-Attention deep learning model for online text sentiment analysis and train the model with training set data; calculate the loss function each time, calculate the gradient of the output layer neurons and update the network parameter values ​​of each layer through forward and backward propagation until the cutoff condition is reached to obtain the optimized Self-Attention deep learning model and each network parameter;

[0008] Step 3: Collect actual text corpus data, process the data using the optimized Self-Attention deep learning model, and obtain online text sentiment analysis results.

[0009] In step 2, a Self-Attention deep learning model for online text sentiment analysis is constructed, and training the model with training set data includes:

[0010] It is modeled based on the BERT deep learning model under the Self-Attention deep learning model;

[0011] The word embedding of all words in the input sentence is used as the semantic representation of the sentence; for the input semantic representation, the linear operation of matrix multiplication and the nonlinear activation function are used to obtain the semantic representation of the hidden layer; the semantic representation of the input hidden layer is input, and the dimensionality reduction operation is used to obtain the semantic representation at the sentence level; the sentence representation is combined with the user representation and input into the classification layer to merge the role of user information at the sentence level; the classification layer maps the obtained vector into a two-dimensional sentiment space and uses the softmax method for sentiment classification;

[0012] Model input: training set D = {(x 1 ,...,x k )}, k = 1, ..., M, M is the number of training data;

[0013] Model output: The trained and optimized BERT deep learning model and network parameters;

[0014] Among them, (x 1 ,...,x k ) is the word vector, u k It is a sentiment rating level that is manually pre-labeled to represent user characteristics.

[0015] The training set D is obtained by performing word segmentation and stop word removal on sentences in the original online text sample data.

[0016] The sample data is divided into a training set and a validation set. The BERT deep learning model is trained with the data in the training set, and the data in the validation set is used for verification.

[0017] Including: the construction of the text sentiment analysis BERT deep learning model specifically includes:

[0018] 1) The Encoder-Decoder encoding-decoding module is used in the Attention model to perform the following processing:

[0019] In the Encoder encoding module, the input natural sequence or feature matrix sequence is machine encoded. The expression is as follows:

[0020] C=F(x 1 ,x 2 ,x 3 ,…,x n )

[0021] Among them, x 1 ,x 2 ,x 3 ,…,x n is the word vector obtained after word segmentation and stop word removal; F is the encoder encoding function, and C is the word vector form after encoding according to the encoding function;

[0022] In the Decoder decoding module, the natural sequence or feature matrix sequence encoded by the Encoder is decoded. The expression is as follows:

[0023] y i =G(C,y 1 ,y 2 ,y 3 ,…,y i-1 )

[0024] Among them, y 1 ,y 2 ,y 3 ,…,y i-1 ,y i is the decoded word vector, i is the number; G is the Decoder decoding function;

[0025] 2) Segment the sentiment text and obtain the word representation T by fine-tuning the BERT model as follows:

[0026] T i =BERT(y i )

[0027] By embedding the user vector, a new user vector representation is obtained:

[0028]

[0029] Among them, E u is the vector of the newly added user;

[0030] 3) Use the Bi-LSTM layer to obtain the new user vector N i The forward propagation result of And the back propagation results The calculations are as follows:

[0031]

[0032] Get the latest representation of the i-th word in the text:

[0033]

[0034] Attention obtains a high-level feature representation and predicts the category of the text at the classification layer, as follows:

[0035] c i =softmax(W w h i +b w )

[0036] in, is the final parameter of the classification network, b w ∈R l is the bias information of the classification result, c represents the probability distribution of the current i-th class;

[0037] The network is trained using the cross entropy loss function as follows:

[0038]

[0039] Among them, u k represents the true label of sentiment sample i, c i Predict the sample as category y i The network parameters are optimized using stochastic gradient descent (SGD).

[0040] The verification data is input into the BERT deep learning model to obtain the corresponding evaluation level results, and the sentiment rating is manually annotated in advance for comparison. If it meets the error range, the iteration is stopped to obtain the optimized BERT deep learning model and network parameters.

[0041] It also includes: automatically mapping and associating the text sentiment analysis results with the user's original actual text corpus data according to the sentiment rating level, marking the degree of preference, and then visually displaying it to intuitively display the user's degree of preference.

[0042] It uses a color gradient sequence of strong, dark, light and weak to color and annotate information for actual text corpus data of different users.

[0043] Beneficial effects and advantages of the present invention:

[0044] In the present invention, a method based on a deep neural network is used to process text sentiment analysis. The deep learning method is completely different from the traditional machine learning method described above. It is mainly based on a neural network method, which can autonomously perform feature representation learning of discrimination, does not require the design and training of text features and dictionaries in advance, and can deeply capture semantic information. The present invention uses this method for text sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The general framework of the Encoder-Decoder model;

[0046] Figure 2 Schematic diagram of the Self-Attention structure;

[0047] Figure 3 BERT model structure;

[0048] Figure 4 BERT-Attention-BiLSTM network structure diagram;

[0049] Figure 5 Bi-LSTM layer;

[0050] Figure 6 Classification layer;

[0051] Figure 7 Comparison and contrast results of Yelp2013 algorithm and traditional classification methods;

[0052] Figure 8 Comparison and contrast results of Yelp2014 algorithm and traditional classification methods;

[0053] Fig. 9 Comparison and contrast results of Yelp2013 algorithm with other deep learning models;

[0054] Fig.10 Comparison and contrast results of Yelp2013 algorithm with other deep learning models; DETAILED DESCRIPTION

[0055] As a model that simulates the characteristics of the human brain's attention to things of interest, the Attention Model can focus on the identification of areas of interest. Its core idea is to draw on the characteristics of the human brain that in a specific moment, a specific scene, and a specific area, it pays more attention to a certain part of a specific thing, while ignoring other minor parts or parts of no interest. This is a model for optimizing the allocation of human brain resources. The principle is to allocate more attention to some key parts and parts that people are interested in, and allocate less attention or even no attention to other uninteresting parts, so as to reasonably utilize the computing resources of the human brain and remove the influence of non-critical factors or interference factors on the human brain. The Attention Model was first applied in the field of computer vision for tasks such as image recognition, classification, and target detection, and achieved good results. Later, the attention model was used to deal with the problem of image-to-text conversion, that is, converting images into natural language descriptions that can be easily understood by humans. This method makes traditional models achieve better results in this regard.

[0056] The data sets used in the present invention are Yelp2013 and Yelp2014. The data sets include 4.7 million user reviews, more than 150,000 business information, 200,000 pictures, and 12 metropolitan areas. In addition, it also covers 1 million tips from 1.1 million users, more than 1.2 million business attributes (such as business hours, whether there is a parking lot, whether it can be booked and environment, etc.), and the total number of users who sign in at each business over time. The comments in the data set are divided into 5 levels, which are: "Eek, Methins not", "Meh, I have experienced better", "A-OK", "Yay! I am a fun", "Woohoo! As good as it gets!". As shown in Table 1, the more stars a user has for a business, the better the customer's evaluation.

[0057] Table 1 An example of the review content and ratings of a sandwich restaurant on Yelp

[0058]

[0059]

[0060] This invention mainly describes the Attention Model in the field of Natural Language Processing (NLP). In the field of NLP, the Attention Model is often used in combination with the Encoder-Decoder model. This invention explains the use principle and effect of the Attention Model through the use method and process of the Encoder-Decoder module in the Attention Model, and introduces the application method and experimental effect of the Attention Model in personalized emotion classification.

[0061] The Encoder-Decoder model is a classic encoding and decoding natural language processing model. It uses the idea that natural sequences or feature matrix sequences are labeled, and inputs a set of natural sequences or feature matrix sequences to transform them into another set of changed (encoded) sequences. The core idea of ​​the encoder-decoder model is to encode the input natural sequence (which can be a matrix sequence after word vectorization or a feature obtained through a deep neural network) through the encoder module to form a form that is easy for computers to calculate and process, and then obtain the encoded code natural sequence or feature matrix sequence. Then, the code natural sequence or feature matrix sequence encoded by the encoder is input into the Decoder module for interpretation (decoding), and finally the changed natural sequence or feature matrix sequence that is easy to identify and classify is output. The Encoder-Decoder model has strong versatility and ease of use, and can easily combine various traditional neural network models and deep neural network models. In the Encoder module, various encoding models can be used to encode natural sequences or feature matrix sequences. From it, we can see the general framework structure of the Encoder-Decoder model.

[0062] The input is generally a natural sequence or feature matrix sequence X = (x 1 ,x 2 ,x 3 ,…,x n ), output is the decoded natural sequence or feature matrix sequence Y = {y 1 ,y 2 ,y 3 ,…,y n Generally, the Encoder module performs specific machine encoding on the input natural sequence or feature matrix sequence. Here, the present invention uses C to represent the encoded code natural sequence or feature matrix sequence, and the expression is as follows:

[0063] C=F(x 1 ,x 2 ,x3 ,…,x n )

[0064] Among them, x 1 ,x 2 ,x 3 ,…,x n is the word vector obtained after word segmentation and stop word removal; F is the encoder encoding function, and C is the word vector form after encoding according to the encoding function;

[0065] In the Decoder module, the natural sequence or feature matrix sequence after the Encoder is decoded. For example, the computer output y i C and the previously generated y are needed 1 ,y 2 ,y 3 ,…,y i-1 , so y i The calculation formula is:

[0066] y i =G(C,y 1 ,y 2 ,y 3 ,…,y i-1 )

[0067] Among them, y 1 ,y 2 ,y 3 ,…,y i-1 ,y i is the decoded word vector, i is the number; G is the Decoder decoding function;

[0068] It can be seen that the output y is calculated in the Decoder decoding module i As a result, the semantic information used is the same, all generated by x 1 ,x 2 ,x 3 ,…,x n After the Encoder (encoder), the code natural sequence or feature matrix sequence is generated. That is to say, for the input sequence x 1 ,x 2 ,x 3 ,…,x nThe interference ability and influence ability of each element on the output sequence are the same. The size of these influences is determined by the order of elements in the input sequence and the correlation between each element and the target. In addition, for the input of longer natural sequences or feature matrix sequences, although some encoding and decoding models can effectively save the influence information generated by historical data, due to the dimensionality limitation of the semantic Encoder (encoding) vector in natural language, some information that is effective for the result is lost. From the above analysis, the present invention finds that for sequences encoded with the same encoding rules, the influence of the decoded data generated in the decoding stage on the output is the same. This encoding and decoding mechanism is very different from Renmao's attention allocation thinking mode. Therefore, in order to make the machine achieve the same or similar effect as the attention model of the human brain in the encoding attention method, an Attention model mechanism similar to the human brain attention method is proposed. The principle of this attention mechanism is to calculate the influence of the input natural sequence or feature matrix sequence on the current output sequence y in the decoder (decoding) stage. i When the attention probability distribution is calculated, a unique semantic encoding corresponding to the target of interest is calculated for each output. This encoding combines the attention probability distribution of the feature sequence encoded during input to the current output feature sequence, which can optimize the current output result.

[0069] The Attention-Model structure of the present invention adopts the most popular Self-Attention structure to perform attention processing on the data.

[0070] Self-Attention: Structure like Figure 2 As shown. The three matrices Q (Query), K (Key), and V (Value) all come from the same input. First, the present invention calculates the dot product between Q and K, and then to prevent the result from being too large, it is divided by the square root of a dimension, where is the dimension of a query and key vector. The result is then normalized to a probability distribution using the Softmax operation, and then multiplied by the matrix V to obtain a weighted sum representation. The attention weight of the output element is calculated as follows:

[0071]

[0072] Most existing models use Word2Vec pre-trained word vectors, but there is a problem when using the word vectors trained by these models to generate the word vectors required by the present invention. Because the word vectors obtained by this method are a kind of static coding, the same word is still the same expression in different contexts, resulting in deviations in the model's semantic understanding of different scenarios. To address this problem, the present invention selects the BERT pre-trained language model as a word vector generation model. The BERT processing structure is a new natural language representation model. The BERT model can better reflect the spatial relationship between the semantic information of the sentence in the entire text, that is, it reflects the different expression meanings and information of the same text information in different contexts. On the one hand, sentences in the same or similar contexts have similar expression meanings, and theoretically their distance in space is relatively close. On the other hand, the BERT model uses an operation method that is roughly the same as the human brain's understanding when processing vector operations between sentences. Its model structure is as follows Figure 3 shown.

[0073] The input of the BERT model is the sum of three vectors. For each input word, its representation consists of three parts, namely, the token embeddings, the segment embeddings, and the positions embeddings. The token embeddings represent the encoding of the current word, the segment embeddings represent the position encoding of the sentence where the current word is located, and the positions embeddings represent the position encoding of the current word. Each sentence uses CLS and SEP as the beginning and end markers.

[0074] The most important part of the BERT model is the bidirectional Transformer encoding layer, which extracts text features using the Transformer Encoder feature extractor. The Encoder consists of a self-attention mechanism and a feed-forward-neural-network. The core of the Encoder is self-attention, which can find the degree of association between each word and other words in the sentence, and there is no distance limit. The relationship between dozens or even hundreds of words can still be found, so that the left and right context information of each word can be fully mined, so that the bidirectional representation of the word can be obtained.

[0075] The model is mainly composed of three layers, namely the BERT structure layer, the BiLSTM structure layer and the Attention structure layer. The model structure is as follows Figure 4 As shown in the figure, the model consists of five layers: BERT layer, user vector embedding layer, Bi-LSTM layer, attention layer and classification layer.

[0076] BERT layer: First, the sentiment text is segmented and the word representation T is obtained by fine-tuning the BERT model. That is, given a text D = {x 1 ,x 2 ,x 3 ,…,x n}, x i Represents the words in a given text. The BERT word vector model can be used to obtain the word vector representation of each word:

[0077] T i =BERT(y i )

[0078] Where T i ∈R d , represents the word vector representation of each word, and d represents the dimension of the word vector.

[0079] User vector embedding layer: Each user is represented by a vector E u ∈R d , finally the present invention can obtain a new user vector representation:

[0080]

[0081] Bi-LSTM layer: Perform two operations on the obtained new user vector. Each vector N i The forward propagation result and the backward propagation result of . The forward propagation result can be calculated according to the following formula:

[0082]

[0083] The back propagation result can be obtained according to the formula:

[0084]

[0085] in, d h is the number of hidden neurons in each LSTM. So we get the latest representation of the i-th word in the text:

[0086]

[0087] So we can get the representation matrix H = [h 1 ;h 2 ;h 3 ;…;h n ],in The Bi-LSTM layer network structure is as follows Figure 5 shown.

[0088] Attention layer: In this section, the present invention proposes a new attention representation, that is, calculating the attention of the i-th word and all words, so as to find the emotion-related words, which is specifically expressed as follows:

[0089]

[0090] Among them, h i and h j Different words are represented by Bi-LSTM features. i The representation of is defined as:

[0091]

[0092] final Represents the high-level feature representation in sentiment sentences.

[0093] Classification layer: Based on the high-level feature representation obtained by Attention, the classification layer predicts the category of the text:

[0094] c i =softmax(W w h i +b w )

[0095] in, is the final parameter of the classification network, b w ∈R l is the bias information of the classification result, and c represents the probability distribution. The classification layer structure is as follows Figure 6 .

[0096] This section still chooses to use the cross entropy loss function to train the network:

[0097]

[0098] Among them, u k represents the true label of sentiment sample i, c i Predict the sample as category y i The probability of. In this section, the optimizer chooses to use stochastic gradient descent (SGD) to optimize the network parameters.

[0099] During model training, the present invention is trained and tested based on a word vector classification model and a sentence vector classification model. The word vector-based classification model is a method in which a document is divided into a series of words during training and the words are input into a deep neural network for training; the sentence vector-based classification model refers to a method in which word vectors are input into a deep neural network as a whole in the form of sentences for training. When calculating word vectors, special characters are screened, and special characters such as repeated marks and garbled characters are deleted. The sentence vector is the word vector spliced ​​into a fixed length and sent to the network for training. The hyperparameters used in the model training process are shown in Table 2.

[0100] Table 2 Hyperparameter settings

[0101]

[0102] The model training process is shown in Table 3:

[0103] Table 3 Algorithm flow

[0104]

[0105]

[0106] In order to evaluate the performance of the model proposed in the present invention, the following evaluation criteria are used for evaluation: Sentiment classification is generally divided into two-item classification and multi-item classification.

[0107] For binomial classification, if the sample has category C:

[0108] Table 4 Binomial classification result matrix

[0109]

[0110]

[0111] For multi-class classification, suppose there are categories C i ,i∈[1…N c ](N c is the number of categories):

[0112] Table 5 Multi-classification result matrix

[0113]

[0114] Where N ij Represents the number of samples of category i in class j.

[0115] Then, the evaluation indicators are:

[0116] (1) Accuracy

[0117] Accuracy is a common evaluation indicator for neural network models. Its definition is: for a given data set, the ratio of the number of samples correctly classified by the classifier to the total number of samples. The specific calculation formula can be used to determine whether a classifier is effective in a certain sense from the accuracy, but it is not always effective in evaluating the performance of a classifier. For a simple example, 90% of the samples are positive and 10% are negative, and the samples are seriously unbalanced. In this case, you only need to predict all samples as positive samples to get an accuracy of 90%, but in fact you cannot determine whether the classifier is effective. This shows that the high accuracy is highly unbalanced due to the problem of sample imbalance.

[0118] Accuracy:

[0119] (2) Accuracy

[0120] The probability of all samples predicted as correct that are actually positive is calculated as follows. Precision and accuracy seem similar, but they are two completely different concepts. Precision represents the degree of prediction of the positive sample results, while accuracy represents the overall prediction accuracy, including both positive and negative samples.

[0121] Precision:

[0122] (3) Recall rate

[0123] It is defined as the probability of being predicted as a positive sample among samples that are actually positive, and its calculation formula is shown below. Application scenario of recall rate: For example, taking users with negative reviews, compared with users with good reviews, in website comments, the present invention is more concerned with users with negative reviews. Because if too many users with negative reviews are classified as users with good reviews, this will affect the judgment of subsequent users. The higher the recall rate, the higher the probability that users with negative reviews are actually predicted.

[0124] Recall:

[0125] (4)F-score

[0126] Although there is no necessary correlation between Precision and Recall from the formula, in large-scale data sets, these two indicators are often mutually constrained. Ideally, it would be best to have both indicators high. But in general, the higher the precision, the lower the recall. Therefore, in practical applications, it is often necessary to make trade-offs based on specific circumstances. Therefore, in order to comprehensively balance these two indicators, a new indicator F-score is introduced. Its calculation formula is shown in the following formula, which is a harmonic value that comprehensively considers precision and recall.

[0127] F-score (F measure):

[0128] (5)RMSE

[0129] The root mean square error represents the degree of dispersion of the predicted value, also known as the standard error. The best fit is RMSE = 0. The root mean square error is also one of the comprehensive indicators of error analysis. Its calculation formula is as follows

[0130] RMSE (minimum error rate):

[0131] The experiments conducted in this invention are developed in Python 3.7 and in conjunction with the Pytorch 1.5 deep learning framework on the Jupyter Notebook platform. The datasets used are: "Yelp13" and "Yelp14". The datasets are divided into 80% of the dataset as training sets, 10% as validation sets to save the model, and 10% as test sets to calculate the effect of the model. This section compares the three models with the base model respectively. The performance evaluation criteria used in the experiment are accuracy, precision, recall, F-score, and RMSE.

[0132] Table 6 Algorithm comparison results on Yelp2013 dataset

[0133]

[0134] Table 7 Algorithm comparison results on Yelp2014 dataset

[0135]

[0136]

[0137] From Tables 6 and 7 and the Figure 7 and Figure 8It can be seen that Bert-Attention has the best results in all evaluation criteria. The Bert model has been successfully applied to many tasks in natural language processing. Using Bert to obtain the representation of words can capture the semantics of words well. In Table 6, for the F-score index, the Bert-Attention model is nearly 3% higher than the Attention-LSTM model. In Table 7, the Bert-Attention model is nearly 4% higher than the Attention-LSTM model. Attention-LSTM and Attention-GRU have similar performance on the two data sets, precisely because the two structures are similar and the ability to capture semantic features is similar. It can be seen from the table that these three models are superior to other basic models, which once again verifies that the deep learning designed by the present invention has good performance in personalized sentiment analysis, and can further improve the accuracy, recall rate and precision of sentiment classification. The present invention designs a deep neural network sentiment analysis model based on attention for user personalization. After the deep learning model adds the attention structure, it can effectively obtain the relevant feature information of the text at different positions, and can effectively extract similar semantic features. It can be seen from the bar chart that the attention-based deep learning model achieved a lower RMSE result, indicating that in personalized sentiment analysis, the deep learning model is more stable than the traditional machine learning model when performing sentiment analysis.

[0138] In order to test whether the attention mechanism of the model is effective, the present invention designed an ablation experiment and compared the experimental results of the model with and without attention, as shown in Tables 8 and 9.

[0139] Table 8 Algorithm comparison results of Yelp2013 dataset

[0140]

[0141] Table 9 Algorithm comparison results of Yelp2014 dataset

[0142]

[0143] From Table 8, Table 9 and the column Fig. 9 and Fig.10It can be seen that the structure with the attention model has higher F-score and accuracy than the structure without the attention model. From Table 8, it can be seen that the Attention-LSTM is 1.41% higher in accuracy than the LSTM model, the Attention-GRU model has a 1.67% higher F-score than the GRU model, and the Bert-Attention model has a 1.23% higher F-score than the Bert model. These illustrate the importance of the attention model in personalized sentiment analysis. The Attention model can well capture the important features of the sentence, more accurately locate the key information, filter out the unimportant information, and then extract the important personalized sentiment features, further improving the classification performance of the model.

[0144] From the attention mechanism and principle of the Attention model and the calculation method of the attention weight calculation formula, it can be seen that the Attention model can highlight the role of the key natural sequence of the input or the accumulation of the feature matrix for the analysis of the output feature coding sequence by calculating the probability distribution of attention, and has a good optimization effect on the traditional deep network model. At the same time, by analyzing the application of the Attention model in various fields, it can be seen that the idea of ​​the Attention model has a wide range of applications and has a good effect on tasks such as text classification and sentiment analysis in the current natural language processing field.

[0145] In traditional models, models rely heavily on sentiment dictionaries, and it is difficult to build accurate sentiment dictionaries. These shortcomings have led researchers to seek more convenient solutions. Analysis methods based on machine learning have emerged. Methods based on machine learning are very effective, but for different data, it is necessary to select appropriate classifiers and appropriate methods for extracting text features to obtain good analysis results. With the research and application of deep learning and neural networks, sentiment analysis based on deep learning and neural networks is a universal solution. The model based on the combination of deep learning and attention mechanism proposed in the present invention has strong generalization ability for different data sets, can be applied to sentiment analysis of multiple data sets, and does not need to select models like traditional machine learning models. The model proposed in the present invention takes into account the personalized differences between users and the potential personalized factors of users: language habits, user personality, opinion bias, etc. The model proposed in the present invention simply and effectively solves the user cold start problem. The most important thing is that the model proposed in the present invention is far superior to the traditional machine learning model in various evaluation indicators, and provides a better solution for solving the sentiment analysis problem.

Claims

1. A text sentiment analysis method based on deep learning, It is characterized in that include: Step 1: Preprocess the text sample data and manually label the sentiment rating level in advance; Step 2: Build a Self-Attention deep learning model for online text sentiment analysis and train the model with training set data; calculate the loss function each time, calculate the gradient of the output layer neurons and update the network parameter values ​​of each layer through forward and backward propagation until the cutoff condition is reached to obtain the optimized Self-Attention deep learning model and each network parameter; In step 2, a Self-Attention deep learning model for online text sentiment analysis is constructed, and training the model with training set data includes: Modeling is based on the BERT deep learning model under the Self-Attention deep learning model, including: 1) The Encoder-Decoder encoding-decoding module is used in the Attention model to perform the following processing: In the Encoder encoding module, the input natural sequence or feature matrix sequence is machine encoded. The expression is as follows: C=F(x 1 ,x 2 ,x 3 ,…,x n ) Among them, x 1 ,x 2 ,x 3 ,…,x n is the word vector obtained after word segmentation and stop word removal; F is the encoder encoding function, and C is the word vector form after encoding according to the encoding function; In the Decoder decoding module, the natural sequence or feature matrix sequence encoded by the Encoder is decoded. The expression is as follows: and i =G(C,y 1 ,and 2 ,and 3 ,…,and i-1 ) Among them, y 1 ,y 2 ,y 3 ,…,y i-1 ,y i is the decoded word vector, i is the number; G is the Decoder decoding function; 2) Segment the sentiment text and obtain the word representation T by fine-tuning the BERT model as follows: T i =BERT(y i ) By embedding the user vector, a new user vector representation is obtained: Among them, E u is the vector of the newly added user; 3) Use the Bi-LSTM layer to obtain the new user vector N i The forward propagation result of And the back propagation results The calculations are as follows: Get the latest representation of the i-th word in the text: The word embedding of all words in the input sentence is used as the semantic representation of the sentence; for the input semantic representation, the linear operation of matrix multiplication and the nonlinear activation function are used to obtain the semantic representation of the hidden layer; the semantic representation of the input hidden layer is input, and the dimensionality reduction operation is used to obtain the semantic representation at the sentence level; the sentence representation is combined with the user representation and input into the classification layer to merge the role of user information at the sentence level; the classification layer maps the obtained vector into a two-dimensional sentiment space and uses the softmax method for sentiment classification; Model input: training set D = {(x 1 ,...,x k )}, k = 1, ..., M, M is the number of training data; Model output: The trained and optimized BERT deep learning model and network parameters; Among them, (x 1 ,...,x k ) is the word vector, u k It is a sentiment rating level pre-labeled by humans to represent user characteristics; Attention obtains a high-level feature representation and predicts the category of the text at the classification layer, as follows: c i =softmax(W w h i +b w ) in, is the final parameter of the classification network, b w ∈R I is the bias information of the classification result, c represents the probability distribution of the current i-th class; The network is trained using the cross entropy loss function as follows: Among them, u k represents the true label of sentiment sample i, c i Predict the sample as category y i The probability of using stochastic gradient descent to optimize the network parameters; Step 3: Collect actual text corpus data, process the data using the optimized Self-Attention deep learning model, and obtain online text sentiment analysis results.

2. According to the text sentiment analysis method based on deep learning according to claim 1, It is characterized in that The training set D is obtained by performing word segmentation and stop word removal on sentences in the original online text sample data.

3. According to a text sentiment analysis method based on deep learning according to claim 1, It is characterized in that The sample data is divided into a training set and a validation set. The BERT deep learning model is trained with the data in the training set, and the data in the validation set is used for verification.

4. According to the text sentiment analysis method based on deep learning according to claim 1, It is characterized in that The verification data is input into the BERT deep learning model to obtain the corresponding evaluation level results, and the sentiment rating is manually annotated in advance for comparison. If it meets the error range, the iteration is stopped to obtain the optimized BERT deep learning model and network parameters.

5. A text sentiment analysis method based on deep learning according to any one of claims 1 to 4, It is characterized in that Also includes: The text sentiment analysis results are automatically mapped and associated with the user's original actual text corpus data according to the sentiment rating level, and the degree of preference is marked, and then visualized to intuitively show the user's preference level.

6. A text sentiment analysis method based on deep learning according to claim 5, It is characterized in that It uses a color gradient sequence of strong, dark, light and weak to color and annotate information for actual text corpus data of different users.

Citation Information

Cited By

  • Business leader characteristic and management belief analysis method based on long text

    CN120471505A