A method for sentiment analysis of conversation text based on deep learning

Abstract

The present invention discloses a method for sentiment analysis of conversational text based on deep learning, comprising the following steps: S1, classifying and labeling a data set; S2, normalizing the divided data set; S3, extracting features from the text using a hierarchical GRU model; S4, initializing the training parameters of the GRU model; S5, training the GRU model; S6, inputting a prediction sentence to obtain a training result. The present invention proposes a hierarchical GRU model, wherein the bottom layer is a bidirectional GRU model that extracts single sentence features, and the upper layer bidirectional GRU models context information to obtain interaction features between sentences; an attention mechanism is added to the hidden layer of the bidirectional GRU, and its output is fused with a single word or utterance embedding to strengthen the information of each word or utterance in the context embedding. The present invention uses a pre-trained model to obtain single sentence text features, which can effectively solve the problem of small database size.

Description

Technical Field

[0001] The present invention relates to the field of text analysis, and in particular to a conversation text sentiment analysis method based on deep learning. Background Art

[0002] Sentiment analysis is the process of extracting and analyzing the emotions expressed in text. Its most direct goal is to determine emotional trends and predictions. With the increasing popularity of the internet, people are expressing their opinions in a growing variety of ways. Therefore, research on text-based sentiment analysis is becoming increasingly important. Currently, research on text sentiment primarily focuses on user reviews, such as Taobao reviews and website comments, while little research has been conducted on the mining of textual information in interactive conversations. Interactive text holds considerable value from the perspectives of data volume, user levels, and social influence. Therefore, sentiment analysis based on conversational text has both practical significance and potential value.

[0003] Traditional sentiment analysis focuses on analyzing the evaluative attitude expressed in text, typically using a binary classification approach: positive or negative, supportive or opposed. Commonly used text classification methods, such as support vector machines and convolutional neural networks, can be applied to sentiment classification in conversation-based text. However, traditional text classification methods cannot contextualize sentiment, resulting in poor sentiment detection performance.

[0004] Therefore, how to design a method that can analyze text sentiment based on context is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0005] The present invention provides a method for sentiment analysis of conversational text based on deep learning. A hierarchical GRU model is proposed. The bottom layer is a bidirectional GRU model to extract single sentence features, and the upper layer is a bidirectional GRU to model contextual information and obtain interactive features between sentences. An attention mechanism is added to the hidden layer of the bidirectional GRU, and its output is fused with the embedding of single words or utterances to strengthen the information of each word or utterance in the context embedding, so as to solve the above-mentioned problems.

[0006] The present invention provides the following technical solutions:

[0007] A deep learning-based conversation text sentiment analysis method is based on the hierarchical GRU sentiment analysis algorithm. Its specific operation steps are as follows:

[0008] S1. Label and classify the data set;

[0009] S2, normalize the divided data set;

[0010] S3. Use the GRU model to extract features from text;

[0011] S4. Initialize the training parameters of the GRU model;

[0012] S5. Train the GRU model;

[0013] S6. Input the prediction statement and obtain the training result.

[0014] Preferably, in said S1, the labels include anger, happiness, sadness and neutrality.

[0015] Preferably, the training set, validation set and test set all include input data and output data.

[0016] Preferably, in S2, the normalization process includes converting uppercase words into lowercase, restoring special parts of speech to their original forms, and removing non-alphanumeric symbols, retaining question marks and exclamation marks; and using a pad mechanism to fill in the length of the conversation so that the conversation length remains consistent.

[0017] Preferably, in S3, the feature extraction process includes extracting single utterance information, introducing an attention mechanism, and extracting context information.

[0018] More preferably, the single speech information extraction includes obtaining an open source word vector matrix to vectorize the input text and obtain a single word w k The embedded sequence Conversation Collection Where L is the number of conversations, In each dialogue D i N in i A sequence of utterances, u j For the speaker j ∈S said with a specific emotion c j ∈C, where s j represents a single speaker, S represents the set of speakers, c j represents a single utterance, C represents a collection of utterances, and w k Represents a single word; for the jth utterance Among them, M j It is discourse j The number of words in the word list is fed into the low-level bidirectional GRU model to learn the single utterance embedding from two opposite directions. The low-level hidden layer states in both directions are k∈{1,…,k}, k represents the number of words, and the formula is as follows:

[0019]

[0020]

[0021] The calculation process of one GRU is:

[0022]

[0023]

[0024]

[0025]

[0026] where r k ,z k are reset gate and update gate respectively; W,W r ,W z are the parameters of each neuron, which need to be learned during the training process; σ is the sigmoid function; is a candidate hidden state.

[0027] More preferably, the attention mechanism uses a masked mechanism to assign a minimum value to the position of the padding 0, so that the probability of the pad position in the softmax layer is 0 and does not enter the calculation.

[0028] More preferably, the context-dependent word embedding e c (w k ) is performed through the maximum pooling operation to obtain a single utterance embedding e(u j ); for i conversations The learned word embeddings It is fed into a high-level bidirectional GRU to capture contextual information and is fed into a high-level bidirectional GRU to capture the sequence and contextual relationships of the discourse in the conversation. The calculation is as follows:

[0029]

[0030]

[0031] Among them, the value of J is equal to the value of j, j∈{1,…,j} represents the number of utterances, and the sentence embedding e related to the context is obtained through the attention mechanism and tanh function. c (u j ); Finally, the softmax function is used to determine the sentiment label corresponding to the sentence.

[0032] Preferably, in S4, weighted cross entropy is used as the loss function to solve the problem of sample imbalance, and weights are assigned to different label classes, giving more attention to labels with fewer samples. The loss function loss is as follows:

[0033]

[0034]

[0035] where w(c j ) is the loss weight; I c is the loss weight and sentiment category c j The ratio of the number of training utterances in , and a constant α is used to adjust the smoothness of the distribution; Indicates the actual sentiment label.

[0036] Preferably, in S5, the data set information processed by S2 is put into the hierarchical GRU model in S3 to extract features, and the GRU model is trained according to the GRU model training parameters set in S4.

[0037] Preferably, in S6, the prediction statement is put into the model trained in S5 for prediction, and finally a training result is obtained.

[0038] The beneficial effects of the present invention are:

[0039] This paper uses a GRU-based conversational text sentiment analysis model to analyze the sentiment of individual sentences by separately modeling both single-sentence features and inter-sentence information. It also employs an attention mechanism to extract contextual information from long sequences. Evaluation of this sentiment analysis model and comparison with conventional conversational sentiment analysis models demonstrates its superior performance in conversational text sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0041] Figure 1 This is a flow chart of a method for analyzing conversational text sentiment based on deep learning according to an embodiment of the present invention;

[0042] Figure 2 : is a hierarchical structure diagram of the GRU model according to an embodiment of the present invention;

[0043] Figure 3 Schematic diagram of the self-attention mechanism of the forward hidden state in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following is a further detailed description of a conversation text sentiment analysis method based on deep learning in conjunction with specific embodiments. These embodiments are only for comparison and explanation purposes, and the present invention is not limited to these embodiments.

[0045] Example

[0046] See also Figure 1 The embodiment of the present invention provides a method for analyzing conversational text sentiment based on deep learning, which includes the following steps:

[0047] S1. Label and classify the data set;

[0048] In one embodiment, experiments were conducted using the Friends and EmotionPush datasets. The Friends dataset, derived from the television series "Friends," contains 1,000 conversations, each with multiple speakers. The EmotionPush dataset is a collection of conversations between friends collected on Facebook by an application called EmotionPush. The dataset contains 1,000 conversations, as shown in Table 1. Because the EmotionPush data contains potentially private information such as names and locations, this information has been processed in the dataset. For example, names are uniformly replaced with "person+number."

[0049] The Friends and EmotionPush datasets contain a large amount of data for neutral emotions, far more than for other emotions. As shown in Table 2, the data for negative emotions such as anger and sadness is far less than that for happiness and neutral emotions. In this paper, only the data corresponding to the four labels of anger, joy, sadness, and neutrality are retained, and the loss function for the remaining labels is set to zero to eliminate interference from other emotions.

[0050] Table 1. Training set and test set division of the dataset

[0051]

[0052] Table 2 Dataset examples

[0053] Dataset Anger Joy Sadness Neutral Others Friends 759 1710 498 6530 5066 EmotionPush 140 2100 514 9855 2133

[0054] S2, normalize the divided data set;

[0055] The S1 dataset was normalized. Words were converted from uppercase to lowercase, special parts of speech were restored to their original form, and non-alphanumeric symbols were removed. Since "?" and "!" can express strong emotions, such as surprise and excitement, these two characters were retained during the removal process. A padding mechanism was used to pad the length of conversations to ensure consistency.

[0056] A dictionary is built based on words and symbols. The word2vec model, which is publicly available on Google News and has 100 billion words, is pre-trained. The resulting word vector matrix is ​​used to vectorize the text. For words that do not appear in word2vec, a random uniform distribution strategy is used to assign values.

[0057] S3. Use the GRU model to extract features from text;

[0058] according to Figure 2 The GRU model hierarchy shown performs feature extraction on the text information processed by S1 and S2, and the extraction steps are shown in the following three processes.

[0059] S3.1. Single utterance information extraction: In one embodiment, the open-source Word2vec tool from Google is used to pre-train the model using the 300-dimensional Google News corpus. The open-source word vector matrix is ​​obtained to vectorize the input text and obtain the text embedding.

[0060] For D i The jth utterance in Among them, M j It is discourse j The number of words in the corresponding single word embedding sequence It is fed into the bidirectional GRU at the lower layer to learn a single utterance embedding from two opposite directions. The two-way extraction formulas are shown in Equation 1 and Equation 2:

[0061]

[0062]

[0063] S3.2. Add an attention mechanism. To fully consider the semantic and grammatical connections between different words, a self-attention mechanism layer is placed on the hidden state of the GRU. The attention output is fused with the individual utterance embedding and hidden state, strengthening the information of each word in the embedding. The self-attention mechanism has three matrices: query, key, and value. The input vector is multiplied by these three matrices to obtain a new vector. This vector is used to calculate the similarity between each word and other words. This new vector is used as a weighted value to use the information of surrounding words to strengthen the expression of the current word.

[0064] The self-attention layer is applied to the forward and backward GRU hidden states to produce the left context embedding respectively. and right context embedding It is able to learn the information of the current moment in two opposite directions. The output of the attention layer performs feature fusion and strengthens the information of each word in its sentence embedding. The tanh activation function on the linear transformation is used to generate word embedding related to the context:

[0065]

[0066] Among them, the two hidden states are connected as are model parameters, d0 is the word embedding dimension, and d1 is the hidden state of the low-level bidirectional GRU; Figure 3 This demonstrates the process of learning left-context embeddings on the forward hidden state of a GRU. A vector is multiplied by all vectors in the sequence, including itself, to obtain a new vector. The new vector represents both the original word and its relationship with other surrounding words. Each element in the attention matrix is ​​calculated as shown in Equation 4:

[0067]

[0068] where h k =h p , k and p are two different numbers taken in each calculation, and T is the transposition symbol. Since the length of text is generally uneven, it is necessary to truncate the long text sequence when embedding the text, and perform padding operation on the sequence of insufficient length so that the data can be aligned in dimension. However, padding does not contribute to the calculation of the entire network and should not be involved in the calculation of the attention mechanism. The present invention uses an attention mask to eliminate the internal attention between the sequence input and the padding. At each step, the corresponding left context is calculated by the weighted sum of all forward hidden states as shown in Formula 5:

[0069]

[0070] where α kp Is included in the left upper and lower embedded text The weight of the right context embedding can be obtained by a similar method.

[0071] The above is the attention mechanism layer in the low-level GRU. Since the high-level GRU is a vectorized representation of the sentence level, there is no padding, so there is no need to calculate the attention mask.

[0072] S3.3, context information extraction. The single utterance embedding is obtained by max pooling on the context-related word embeddings in the utterance, which helps to preserve information in different dimensions. For the i-th dialogue The learned single utterance embeddings It is fed into a high-level bidirectional GRU to capture the sequence and contextual relationships of the discourse in the conversation. The hidden states in the two directions are calculated as shown in Equation 6 and Equation 7:

[0073]

[0074]

[0075] Similar to the low-level GRU, it passes through a self-attention mechanism layer and fuses its output with the embedding of a single utterance to produce the left context embedding and right context embedding It can learn the current context information in two opposite directions. The calculation formula is shown in Equation 8:

[0076]

[0077] Among them, the two hidden states are connected as is the model parameter, and d2 is the hidden state of the high-level bidirectional GRU.

[0078] Since sentiment is identified at the utterance level, the learned context-dependent utterance embeddings e c (u j ) is directly sent to the fully connected layer FC, and then a softmax function is used to determine the corresponding emotion label The calculation formula is shown in Formula 9:

[0079]

[0080] in b fc ∈R |C| are model parameters; d2 is the hidden state of the high-level bidirectional GRU; C is a set of complex numbers, and |C| represents the modulus of the complex number.

[0081] S4. Initialize the training parameters of the GRU model;

[0082] In one embodiment, the problem of sample imbalance is solved by using weighted cross entropy as the loss function. Different classes are weighted and more attention is paid to classes with fewer samples. The loss function is shown in Equations 10 and 11:

[0083]

[0084]

[0085] In one embodiment, the model's embedding layer is initialized using publicly available 300-dimensional word2vec vector data, trained on 100 billion words from the Google News corpus. Convolutional kernels of size 3, 4, or 5 are used for the neural network layers, with 100 kernels each. The number of hidden states at both levels is set to 300, and the final fully connected layer has two sublayers, each with 100 neurons. To prevent overfitting, a dropout of 0.5 is used in the embedding and FC layers.

[0086] In one embodiment, the Adam optimization algorithm is used, with an initial learning rate set to 0.00025. The Adam algorithm differs from traditional stochastic gradient descent. While stochastic gradient descent does not change the learning rate during training, the Adam algorithm continuously adjusts the learning rate of each parameter during training.

[0087] S5. Train the GRU model;

[0088] The text information processed by S2 is placed into the hierarchical GRU model in S3 to extract features. The model is then trained according to the training parameters set in S4. An early stopping strategy is used during model training to effectively avoid overfitting. During training, the model is verified at each stage. After a period of time, if the results of the verification set do not improve, the training automatically terminates, thus avoiding overfitting issues caused by an excessive number of training sets. During training, the model is trained and verified on each training set. If the model's accuracy does not improve after more than 10 training runs, the model is terminated.

[0089] S6. Input the prediction statement and obtain the training result;

[0090] The prediction statement is put into the model trained in S5 for prediction, and the training results are finally obtained. The training results on the two test sets are shown in Table 3:

[0091] Table 3 Training results (%)

[0092]

[0093] The present invention uses a pre-trained model to obtain single sentence text features, which can better solve the problem of small database size.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for sentiment analysis of conversation text based on deep learning, characterized in that: The following steps are involved: S1. Label and classify the data set; S2, normalize the divided data set; S3. Use the GRU model to extract features from text; The feature extraction process includes extracting single utterance information, introducing attention mechanism and extracting context information; The single utterance information extraction includes obtaining an open source word vector matrix to vectorize the input text and obtain a single word The embedded sequence ;Dialogue Collection , where L is the number of conversations, In every conversation in A sequence of words, For the speaker Spoken with a specific emotion The words, in which represents a single speaker, S represents the set of speakers, represents a single utterance, C represents a collection of utterances, Represents a single word; For the jth utterance ,in It is words The number of words in the word list is fed into the low-level bidirectional GRU model to learn the single utterance embedding from two opposite directions. The low-level hidden layer states in both directions are , k∈{1,…,k}, k represents the number of words, and the formula is as follows: ； The calculation process of one GRU is: ； ； ； ； in , They are reset gate and update gate respectively; , , are the parameters of each neuron, which need to be learned during the training process; is the sigmoid function; is the candidate hidden state; The attention mechanism uses a mask mechanism to assign a minimum value to the position of the 0-filled position, and the hidden layer state generates a word embedding related to the context through a tanh activation function on a linear transformation. ; S4. Initialize the training parameters of the GRU model; We use weighted cross entropy as the loss function to solve the problem of sample imbalance, assign weights to different label classes, and give more attention to labels with fewer samples. The loss function loss is as follows: ； in is the loss weight; is the loss weight and sentiment category The ratio of the number of training utterances in , and a constant α is used to adjust the smoothness of the distribution; Indicates the actual emotion label; S5. Train the GRU model; S6. Input the prediction statement and obtain the training result.

2. The method for analyzing conversational text sentiment based on deep learning according to claim 1, characterized in that: In S1, the labels include anger, happiness, sadness, and neutral.

3. The method for analyzing conversational text sentiment based on deep learning according to claim 1, characterized in that: In S2, the normalization process includes converting uppercase words into lowercase, restoring special parts of speech to their original forms, removing non-alphanumeric symbols, and retaining question marks and exclamation marks; and using a pad mechanism to fill in the length of the conversation to keep the conversation length consistent.

4. The method for analyzing conversational text sentiment based on deep learning according to claim 1, characterized in that: The context-sensitive word embeddings Obtain a single utterance embedding through a max pooling operation ; For i conversations The learned word embeddings It is fed into a high-level bidirectional GRU to capture contextual information and is fed into a high-level bidirectional GRU to capture the sequence and contextual relationships of the discourse in the conversation. The calculation is as follows: ； Among them, the value of J is equal to the value of j, j∈{1,…,j} represents the number of utterances, and the sentence embedding related to the context is obtained through the attention mechanism and tanh function. ; Finally, the softmax function is used to determine the sentiment label corresponding to the sentence.

5. The method for analyzing conversational text sentiment based on deep learning according to claim 1, characterized in that: In S5, the data set information processed by S2 is put into the hierarchical GRU model in S3 to extract features, and the GRU model is trained according to the GRU model training parameters set in S4.

6. The method for analyzing conversational text sentiment based on deep learning according to claim 1, characterized in that: In S6, the prediction statement is put into the model trained in S5 for prediction, and finally the training result is obtained.

Images