A method for extracting emotional reasons based on a question-and-answer style dual attention mechanism

Abstract

A method for extracting emotional causes based on a dual attention mechanism in a question-and-answer format, the present invention relates to text classification technology. It is used to solve the problem that the potential causes of emotions in text cannot be accurately extracted in text-oriented sentiment analysis tasks. Emotional cause extraction can be used for public opinion analysis and customer evaluation analysis, which is of great significance for governments to serve the society and enterprises to improve their products. However, due to the lack of standardization of online short texts and the fact that the emotional information therein is greatly affected by the scenes in which the characters are located, it becomes very difficult to accurately extract the reasons behind the emotions. The present invention proposes an emotional cause extraction method based on a dual attention mechanism in a question-and-answer format. Experiments show that this method can effectively improve the accuracy of emotional cause extraction and accurately extract the reasons behind the generation of emotions in redundant short texts. The present invention is mainly used for the extraction of emotional causes from flexible and non-standardized online short texts.

Description

Technical Field

[0001] The present invention relates to text classification related technologies, in particular to sentiment reason extraction. Background Art

[0002] With the advent of the big data era, the world is facing the challenge of an unprecedented information explosion. Using sentiment analysis for public opinion analysis, public opinion monitoring, and customer review analysis is crucial for governments to better serve society and for businesses to improve their products. However, short texts on the internet are free-flowing, flexible, and lack standardization, and the emotional information contained therein is influenced by time, location, environment, and context. These factors complicate the identification of emotional elements in traditional text sequence annotation methods. Furthermore, existing text sentiment analysis methods often focus on superficial analysis, such as polarity classification, ignoring the interdependence between emotion and cause, and failing to identify the root causes of emotional events in text.

[0003] Currently, there are three types of methods for extracting emotional causes: rule-based methods, feature-based methods, and deep learning-based methods.

[0004] Rule-based methods were first used to solve the task of extracting emotional causes. Chen et al. divided the language prompts of causes into six categories and designed rules for cause detection based on the position of the prompts and their relative emotional words. Russo et al. used a common sense database to automatically expand the extraction rules of emotional causes through clustering and classification. The performance of rule-based methods depends on the quality and quantity of rule formulation. They are more applicable when data is scarce, but there are performance bottlenecks when processing large-scale data. Therefore, researchers began to study feature-based methods, defining event trees based on syntactic dependencies and converting the cause extraction task into a binary classification task of whether the event tree contains a cause clause.

[0005] In recent years, with the rapid development of deep learning technology, many researchers have used deep neural networks to solve the problem of sentiment cause extraction. Inspired by memory networks in question-answering (QA) systems, Gui et al. compared the sentiment cause extraction task to QA and designed a sentiment cause extraction model based on QA systems. This approach limits query input to sentiment words and ignores the context of sentiment words in the text (i.e., the clauses in which the sentiment words appear). Yu et al. proposed using a hierarchical network to model sentiment at the word, phrase, and clause levels. Deep learning methods leverage the vast amount of textual information in the network to fully learn and mine hidden information in the text, achieving a deep understanding of the text. They have achieved good results in text sentiment classification. However, due to the limited variety of public datasets, contextual information is difficult to fully utilize. Most methods only analyze sentiment polarity, ignoring the dependencies between the generated emotions and their causes. Summary of the Invention

[0006] The purpose of the present invention is to solve the problem of better extraction of emotional causes in the field of sentiment analysis, and a method is proposed to model the problem into a triple problem (D, Q, A), where D represents a set of candidate cause clauses, Q represents a sentiment clause, and A represents a cause clause. The sentiment clauses and candidate cause clauses are encoded by a bidirectional LSTM, and the correlation between the sentiment clauses and the candidate cause clauses is established using a dual attention mechanism, which makes full use of the relationship between the text context to perform better emotional cause extraction.

[0007] The above invention objectives are mainly achieved through the following technical solutions:

[0008] 1. A question-answering-based dual attention mechanism emotion cause extraction method, including the following steps:

[0009] S1. Process the dataset and model the question into a question-answer triple (D, Q, A). Find the answer A to question Q in text D. D represents the candidate cause clause, Q represents the sentiment clause, and A represents the cause clause. Obtain a shared word embedding matrix through the word embedding model.

[0010] S2, build network structure;

[0011] S21. Obtain the word vectors of all words in the clause through the word embedding matrix;

[0012] S22. Use bidirectional LSTM to encode text and questions respectively to obtain semantic representations of text and questions containing contextual information;

[0013] S23, performing dot product calculation on the encoded text and the question to obtain a matching matrix M;

[0014] S24. Obtain the attention weight α of each column in the matrix column, which represents the degree of attention paid by the question word to each word in the text;

[0015] S25. Obtain the attention weight β of each row on the matrix row, which represents the degree of attention of the text word to each word in the question, and form a question-level attention distribution;

[0016] S26. Finally, through the dot product operation of α and β, the importance of different attention α is distinguished and they are combined more effectively, rather than using naive addition or average fusion into the final attention;

[0017] S3. Train the model to extract sentiment reason clauses.

[0018] 2. In step S1, the question is modeled as a question-answer triple (D, Q, A). The answer A to the question Q is found in the text D. Then the text and question are converted into one-hot vectors, which are then input into the embedding layer to generate a shared word embedding matrix.

[0019] 3. In step S2, the model is built, which includes the following steps:

[0020] S21. Obtain word vectors for all words in the clause through the word embedding layer.

[0021] S22. Encode the text and question. Use two bidirectional LSTMs to obtain the hidden states of the text and question containing context information. The state parameters are updated as follows:

[0022] i k =σ(W i w w k +W i h h k-1 +b i ) (1)

[0023] f k =σ(W f w w k +W f h w k-1 +b f ) (2)

[0024] o k =σ(W o w w k +W o h h k-1 +b o) (3)

[0025]

[0026] h k =o k ⊙tanh(c k ) (6)

[0027] Where k is the word index in the context; i, f, o are the input gate, forget gate, and output gate; σ is the activation function; W and b are the weight matrix and bias respectively; ⊙ is the element-wise multiplication; then the hidden layer state of the text is obtained and questions

[0028] S23. Perform dot product calculation on the encoded text and the question to obtain a matching matrix M, as follows:

[0029]

[0030] M(i, j) represents the sum of the dot products of the semantic encoding of the i-th word in the text and the semantic encoding of the j-th word in the question, which indicates the degree of pairwise matching between a text word and a question word;

[0031] S24. Perform softmax normalization on the columns of the matrix M to indicate the degree of attention paid by the question segmentation to each segmentation in the text. When considering a single word in the question, each column represents a separate document-level attention:

[0032] α(t)=softmax(M(1,t),…,M(n,t)) (8)

[0033] α=[α(1),α(2),…,α(m)] (9)

[0034] Where t∈(1,m), m is the length of the question; n is the length of the text;

[0035] S25. Perform softmax normalization on the M rows of the matrix to obtain the attention weight β of each row, which represents the degree of attention paid to each word in the document to the question, forming a question-level attention distribution:

[0036] β(t)=softmax(M(t,1),…,M(t,m)) (10)

[0037]

[0038] Where t∈(1,n), n is the length of the text; m is the length of the question;

[0039] S26. Finally, through the dot product operation of α and β, the importance of different attention α is distinguished and they are combined more effectively, rather than using naive addition or average fusion of the final attention:

[0040] s=α T β (12).

[0041] 4. In step S3, the goal of our training algorithm is to minimize the cross entropy loss of the training data. Adam is used to solve the optimization problem. The cross entropy loss function is defined as:

[0042]

[0043] N is the number of training instances, c is the text, c e Indicates a question, y i is the label l of c i The truth value of is the predicted probability distribution of c, l = {0, 1} is a set of labels, where 1 indicates that the text contains the answer (reason clause) and 0 indicates that it does not, and θ is the model parameter that needs to be updated.

[0044] Effects of the Invention

[0045] The invention proposes a method for extracting emotional causes based on a dual attention mechanism in a question-answering manner. We model the question as a question-answering triple (D, Q, A), and find the answer A to the question Q in the text D. D represents the candidate cause clause, Q represents the emotion clause, and A represents the cause clause. First, a word embedding model is used to obtain a shared word embedding matrix, and the word embedding matrix is used to obtain the text and question word vector representations. A bidirectional LSTM pair is used to encode them respectively, and then the dot product is calculated for the text and question to obtain a matching matrix M. In view of the problem that short online texts are flexible and lack standardization and it is difficult to utilize text context information, we use the matrix column to obtain the attention weight α of each column to represent the degree of attention of the question word to each word in the text, and the matrix row to obtain the attention weight of each row to represent the degree of attention of the text word to each word in the question. Finally, through the dot product operation of α and β, the importance of different attention α is distinguished, and they are more effectively combined, instead of using naive addition or average fusion into the final attention. Experiments have verified that this method can effectively improve the accuracy of emotional cause extraction and has good results on public datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Emotional cause extraction model structure diagram;

[0047] Figure 2 Statistics of the dataset used for model validation;

[0048] Figure 3Test results of different models on the dataset; Specific implementation methods Specific implementation method one:

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] like Figure 1 The paper provides a dual attention mechanism emotional reason extraction method based on question-answering:

[0052] The method mainly comprises the steps of:

[0053] 1. A question-answering-based dual attention mechanism emotion cause extraction method, including the following steps:

[0054] S1. Process the dataset and model the question into a question-answer triple (D, Q, A). Find the answer A to question Q in text D. D represents the candidate cause clause, Q represents the sentiment clause, and A represents the cause clause. Obtain a shared word embedding matrix through the word embedding model.

[0055] S2, build network structure;

[0056] S21. Obtain the word vectors of all words in the clause through the word embedding matrix;

[0057] S22. Use bidirectional LSTM to encode text and questions respectively to obtain semantic representations of text and questions containing contextual information;

[0058] S23, performing dot product calculation on the encoded text and the question to obtain a matching matrix M;

[0059] S24. Obtain the attention weight α of each column in the matrix column, which represents the degree of attention paid by the question word to each word in the text;

[0060] S25. Obtain the attention weight β of each row on the matrix row, which represents the degree of attention of the text word to each word in the question, and form a question-level attention distribution;

[0061] S26. Finally, through the dot product operation of α and β, the importance of different attention α is distinguished and they are combined more effectively, rather than using naive addition or average fusion into the final attention;

[0062] S3. Train the model to extract sentiment reason clauses.

[0063] 2. In step S1, the question is modeled as a question-answer triple (D, Q, A). The answer A to the question Q is found in the text D. Then the text and question are converted into one-hot vectors, which are then input into the embedding layer to generate a shared word embedding matrix.

[0064] 3. In step S2, the model is built, which includes the following steps:

[0065] S21. Obtain word vectors for all words in the clause through the word embedding layer.

[0066] S22. Encode the text and question. Use two bidirectional LSTMs to obtain the hidden states of the text and question containing context information. The state parameters are updated as follows:

[0067] i k =σ(W i w w k +W i h h k-1 +b i ) (1)

[0068] f k =σ(W f w w k +W f h w k-1 +b f ) (2)

[0069] o k =σ(W o w w k +W o h h k-1 +b o ) (3)

[0070]

[0071] h k =o k ⊙tanh(c k ) (6)

[0072] Where k is the word index in the context; i, f, o are the input gate, forget gate, and output gate; σ is the activation function; W and b are the weight matrix and bias respectively; ⊙ is the element-wise multiplication; then the hidden layer state of the text is obtained and questions

[0073] S23. Perform dot product calculation on the encoded text and the question to obtain a matching matrix M, as follows:

[0074]

[0075] M(i, j) represents the sum of the dot products of the semantic encoding of the i-th word in the text and the semantic encoding of the j-th word in the question, which indicates the degree of pairwise matching between a text word and a question word;

[0076] S24. Perform softmax normalization on the columns of the matrix M to indicate the degree of attention paid by the question segmentation to each segmentation in the text. When considering a single word in the question, each column represents a separate document-level attention:

[0077] α(t)=softmax(M(1,t),…,M(n,t)) (8)

[0078] α=[α(1),α(2),…,α(m)] (9)

[0079] Where t∈(1,m), m is the length of the question; n is the length of the text;

[0080] S25. Perform softmax normalization on the M rows of the matrix to obtain the attention weight β of each row, which represents the degree of attention paid to each word in the document to the question, forming a question-level attention distribution:

[0081] β(t)=softmax(M(t,1),…,M(t,m)) (10)

[0082]

[0083] Where t∈(1,n), n is the length of the text; m is the length of the question;

[0084] S26. Finally, through the dot product operation of α and β, the importance of different attention α is distinguished and they are combined more effectively, rather than using naive addition or average fusion of the final attention:

[0085] s=α T β (12).

[0086] 4. In step S3, the goal of our training algorithm is to minimize the cross entropy loss of the training data. Adam is used to solve the optimization problem. The cross entropy loss function is defined as:

[0087]

[0088] N is the number of training instances, c is the text, ce Indicates a question, y i is the label l of c i The truth value of is the predicted probability distribution of c, l = {0, 1} is a set of labels, where 1 indicates that the text contains the answer (reason clause) and 0 indicates that it does not, and θ is the model parameter that needs to be updated.

[0089] The present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of the present invention.

Claims

1. A question-answering-based dual attention mechanism emotion cause extraction method, including the following steps: S1. Process the dataset and model the question into a question-answer triple (D, Q, A). Find the answer A to question Q in text D. D represents the candidate cause clause, Q represents the sentiment clause, and A represents the cause clause. Obtain a shared word embedding matrix through the word embedding model. S2, build network structure; S21. Obtain the word vectors of all words in the clause through the word embedding matrix; S22. Use bidirectional LSTM to encode text and questions respectively to obtain semantic representations of text and questions containing contextual information; S23, performing dot product calculation on the encoded text and the question to obtain a matching matrix M; S24. Obtain the attention weight α of each column in the matrix column, which represents the degree of attention paid by the question word to each word in the text; S25. Obtain the attention weight β of each row on the matrix row, which represents the degree of attention of the text word to each word in the question, and form a question-level attention distribution; S26. Finally, through the dot product operation of α and β, the importance of different attention α is distinguished and they are combined more effectively, rather than using naive addition or average fusion into the final attention; S3. Train the model to extract sentiment reason clauses.

2. The emotional cause extraction method based on the question-answering dual attention mechanism according to claim 1, characterized in that: In step S1, the question is modeled as a question-answer triple (D, Q, A), the answer A to the question Q is found in the text D, and then the text and question are converted into one-hot vectors, which are then input into the embedding layer to generate a shared word embedding matrix.

3. The emotional cause extraction method based on the question-answering dual attention mechanism according to claim 1, characterized in that: The model is built in step S2, which includes the following steps: S21. Obtain word vectors for all words in the clause through the word embedding layer. S22. Encode the text and question. Use two bidirectional LSTMs to obtain the hidden states of the text and question containing context information. The state parameters are updated as follows: i k =σ(W i w w k +W i h h k-1 +b i ) (1) f k =σ(W f w w k +W f h w k-1 +b f ) (2) o k =σ(W o w w k +W o h h k-1 +b o ) (3) h k =o k ⊙tanh(c k ) (6) Where k is the word index in the context; i, f, o are the input gate, forget gate, and output gate; σ is the activation function; W and b are the weight matrix and bias respectively; ⊙ is the element-wise multiplication; then the hidden layer state of the text is obtained and questions S23. Perform dot product calculation on the encoded text and the question to obtain a matching matrix M, as follows: M(i, j) represents the sum of the dot products of the semantic encoding of the i-th word in the text and the semantic encoding of the j-th word in the question, which indicates the degree of pairwise matching between a text word and a question word; S24. Perform softmax normalization on the columns of the matrix M to indicate the degree of attention paid by the question segmentation to each segmentation in the text. When considering a single word in the question, each column represents a separate document-level attention: α(t)=softmax(M(1,t),…,M(n,t)) (8) α＝[α(1),α(2),…,α(m)] (9) Where t∈(1,m), m is the length of the question; n is the length of the text; S25. Perform softmax normalization on the M rows of the matrix to obtain the attention weight β of each row, which represents the degree of attention paid to each word in the document to the question, forming a question-level attention distribution: β(t)=softmax(M(t,1),…,M(t,m)) (10) Where t∈(1,n), n is the length of the text; m is the length of the question; S26. Finally, through the dot product operation of α and β, the importance of different attention α is distinguished and they are combined more effectively, rather than using naive addition or average fusion of the final attention: s＝α T β (12)。 4. The emotional cause extraction method based on the question-answer dual attention mechanism according to claim 1, characterized in that: The goal of our training algorithm in step S3 is to minimize the cross entropy loss of the training data. Adam is used to solve the optimization problem. The cross entropy loss function is defined as: N is the number of training instances, c is the text, c e Indicates a question, y i is the label l of c i The truth value of is the predicted probability distribution of c, l = {0, 1} is a set of labels, where 1 indicates that the text contains the answer (reason clause) and 0 indicates that it does not, and θ is the model parameter that needs to be updated.

Images