A fine-grained sentiment analysis method for social media text based on attention network

Abstract

The present invention discloses a method for fine-grained sentiment analysis of social media text based on an attention network, which relates to the field of deep learning. The present invention adopts a nested attention mechanism to extract semantic features by calculating the bidirectional attention between context text and aspect words, fully obtaining the bidirectional attention weights between aspect words and context, and capturing the interactive relationship between the two. At the same time, an enhanced feature extraction network is also used, using the output of the Transformer as the input of a bidirectional LSTM. The bidirectional LSTM network structure is used to compensate for the problem that the position encoding in the Transformer only has the absolute position information of the sequence, but loses the relative position information. This makes the network have more powerful feature extraction capabilities, thereby enabling the method to fully understand the semantic information of the text, correctly divide the emotional expression range of the target aspect, and improve the accuracy of sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a fine-grained sentiment analysis method for social media text based on an attention network. Background Art

[0002] The rapid development of the internet has generated a vast amount of meaningful commentary from diverse audiences on blogs, forums, and other social media platforms. These comments embody diverse emotions and tendencies, such as joy, anger, sorrow, and happiness, as well as praise and criticism. Analyzing the emotional undertones of these potential user comments can help us understand public opinion responses to current hot topics.

[0003] For some time, researchers have been dedicated to testing the feasibility of machine-human interaction. With the proliferation of the internet and the popularization of social media, various online platforms have continued to grow and proliferate, transforming the world into a data-rich world, ushering in the era of big data. The vast amount of natural language data available on online platforms has fueled advances in natural language processing (NLP), unlocking enormous potential for more advanced methods like sentiment analysis in modern applications. Ever since Alan Turing devised the Turing test in 1950, experiments have been underway to determine whether and how artificial intelligence can rival human natural language. This has raised the question of whether artificial intelligence can accurately understand naturally occurring linguistic information and context, achieving human-like comprehension capabilities.

[0004] Sentiment analysis, a branch of natural language processing (NLP), focuses on studying people's emotional reactions and thoughts about events and products. It involves identifying sentiment trends around specific text. In today's society, the internet is a great choice for gaining advice, as a vast amount of data and discussion is readily available online. Comments can easily reveal what you're looking for. Organizations no longer need to resort to traditional methods to gather public opinion, given the abundance of publicly available information. In recent years, using social media to influence public opinion and brand certain companies or individuals has become commonplace. The rapid growth of social networks has also brought with it numerous challenges and contradictions. Consequently, sentiment analysis has become a crucial research tool for social media commentary. Research on sentiment analysis of social media comments can help identify online sentiment changes and tendencies regarding specific events, thereby preventing potential problems.

[0005] When sentiment analysis was first introduced, it primarily focused on extracting and identifying emotionally charged words within text. For example, "happy" was considered a positive word, and "sad" was considered a negative word. However, with increasing research, mainstream sentiment analysis methods have evolved into three basic models: those based on sentiment lexicons, machine learning, and deep learning. Among these, those based on deep learning are currently the most popular. They can develop different deep learning algorithms for different task types, achieving excellent results. Furthermore, sentiment analysis can be categorized into paragraph-level, sentence-level, and aspect-level sentiment analysis based on the granularity of the content. In the early days of sentiment analysis research, paragraph-level and sentence-level sentiment analysis were the main focus. These approaches assume that a text, regardless of its length or aspects, has a single sentiment and determine its overall sentiment through analysis and judgment. The difference between paragraph-level and sentence-level analysis lies solely in the length of the text being processed. Currently, sentiment analysis techniques for the entire text have matured and offer good performance. However, this analysis cannot effectively reflect people's fine-grained emotions regarding the target opinion, and can obscure some details within the text. Therefore, aspect-level sentiment analysis has emerged to investigate this hidden content. Aspect-level sentiment analysis requires determining the emotional scope of each target aspect within the context, whether explicit or implicit. However, natural language is flexible and contains abstract expressions such as pronouns. Therefore, the difficulty of aspect-level sentiment analysis lies in fully understanding the semantic information of the text and accurately categorizing the emotional expression range of the target aspect.

[0006] Text sentiment analysis of social media comments currently has a wide range of applications. By collecting a large number of relevant comments, it can identify key product features and their corresponding sentiment, providing user assistance. In public opinion monitoring, the intersection of online and social information is increasingly impacting society. Using sentiment analysis to monitor public opinion information can effectively address the limitations of manual methods in processing massive amounts of information. Regarding information prediction, discussions about hot online events significantly influence the thinking of online users. Sentiment analysis can effectively analyze textual information such as news and posts online, making predictions and forecasts about the future direction of hot topics. Sentiment analysis also plays a crucial role in human-computer interaction fields such as information extraction and question-answering systems. Summary of the Invention

[0007] In response to the above-mentioned deficiencies in the prior art, the present invention provides a fine-grained sentiment analysis method for social media text based on an attention network, which solves the problem that existing sentiment analysis methods are difficult to correctly divide the emotional expression range of the target aspect, resulting in low sentiment analysis accuracy.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0009] A fine-grained sentiment analysis method for social media text based on an attention network is provided, which includes the following steps:

[0010] S1. Extract aspect words from the target sentence and perform word embedding operations on the target sentence and aspect words respectively to obtain the target sentence vector and aspect word vector respectively;

[0011] S2. Feature extraction is performed on the target sentence vector and the aspect word vector respectively to obtain the hidden state vector of the target sentence and the hidden state vector of the aspect word respectively;

[0012] S3. Obtain the attention from the hidden state vector of the target sentence to the hidden state vector of the aspect word and the attention from the hidden state vector of the aspect word to the hidden state vector of the target sentence through the attention mechanism; perform weighted operations on the hidden state vector of the aspect word and the hidden state vector of the target sentence with the corresponding attention, and obtain two weighted results;

[0013] S4. The two weighted results are combined to obtain classification features. The classification features are probability-calculated using softmax to obtain the sentiment polarity of the aspect words, completing fine-grained sentiment analysis of social media texts.

[0014] Furthermore, the specific method of step S1 is:

[0015] Aspect words are extracted from the target sentence, and the target sentence and aspect words are projected into a low-dimensional word vector through the Glove model. The projection result corresponding to the target sentence is used as the target sentence vector, and the projection result corresponding to the aspect word is used as the aspect word vector.

[0016] Furthermore, the specific method of step S2 includes the following sub-steps:

[0017] S2-1. Generate positional encoding of the same dimension for the aspect word vector using the transformer encoder according to the trigonometric function, and superimpose the positional encoding on the aspect word vector to obtain the word embedding vector;

[0018] S2-2, input the word embedding vector into the multi-head attention model to calculate the serial correlation and obtain the serial correlation;

[0019] S2-3, add the word embedding vector and the sequence correlation through residual connection, and normalize the addition result through feedforward neural network to obtain the encoding result corresponding to the aspect word;

[0020] S2-4, input the encoding results corresponding to the aspect words into the Bi-LSTM network to obtain the forward hidden state and reverse hidden state corresponding to the aspect words;

[0021] S2-5, concatenate the forward hidden state and reverse hidden state corresponding to the aspect word to obtain the hidden state vector of the aspect word ;

[0022] S2-6, process the target sentence vector using the same method as steps S2-1 to S2-5 to obtain the hidden state vector of the target sentence .

[0023] Furthermore, the specific method of step S3 includes the following sub-steps:

[0024] S3-1, according to the formula:

[0025]

[0026] Get the hidden state vector of the target sentence and the hidden state vector of the aspect word Interaction Matrix I ;

[0027] S3-2, according to the formula:

[0028]

[0029]

[0030] Get the aspect words to the target sentence j Attention weights , and the target sentence to the aspect word i Attention weights ;in Represents the interaction matrix I Middle i Rank j Elements of the column; represents an exponential function with the natural constant e as the base;

[0031] S3-3. Average all attention weights from aspect words to target sentences to obtain the attention weight at the aspect word level ; Average all attention weights from the target sentence to the aspect words to get the attention weight at the target sentence level ;

[0032] S3-4, according to the formula:

[0033]

[0034]

[0035] Get the attention from the hidden state vector of the target sentence to the hidden state vector of the aspect word Attention from the hidden state vector of the aspect word to the hidden state vector of the target sentence ;in Represents the transpose of a matrix;

[0036] S3-5, according to the formula:

[0037]

[0038]

[0039] Get the weighted result corresponding to the target sentence Weighted results corresponding to aspect words .

[0040] Furthermore, the specific method of step S4 includes the following sub-steps:

[0041] S4-1. Weighted result corresponding to the target sentence Weighted results corresponding to aspect words Perform vector splicing to obtain classification features r ;

[0042] S4-2, according to the formula:

[0043]

[0044]

[0045] Get classification features r The probability of belonging to sentiment polarity c ;in The calculation result of the emotional polarity c is represented; For the i The calculation results of the emotional polarity; C is the total number of emotional polarity categories; and Represent the weight matrix and bias respectively;

[0046] S4-3, classification features r The corresponding maximum probability sentiment polarity is used as the sentiment polarity of the corresponding aspect word to complete the fine-grained sentiment analysis of social media text.

[0047] The beneficial effects of the present invention are as follows: the present invention adopts a nested attention mechanism to calculate the bidirectional attention between the context text and the aspect words to extract semantic features, fully obtain the bidirectional attention weights between the aspect words and the context, and capture the interactive relationship between the two. At the same time, an enhanced feature extraction network is also used, and the output of the Transformer is used as the input of the bidirectional LSTM. The network structure of the bidirectional LSTM is used to make up for the problem that the position encoding in the Transformen only has the absolute position information of the sequence, but loses the relative position information, so that the network has a more powerful feature extraction capability, thereby enabling the method to fully understand the semantic information of the text, correctly divide the emotional expression range of the target aspect, and improve the accuracy of the sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the process of this method. DETAILED DESCRIPTION

[0049] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0050] like Figure 1 As shown in Figure 1, the attention network-based fine-grained sentiment analysis method for social media text includes the following steps:

[0051] S1. Extract aspect words from the target sentence and perform word embedding operations on the target sentence and aspect words respectively to obtain the target sentence vector and aspect word vector respectively;

[0052] S2. Feature extraction is performed on the target sentence vector and the aspect word vector respectively to obtain the hidden state vector of the target sentence and the hidden state vector of the aspect word respectively;

[0053] S3. Obtain the attention from the hidden state vector of the target sentence to the hidden state vector of the aspect word and the attention from the hidden state vector of the aspect word to the hidden state vector of the target sentence through the attention mechanism; perform weighted operations on the hidden state vector of the aspect word and the hidden state vector of the target sentence with the corresponding attention, and obtain two weighted results;

[0054] S4. The two weighted results are combined to obtain classification features. The classification features are probability-calculated using softmax to obtain the sentiment polarity of the aspect words, completing fine-grained sentiment analysis of social media texts.

[0055] In one embodiment of the present invention, the input is a sentence and an aspect word .in Represents a word in a sentence. Aspect words can be one or more words, expressing a certain aspect of a sentence. This implementation uses the pre-trained Glove model to vectorize the text sequence. and aspect words Projected into a low-dimensional word vector, Represents the word embedding matrix (i.e., low-dimensional word vector), where v represents the vocabulary size, Indicates the dimension of the word vector obtained after projection. can be mapped to a vector After the word embedding operation, the vector representation of the sentence text is obtained and the vector representation of the target (i.e., target sentence vector and aspect word vector), which are used as input to the next stage. and denote the length of the sentence and target aspects respectively.

[0056] In step S2, the word vector embedding result as well as As input, they are input into the encoder of the Transformer. First, according to the trigonometric function, the position encoding of the input word vector is generated with the same dimension, and the position encoding is superimposed on the word vector to obtain a new word embedding vector. , this word embedding vector contains the absolute position information of each element in the sequence. The superimposed word embedding vector is then used as input to the multi-head attention part to calculate the correlation between sequences. The correlation calculation formula is:

[0057]

[0058]

[0059] in The embedding vectors of the input are respectively Multiply by the parameter matrix to get; for The number of columns of the matrix; Indicates the i an attention head; is the weight matrix.

[0060] pass With softmax normalization, the model can design the attention matrix into a standard normal distribution, so that the sum of the attention weights between each word and other words is 1, so that the embedding of each word contains the information of all words in the sentence. In Transformer, n is usually equal to 8. Next, output Input into the residual connection and feedforward neural network. The residual connection adds the input of the previous layer to the output of the multi-head attention. The result after addition is normalized to the standard normal distribution to accelerate convergence. The calculation formula is:

[0061]

[0062] The feedforward neural network undergoes two linear mappings and activation functions, and the calculation formula is:

[0063]

[0064] Finally, after a residual connection and normalization, the output of the Transformer-encoder layer is obtained for the target sentence and aspect words respectively. and , its dimension is consistent with the input dimension.

[0065] Then the output is sent to the Bi-LSTM network, which uses two parallel and reverse long short-term memory networks to learn the hidden semantics of the words in the target sentence and aspect words. Each Bi-LSTM is composed of two LSTM networks stacked in reverse. Input into Bi-LSTM to obtain the forward hidden state and the reverse hidden state , and finally through concat splicing, the final output is obtained, that is, the hidden state vector of the target sentence In the same way, we can get the output corresponding to the aspect word, that is, the hidden state vector of the aspect word .

[0066] In step S3, first, the sentence output by the Bi-LSTM network structure is represented as and aspect words As input, the interaction matrix between the target sentence and the aspect words is calculated based on these two parts of input , the calculation formula is:

[0067]

[0068] Interaction Matrix Each element in The value of represents the correlation between the aspect word and the word pair of the target sentence, but this correlation cannot represent the influence weight of each word on each other. Therefore, the attention weight of the aspect word to the sentence is calculated based on the column and row respectively. And the attention weight from sentence to aspect word .

[0069]

[0070]

[0071] Through as well as After averaging column by column and row by row, the attention weight of the current sentence level is obtained And the attention weight at the aspect word level , which respectively represent the important part of the current sentence considering the influence of each word in the aspect word, and the important part of the aspect word considering the influence of each word in the sentence.

[0072]

[0073]

[0074] The final interactive attention calculation between the sentence and the aspect word will be nested with another layer of attention mechanism based on the previous attention weight, which is weighted by the current level of attention and the previous interactive attention weight. , the attention weight from each individual target to the sentence As a weighting coefficient, the contribution of each aspect word is considered to obtain the attention weight of each word in the sentence. , the attention weight from sentence to target As a weighting coefficient, considering the contribution of each word in the sentence, the attention weight of each word in the aspect word is obtained, and the calculation formula is:

[0075]

[0076]

[0077] After obtaining the attention weights of the sentence and aspect words, the final sentence-level representation , by the final sentence attention And the sentence-level hidden layer state after the Bi-LSTM network structure Calculate, similarly, the final representation of the aspect word It is also composed of the final aspect word attention and the aspect word hidden layer state through the Bi-LSTM network Calculated.

[0078]

[0079]

[0080] The two vectors we finally get are as well as It is the representation of sentences and aspect words obtained by dual attention weight calculation, which fully reflects the characteristic relationship of two-way interaction between sentence context and aspect words.

[0081] In step S4, this embodiment converts the final sentence level representation The final expression of the aspect word Perform feature fusion by vector splicing to obtain the final classification features , input the classification features into the linear layer, and use linear transformation to map the high-dimensional fusion feature vector to the dimensional vector of the number of classification categories. and Represent the weight matrix and bias in turn. Finally, softmax is used to calculate its probability distribution on each label category. The calculation formula is:

[0082]

[0083]

[0084] The final result of sentiment polarity prediction for aspect targets is to take the label value with the largest probability value in the probability distribution obtained by the softmax function, and use this label value as the final prediction result.

[0085] In this embodiment, the cross entropy loss function with an L2 regularization term can be used as the loss function of the entire solution, and its calculation formula is:

[0086]

[0087] in represents the indicator function, represents the L2 regularization parameter in Bi-LSTM and linear layers, Represents a set of weight coefficient matrices.

[0088] In summary, the present invention employs a nested attention mechanism to extract semantic features by calculating the bidirectional attention between the context text and the aspect words. This mechanism fully captures the bidirectional attention weights between the aspect words and the context, and captures the interaction between the two. Furthermore, an enhanced feature extraction network is employed, using the Transformer output as the input to a bidirectional LSTM. The bidirectional LSTM network structure overcomes the problem in Transformer that position encoding only provides the absolute position information of the sequence, while losing the relative position information. This results in a more powerful feature extraction network.

Claims

1. A fine-grained sentiment analysis method for social media text based on attention network, characterized by: The following steps are involved: S1. Extract aspect words from the target sentence and perform word embedding operations on the target sentence and aspect words respectively to obtain the target sentence vector and aspect word vector respectively; S2. Feature extraction is performed on the target sentence vector and the aspect word vector respectively to obtain the hidden state vector of the target sentence and the hidden state vector of the aspect word respectively; S3. Obtain the attention from the hidden state vector of the target sentence to the hidden state vector of the aspect word and the attention from the hidden state vector of the aspect word to the hidden state vector of the target sentence through the attention mechanism; perform weighted operations on the hidden state vector of the aspect word and the hidden state vector of the target sentence with the corresponding attention, and obtain two weighted results; S4, fuse the two weighted results to obtain classification features; By using softmax to calculate the probability of classification features, we can obtain the sentiment polarity of aspect words and complete fine-grained sentiment analysis of social media texts. The specific method of step S3 includes the following sub-steps: S3-1, according to the formula: Get the hidden state vector of the target sentence and the hidden state vector of the aspect word Interaction Matrix I ; S3-2, according to the formula: Get the aspect words to the target sentence j Attention weights , and the target sentence to the aspect word i Attention weights ;in Represents the interaction matrix I Middle i Rank j Elements of the column; represents an exponential function with the natural constant e as the base; S3-3. Average all attention weights from aspect words to target sentences to obtain the attention weight at the aspect word level ; Average all attention weights from the target sentence to the aspect words to get the attention weight at the target sentence level ; S3-4, according to the formula: Get the attention from the hidden state vector of the target sentence to the hidden state vector of the aspect word Attention from the hidden state vector of the aspect word to the hidden state vector of the target sentence ;in Represents the transpose of a matrix; S3-5, according to the formula: Get the weighted result corresponding to the target sentence Weighted results corresponding to aspect words .

2. The method for fine-grained sentiment analysis of social media text based on attention network according to claim 1 is characterized in that: The specific method of step S1 is: Aspect words are extracted from the target sentence, and the target sentence and aspect words are projected into a low-dimensional word vector through the Glove model. The projection result corresponding to the target sentence is used as the target sentence vector, and the projection result corresponding to the aspect word is used as the aspect word vector.

3. The method for fine-grained sentiment analysis of social media text based on attention network according to claim 1 is characterized in that The specific method of step S2 includes the following sub-steps: S2-1. Generate positional encoding of the same dimension for the aspect word vector using the transformer encoder according to the trigonometric function, and superimpose the positional encoding on the aspect word vector to obtain the word embedding vector; S2-2, input the word embedding vector into the multi-head attention model to calculate the serial correlation and obtain the serial correlation; S2-3, add the word embedding vector and the sequence correlation through residual connection, and normalize the addition result through feedforward neural network to obtain the encoding result corresponding to the aspect word; S2-4, input the encoding results corresponding to the aspect words into the Bi-LSTM network to obtain the forward hidden state and reverse hidden state corresponding to the aspect words; S2-5, concatenate the forward hidden state and reverse hidden state corresponding to the aspect word to obtain the hidden state vector of the aspect word ; S2-6, process the target sentence vector using the same method as steps S2-1 to S2-5 to obtain the hidden state vector of the target sentence .

4. The method for fine-grained sentiment analysis of social media text based on attention network according to claim 1 is characterized in that The specific method of step S4 includes the following sub-steps: S4-1. Weighted result corresponding to the target sentence Weighted results corresponding to aspect words Perform vector splicing to obtain classification features r ; S4-2, according to the formula: Get classification features r The probability of belonging to sentiment polarity c ;in The calculation result of the emotional polarity c is represented; For the i The calculation results of the emotional polarity; C is the total number of emotional polarity categories; and Represent the weight matrix and bias respectively; S4-3, classification features r The corresponding maximum probability sentiment polarity is used as the sentiment polarity of the corresponding aspect word to complete the fine-grained sentiment analysis of social media text.

Images