A social text sentiment analysis method and device fusing emoticons

Abstract

The application discloses a social text sentiment analysis method and device fusing emoticons, and the method comprises the following steps: preprocessing unannotated emoticon-containing text data and annotated social text data, obtaining pre-training text data, and extracting emoticons in the pre-training text data as pre-training labels; a neural network model is built to pre-train the pre-training text data, and a pre-training model taking emoticons as prediction labels is obtained; the pre-processed annotated social text data is input into the pre-training model, the parameters of the pre-training model are updated through a chain unfreezing method, and a social text sentiment classifier is trained; social text needing sentiment classification is input into the social text sentiment classifier, and a prediction sentiment category is obtained; the application has the advantages that the limitation of emotion information loss caused by social text and emoticon segmentation processing is overcome, and the classification accuracy of social text is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and more particularly to a social text sentiment analysis method and device fusing emoticons. BACKGROUND

[0002] With the continuous development of social media, people gradually get used to expressing their own opinions and sharing their daily life on social platforms. The historical public data of social platforms contain a large amount of valuable sentiment information, which can not only be used for business, but also be used for psychology, cognitive linguistics or politics. Compared with traditional text expression, people express emotions in a more diverse way on social platforms. In recent years, emoticons have become popular in people's daily communication on social platforms with the rise of social media, becoming a new and widely used aspect of online communication. As a text symbol rich in emotional information, it can help us better analyze the sentiment of a large number of non-standard social texts.

[0003] With the widespread use of emoticons in social texts, some scholars have also begun to study the sentiment analysis of social texts fusing emoticons, trying to equate emoticons with text information, and converting emoticons and text into word vectors for input model training in sentiment analysis prediction tasks, and have achieved good classification results.

[0004] Currently, the social text sentiment analysis model fusing emoticons mainly includes the following categories: separating emoticons from text, converting emoticons into additional emotional information through manual labeling or data statistics to assist the sentiment classification task of text (rule-based method), such as the microblog sentiment analysis method based on emoticon dictionary and emotional common sense disclosed in Chinese patent publication No. CN111626050A. Maintaining the position relationship of emoticons in the text, fusing emoticons and text for pre-training into word embedding vectors to participate in the subsequent neural network model training (deep learning-based method), such as the text sentiment recognition method, device, storage medium and electronic equipment disclosed in Chinese patent publication No. CN111898384A.

[0005] However, the above-mentioned social text emotion analysis method fusing emoticons ignores the particularity of emoticons. The meanings of emoticons given by users in the process of use have deviated from the official initial definition; meanwhile, emoticons express different meanings due to different time, place, culture and context, so the rule-based method cannot have a good general rule definition for emoticons. In addition, emoticons often play the role of emotional labels in social text, and the emotional information contained in emoticons is richer than pure text. If only emoticons are converted into word vectors and processed with text in the deep learning-based method, the rich emotions given by users to emoticons will be lost, so the single use of the above-mentioned method will cause the loss of part of the key emotional information, and the classification accuracy of social text is not high. SUMMARY

[0006] The technical problem to be solved by the present application is that the existing technology fusion emoticon social text emotion analysis method is easy to cause the loss of part of the key emotional information, and the classification accuracy of social text is not high.

[0007] The present application solves the above-mentioned technical problem by the following technical means: a social text emotion analysis method fusing emoticons, comprising the following steps:

[0008] S1: Preprocessing unannotated emoticon-containing text data and annotated social text data, obtaining pre-training text data, and extracting emoticons in the pre-training text data as pre-training labels;

[0009] S2: Building a neural network model based on a long short-term memory sub-neural network and a self-attention mechanism sub-neural network to pre-train the pre-training text data in S1, obtaining a pre-training model with emoticons as prediction labels;

[0010] S3: Inputting the preprocessed annotated social text data in S1 into the pre-training model, updating the parameters of the pre-training model through a chain unfreezing method, and training to obtain a social text emotion classifier;

[0011] S4: Inputting social text needing emotion classification into the social text emotion classifier to obtain a predicted emotion category.

[0012] The present application proposes a method of pre-training text data by taking emoticons as pre-training model classification labels, which enables the rich emotional information of emoticons to be integrated into the pure text word feature representation in the pre-training process, overcoming the limitation of the loss of emotional information caused by the segmentation of social text and emoticons, and improving the classification accuracy of social text.

[0013] Further, the S1 comprises:

[0014] the text data containing emoticons as unannotated emoticon-containing text data and the social text data with sentiment annotations as annotated social text data;

[0015] filtering noise data in the unannotated emoticon-containing text data and the annotated social text data, the noise data including links, retweet symbols and usernames;

[0016] copying the text containing multiple emoticons in the data after filtering the noise data, so that each copied text contains only a single non-repeated emoticon in the original text and deleting the original text;

[0017] counting and selecting the emoticons with the top N frequencies in all text data to retain the corresponding pure text data as pre-training text data; and extracting the emoticons of the pre-training text data as unique pre-training labels of the corresponding text.

[0018] Further, the neural network model comprises an Embedding layer, two layers of bidirectional long short-term memory neural network layers, two layers of self-attention mechanism layers, a feature connection layer, a single-layer attention mechanism layer and a Softmax layer, the Embedding layer receives the pre-training text data, the output ends of the Embedding layer are connected with the input ends of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers respectively, the output ends of the Embedding layer, the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers are connected with the input end of the feature connection layer, the output end of the feature connection layer is connected with the input end of the single-layer attention mechanism layer, and the output end of the single-layer attention mechanism layer is connected with the Softmax layer.

[0019] Still further, the S2 comprises:

[0020] inputting the pre-training text data into an Embedding layer of a pre-training model to obtain an input vector sequence of the pre-training text data;

[0021] inputting the input vector sequence into two layers of bidirectional long short-term memory neural network layers and two layers of self-attention mechanism layers respectively to obtain syntax-related features and semantic-related features of the pre-training text data respectively;

[0022] performing vector splicing on the output feature vectors of each layer of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers and the input vector sequence output by the Embedding layer to obtain a connection feature vector of the pre-training text data;

[0023] inputting the connection feature vector into a single-layer attention mechanism layer, the single-layer attention mechanism layer being used for allowing the model to focus on key features of the pre-training text data sentence during model training, giving higher weights to key features, and obtaining a text representation vector of the pre-training text data;

[0024] inputting the text representation vector into a Softmax layer for normalization to obtain a predicted expression category label of the pre-training text data sentence;

[0025] based on a difference between the predicted expression category label and an actual expression result contained in the corresponding text, calculating a first loss function corresponding to the pre-training model, updating parameters of each module in the pre-training model, and training and completing a final pre-training model.

[0026] Further, an expression of the first loss function is

[0027]

[0028] wherein N represents a number of non-repeated expressions selected as pre-training category labels in the S1 preprocessing stage, y i represents a prediction of whether the text actually contains the i-th expression, and if the i-th expression is contained, y i is 1, otherwise 0; represents a probability of predicting that the text contains the i-th expression, and a sum of prediction probabilities of N expression labels is 1.

[0029] Further, a calculation method of the text representation vector is:

[0030] e t = h t w a

[0031]

[0032]

[0033] wherein h t is a representation of a word at a time step t, w a is a weight matrix of the single-layer attention mechanism layer, a t is an importance score at the time step t, and T is a total time step.

[0034] Further, the S3 comprises:

[0035] based on the labeled sentiment category of the labeled social text data, adjusting an output dimension of the Softmax layer of the pre-training model in the S2, so that the Softmax output dimension is a number of sentiment categories * a corresponding category probability distribution;

[0036] inputting the labeled social text data into a pre-training model in S2 to obtain a classification result of the social text;

[0037] calculating a second loss function corresponding to the neural network model based on a difference between the actual sentiment label and the model prediction label result of the labeled social text data, updating parameters of each module in the neural network model through a chain unfreezing method, and training to obtain a final social text sentiment classifier;

[0038] The chain unfreezing method is that, in the training process, the topmost parameters of the model are first fine-tuned, then the parameters are adjusted layer by layer upwards from the model Embedding layer to the optimal result, and finally the parameters of all layers of the model are trained and updated.

[0039] Further, the expression of the second loss function is:

[0040]

[0041] wherein N represents a number of non-repeated emojis selected as pre-training category labels in the S1 preprocessing stage, yi represents a prediction result of whether the predicted text actually contains the i-th emoji, yi is 1 if the predicted text contains the i-th emoji, and 0 otherwise, yi represents a probability that the predicted text contains the i-th emoji, and the sum of the probabilities of the N labels is 1. i i

[0042] The application further provides a social text sentiment analysis device fusing emojis, which comprises:

[0043] A data preprocessing module is configured to preprocess unlabeled emoji-containing text data and labeled social text data, obtain pre-training text data, and extract emojis in the pre-training text data as pre-training labels.

[0044] A pre-training module is configured to build a neural network model based on a long short-term memory sub-neural network and a self-attention mechanism sub-neural network, pre-train the pre-training text data in the data preprocessing module, and obtain a pre-training model taking emojis as prediction labels.

[0045] A transfer learning module is configured to input the preprocessed labeled social text data in the data preprocessing module into the pre-training model, update parameters of the pre-training model through a chain unfreezing method, and train to obtain a social text sentiment classifier.

[0046] A sentiment prediction module is configured to input social text that needs to be classified in terms of sentiment into the social text sentiment classifier, and obtain a predicted sentiment category.

[0047] Further, the data preprocessing module is further configured to:​​​

[0048] the text data containing emoticons as unannotated emoticon-containing text data and the social text data with sentiment annotations as annotated social text data;

[0049] filtering noise data in the unannotated emoticon-containing text data and the annotated social text data, the noise data including links, retweet symbols and usernames;

[0050] copying the text containing multiple emoticons in the data after filtering the noise data, so that each copied text contains only a single non-repeated emoticon in the original text and deleting the original text;

[0051] counting and selecting the emoticons with top N frequencies in all text data and retaining the corresponding pure text data as pre-training text data; extracting the emoticons of the pre-training text data as the unique pre-training labels of the corresponding text.

[0052] Further, the neural network model comprises an Embedding layer, two layers of bidirectional long short-term memory neural network layers, two layers of self-attention mechanism layers, a feature connection layer, a single-layer attention mechanism layer and a Softmax layer, the Embedding layer receives the pre-training text data, the output ends of the Embedding layer are connected with the input ends of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers respectively, the output ends of the Embedding layer, the output ends of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers are connected with the input end of the feature connection layer, the output end of the feature connection layer is connected with the input end of the single-layer attention mechanism layer, and the output end of the single-layer attention mechanism layer is connected with the Softmax layer.

[0053] Still further, the pre-training module is further configured to:

[0054] input the pre-training text data into the Embedding layer of the pre-training model to obtain an input vector sequence of the pre-training text data;

[0055] input the input vector sequence into the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers respectively to obtain syntax-related features and semantic-related features of the pre-training text data respectively;

[0056] vector splice the output feature vectors of each layer of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers with the input vector sequence output by the Embedding layer to obtain a connection feature vector of the pre-training text data;

[0057] inputting the connection feature vector into a single-layer attention mechanism layer, the single-layer attention mechanism layer being used for allowing the model to focus on key features of the pre-training text data sentence during model training, giving higher weights to key features, and obtaining a text representation vector of the pre-training text data;

[0058] inputting the text representation vector into a Softmax layer for normalization to obtain a predicted expression category label of the pre-training text data sentence;

[0059] based on a difference between the predicted expression category label and an actual expression result contained in the corresponding text, calculating a first loss function corresponding to the pre-training model, updating parameters of each module in the pre-training model, and training and completing a final pre-training model.

[0060] Further, an expression of the first loss function is

[0061]

[0062] wherein N represents a number of non-repeated expressions selected as pre-training category labels in the S1 preprocessing stage, y i represents a prediction of whether the text actually contains the i-th expression, and if the i-th expression is contained, y i is 1, otherwise 0; represents a probability of predicting that the text contains the i-th expression, and a sum of prediction probabilities of N expression labels is 1.

[0063] Further, a calculation manner of the text representation vector is:

[0064] e t =h t w a

[0065]

[0066]

[0067] wherein h t is a representation of a word at a time step t, w a is a weight matrix of the single-layer attention mechanism layer, a t is an importance score at the time step t, and T is a total time step.

[0068] Further, the transfer learning module is further used for:

[0069] based on the labeled sentiment category of the labeled social text data, adjusting an output dimension of the Softmax layer of the pre-training model in S2, so that the output dimension of the Softmax is the number of sentiment categories * corresponding category probability distribution.

[0070] inputting the labeled social text data into a pre-training model in S2 to obtain a classification result of the social text;

[0071] based on a difference between the actual sentiment label and the model prediction label result of the labeled social text data, calculating a second loss function corresponding to the neural network model, updating parameters of each module in the neural network model through a chain unfreezing method, and training to obtain a final social text sentiment classifier;

[0072] the chain unfreezing method is that, in the training process, the topmost parameters of the model are first fine-tuned, then the parameters are adjusted layer by layer upwards from the model Embedding layer to the optimal result, and finally the parameters of all layers of the model are trained and updated.

[0073] Further, the expression of the second loss function is:

[0074]

[0075] wherein N represents a number of non-repeated emoticons selected as pre-training category labels in the S1 preprocessing stage, y i represents whether the predicted text actually contains the i th emoticon, and y i is 1 if the predicted text contains the i th emoticon, and 0 otherwise; represents a probability that the predicted text contains the i th emoticon, and the sum of the probabilities of the N labels is 1.

[0076] The advantages of the present application are:

[0077] (1) The present application proposes a method of using emoticons as pre-training model classification labels to pre-train text data, which enables the rich emotional information of emoticons to be integrated into the pure text word feature representation in the pre-training process, overcoming the limitations of the loss of emotional information caused by the segmentation of social text and emoticons, and improving the classification accuracy of social text.

[0078] (2) The present application proposes a method of simultaneously extracting text features using a double-layer bidirectional long short-term memory neural network and a double-layer self-attention mechanism, ensuring that the model can focus on the semantic features of related words within the sentence while training syntactic features, and ensuring the integrity of the training data information.

[0079] (3) The present application uses non-emotion-labeled social platform raw data for pre-training, reducing the requirement of deep learning models for labeled training data, and achieving good results using only a small amount of labeled data, thereby reducing the amount of manual work caused by data labeling. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 A flowchart of a social text sentiment analysis method fusing emoticons according to an embodiment of the present application is shown in FIG. 1.

[0081] Figure 2 An implementation flowchart of pre-processing of a sentence in a social text sentiment analysis method fusing emoticons according to an embodiment of the present application is shown in FIG. 2.

[0082] Figure 3 A structure diagram of a pre-training model in a social text sentiment analysis method fusing emoticons according to an embodiment of the present application is shown in FIG. 3.

[0083] Figure 4 A diagram of a chain unfreezing method in a social text sentiment analysis method fusing emoticons according to an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0084] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0085] Embodiment 1

[0086] As shown in FIG. 1, a social text sentiment analysis method fusing emoticons includes the following steps: Figure 1

[0087] S1: Pre-process unannotated emoticon-containing text data and annotated social text data, obtain pre-training text data, and extract emoticons therein as pre-training labels; see FIG. 2. Figure 2 S1: Pre-process unannotated emoticon-containing text data and annotated social text data, filter noise content in the text, which can include:

[0088] Collect large-scale unannotated emoticon-containing English text corpus, annotate a small amount of social text corpus according to the required sentiment categories of the final prediction task, filter link, forwarding symbol, username and other noise content in the corpus through regularization and other methods, and obtain relatively pure text. Further, it can be selected whether to delete punctuation symbols or stop words and the like.

[0089] ​Further, the text data containing multiple emojis in the relatively pure text is copied so that each copied text only contains a single non-repeating emoji in the original text and the original text is deleted. The emojis of the data are extracted as unique pre-training labels of the corresponding text, and the top N emojis in terms of frequency are counted and selected to retain their corresponding pure text data as pre-training text data; there are a few commonly used emojis in the above selected pre-training text data, which occupy most of the data text, and other niche emojis occupy a small part of the text. Since the emoji is used as a pre-training model label, in order to balance the number of different labels, the text containing niche emojis is simply copied through up-sampling, so that the number of pre-training texts with different emojis as labels is the same, so as to balance the data.

[0090] S2: building a neural network model based on a long short-term memory sub-neural network and a self-attention mechanism sub-neural network to pre-train the pre-training text data in S1 to obtain a pre-training model with emojis as prediction labels; see Figure 3 , building a neural network model based on a long short-term memory sub-neural network and a self-attention mechanism sub-neural network to pre-train the pre-training text data in S1 to obtain a pre-training model with emojis as prediction labels can include:

[0091] An exemplary neural network model is built using a deep learning framework such as Tensorflow or Keras, see Figure 3 The pre-training text is input into the model Embedding layer, which is used to vectorize the pre-training text data to obtain the corresponding input vector sequence.

[0092] For example, for the sentence "EDG won the world championship and I was so excited" in the pre-training text corpus, the sentence can be segmented to obtain "EDG", "won", "the", "world", "championship", "and", "I", "was", "so", "excited". Further, the Embedding layer can be used to convert the sentence into a corresponding input vector sequence representation: x = {x1, x2,..., xn}, xn represents the word vector corresponding to the nth word. n}, xn represents the word vector corresponding to the nth word. n

[0093] The obtained input vector sequence is then input into two layers of bidirectional long short-term memory neural network layer (BiLSTM) and two layers of self-attention mechanism layer (Self-Attention) respectively to obtain the syntax-related features and semantic-related features of the pre-training text data.​

[0094] Then, the BiLSTM is used to convert the input vector sequence to obtain the hidden state features corresponding to the pre-trained text data sentence, wherein the BiLSTM can capture the context dependency of each word in the sentence to obtain the overall sequence information of the sentence. For example, the obtained corresponding hidden state features can be represented as H n = {h1, h2,..., hn} n , wherein hn represents the hidden state feature corresponding to the nth word. n

[0095] The Self-Attention is used to extract semantic features of the input vector sequence to obtain semantic features corresponding to the pre-trained text data sentence. The self-attention mechanism layer can calculate the attention score of each pair of elements based on the self-attention mechanism sub-neural network in parallel. Unlike the BiLSTM which depends on external label information, the Self-Attention calculates the word-level relationship by focusing on the internal information of the sentence. A sem ∈ R n×n The obtained attention score matrix pays more attention to the internal information of the sentence. The attention score matrix is calculated in the prior art. Specifically, the attention score matrix A sem can be calculated by the following formula:

[0096]

[0097]

[0098] wherein Q and K are equal to the input vector sequence, WQ and WK represent the learnable parameter weight matrix, d k represents the scaling coefficient, d represents the dimension of the input vector sequence, g represents the number of attention channels, and T represents the transpose.

[0099] Then, the obtained two layers of output syntax-related features, two layers of output semantic-related features, and the input vector sequence output by the Embedding layer are spliced by a feature connection layer. The connection feature vector of the feature connection layer is input into an attention mechanism layer to obtain a text representation vector with the pre-trained text data. Finally, the text representation vector is input into a Softmax layer for normalization to obtain the final predicted label value.

[0100] Specifically, the Attention layer is used to give higher weight to the key features in the pre-trained text data sentence to obtain the final text representation vector. The key text representation vector v can be calculated by the following formula:

[0101] e​t = h t w a

[0102]

[0103]

[0104] where h t is the representation of a word at time step t, w a is the weight matrix of the Attention layer, a t is the importance score at time step t, which is constructed by multiplying the representation of a word at time step t h t with the weight matrix w a and then normalizing to get the probability distribution of the word; finally, the representation vector v of the text is obtained by weighted sum of the word representations at all time steps with the importance score a i as the weight.

[0105] S3: input the preprocessed labeled social text data in S1 into the pre-trained model, update the parameters of the pre-trained model through the chain unfreezing method, and train to obtain a social text sentiment classifier; see Figure 4 , input the above preprocessed labeled social text into the pre-trained model in S2, fine-tune the model parameters layer by layer through the chain unfreezing method, and modify the output dimension according to the provided labeled sentiment classification categories, so that the output dimension of the Softmax layer is equal to the number of sentiment categories * sentiment category probability distribution. It should be noted that in the actual training process, the dimension output by the pre-training part is the same as the number of emoji symbols to be predicted, that is, the output dimension of the Softmax layer input vector (T*X) after normalization should be (number of emoji symbol categories * corresponding category probability distribution) to know which emoji symbol has the highest prediction probability. Similarly, in the model fine-tuning and sentiment prediction part, the output dimension of the Softmax layer needs to be modified to ensure that the output vector dimension is (number of sentiment categories * corresponding category probability distribution) to obtain which sentiment category has the highest prediction probability. Therefore, in the code, changing the output dimension only needs to modify the parameters of this layer, without special transformation. The obtained social text sentiment classifier model can include:

[0106] According to the above modification of the output dimension of the Softmax layer according to the labeled social text label category, the loss function of the model is optionally modified:

[0107]

[0108] where N represents the number of non-repeating emojis selected as pre-training category labels in the S1 preprocessing stage, y iy i represents whether the predicted text actually contains the i th emoji, and y i = 1 if the predicted text contains the i th emoji and y i = 0 otherwise. i y i = 1, otherwise 0; y i represents the probability that the predicted text contains the i th emoji, and the sum of the probabilities of the 30 labels is 1.

[0109] The labeled social text is input into the pre-training model, the chain unfreezing method is used to firstly adjust and update the parameters of the topmost layer to the optimal result, then the parameters are trained and adjusted layer by layer from the Embedding layer upwards, finally all layers of the model are unfrozen and the parameters are trained and updated, and the social text sentiment classifier is obtained.

[0110] S4: The social text needing sentiment classification is input into the social text sentiment classifier, and a predicted sentiment category is obtained.

[0111] According to the technical scheme, the emoji is used as a classification label of a pre-training model to pre-train text data, the pre-training model enables rich emotional information of the emoji to be integrated into pure text word feature representation in the pre-training process, the limitation that emotional information is lost due to segmentation processing of the social text and the emoji is overcome, and the classification accuracy of the social text is improved.

[0112] Embodiment 2

[0113] The application further provides a social text sentiment analysis device fusing emojis, and the device comprises:

[0114] A data preprocessing module is configured to pre-process unannotated emoji-containing text data and annotated social text data, obtain pre-training text data, and extract emojis in the pre-training text data as pre-training labels.

[0115] A pre-training module is configured to build a neural network model based on a long short-term memory sub-neural network and a self-attention mechanism sub-neural network, pre-train the pre-training text data in the data preprocessing module, and obtain a pre-training model taking emojis as prediction labels.

[0116] A transfer learning module is configured to input the annotated social text data pre-processed in the data preprocessing module into the pre-training model, update parameters of the pre-training model by using a chain unfreezing method, and train a social text sentiment classifier.

[0117] A sentiment prediction module is configured to input social text needing sentiment classification into the social text sentiment classifier, and obtain a predicted sentiment category.

[0118] Specifically, the data preprocessing module is further configured to:

[0119] The text data containing emoticons is taken as unannotated emoticon-containing text data, and the social text data with sentiment annotations is taken as annotated social text data;

[0120] Noise data in the unannotated emoticon-containing text data and the annotated social text data is filtered, the noise data including links, forwarding symbols and usernames;

[0121] The text containing multiple emoticons in the data after filtering the noise data is copied so that each copied text contains only a single non-repeated emoticon in the original text and the original text is deleted.

[0122] The emoticons with a frequency of N in all text data are counted and selected, and the corresponding pure text data is reserved as pre-training text data; the emoticons of the pre-training text data are extracted as unique pre-training labels of the corresponding text.

[0123] Specifically, the neural network model comprises an Embedding layer, two layers of bidirectional long short-term memory neural network layers, two layers of self-attention mechanism layers, a feature connection layer, a single-layer attention mechanism layer and a Softmax layer, the Embedding layer receives the pre-training text data, the output ends of the Embedding layer are connected with the input ends of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers respectively, the output ends of the Embedding layer, the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers are connected with the input end of the feature connection layer, the output end of the feature connection layer is connected with the input end of the single-layer attention mechanism layer, and the output end of the single-layer attention mechanism layer is connected with the Softmax layer.

[0124] More specifically, the pre-training module is further configured to:

[0125] input the pre-training text data into the Embedding layer of the pre-training model to obtain an input vector sequence of the pre-training text data;

[0126] input the input vector sequence into the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers respectively to obtain syntax-related features and semantic-related features of the pre-training text data respectively;

[0127] perform vector splicing on the output feature vectors of each layer of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers and the input vector sequence output by the Embedding layer to obtain a connection feature vector of the pre-training text data;

[0128] inputting the connection feature vector into a single-layer attention mechanism layer, the single-layer attention mechanism layer being configured to enable the model to focus on key features of the pre-training text data sentence during model training, and give higher weights to the key features, to obtain a text representation vector of the pre-training text data;

[0129] inputting the text representation vector into a Softmax layer for normalization to obtain a predicted expression category label of the pre-training text data sentence;

[0130] based on a difference between the predicted expression category label and an actual expression result contained in the corresponding text, calculating a first loss function corresponding to the pre-training model, updating parameters of each module in the pre-training model, and training and completing a final pre-training model.

[0131] More specifically, the first loss function has an expression as follows:

[0132]

[0133] wherein N represents a number of non-repeated expressions selected as pre-training category labels in the S1 preprocessing stage, yi represents a predicted expression label of the i-th expression, y i represents a predicted expression label of the i-th expression, and y i is 1 if the i-th expression is contained, and 0 otherwise. represents a probability of predicting that the text contains the i-th expression, and the sum of the prediction probabilities of the N expression labels is 1.

[0134] More specifically, the text representation vector is calculated in the following manner:

[0135] e t = h t w a

[0136]

[0137]

[0138] wherein h t is a representation of a word at a time step t, w a is a weight matrix of the single-layer attention mechanism layer, a t is an importance score at the time step t, and T is a total number of time steps.

[0139] More specifically, the transfer learning module is further configured to:

[0140] based on the labeled sentiment category of the labeled social text data, adjusting an output dimension of the Softmax layer of the pre-training model in S2, so that the output dimension of the Softmax is the number of sentiment categories * corresponding category probability distribution.

[0141] inputting the labeled social text data into the pre-trained model in S2 to obtain a classification result of the social text;

[0142] based on a difference between the actual sentiment label and the model predicted label result of the labeled social text data, calculating a second loss function corresponding to the neural network model, updating parameters of each module in the neural network model through a chain unfreezing method, and training to obtain a final social text sentiment classifier;

[0143] The chain unfreezing method is that, in the training process, the topmost parameters of the model are first fine-tuned, then the parameters are adjusted layer by layer upwards from the model Embedding layer to the optimal result, and finally the parameters of all layers of the model are trained and updated.

[0144] More specifically, the expression of the second loss function is:

[0145]

[0146] wherein N represents the number of non-repeated emojis selected as pre-training category labels in the S1 preprocessing stage, yi represents the presence or absence of the ith emoji in the predicted text, and yi is 1 if the ith emoji is contained, and 0 otherwise; and pi represents the probability of the predicted text containing the ith emoji, and the sum of the probabilities of the N labels is 1. i i

[0147] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.​​​

Claims

1. A method for social text sentiment analysis with fusion of emoticons, characterized in that, The method comprises the following steps: S1: preprocessing unannotated emoticon text data and annotated social text data, obtaining pre-training text data, and extracting emoticons in the pre-training text data as pre-training labels; S2: building a neural network model based on a long short-term memory sub-neural network and a self-attention mechanism sub-neural network to pre-train the pre-training text data in S1 to obtain a pre-training model with emoticons as prediction labels; S2 comprises: inputting the pre-training text data into an Embedding layer of the pre-training model to obtain an input vector sequence of the pre-training text data; inputting the input vector sequence into two layers of a bidirectional long short-term memory neural network layer and two layers of a self-attention mechanism layer respectively to obtain syntax-related features and semantic-related features of the pre-training text data respectively; concatenating the output feature vectors of each layer of the two layers of the bidirectional long short-term memory neural network layer and the two layers of the self-attention mechanism layer with the input vector sequence output by the Embedding layer to obtain a connection feature vector of the pre-training text data; inputting the connection feature vector into a single-layer attention mechanism layer, which is used to make the model pay attention to key features of the pre-training text data during training, give higher weights to the key features, and obtain a text representation vector of the pre-training text data; inputting the text representation vector into a Softmax layer for normalization to obtain a predicted emoticon category label of the pre-training text data sentence; based on the difference between the predicted emoticon category label and the actual emoticon result of the corresponding text, calculating a first loss function corresponding to the pre-training model, updating the parameters of each module in the pre-training model, and training and completing the final pre-training model; S3: inputting the annotated social text data preprocessed in S1 into the pre-training model, updating the parameters of the pre-training model through a chain unfreezing method, and training to obtain a social text sentiment classifier; S3 comprises: adjusting the output dimension of the Softmax layer of the pre-training model in S2 based on the annotated sentiment category of the annotated social text data, so that the output dimension of the Softmax is the number of sentiment categories * corresponding category probability distribution; inputting the annotated social text data into the pre-training model in S2 to obtain the classification result of the social text; based on the difference between the actual sentiment label of the annotated social text data and the model prediction label result, calculating a second loss function corresponding to the neural network model, updating the parameters of each module in the neural network model through a chain unfreezing method, and training to obtain a final social text sentiment classifier; the chain unfreezing method is: first fine-tune the topmost parameters of the model during training, then adjust the parameters layer by layer from the Embedding layer of the model to the optimal result, and finally train and update the parameters of all layers of the model; S4: inputting a social text that needs to be classified into the social text sentiment classifier to obtain a predicted sentiment category.

2. The method of claim 1, wherein the method further comprises: S1 comprises: The text data containing emoticons is used as unannotated emoticon-containing text data, and the social text data with sentiment annotations is used as annotated social text data; Noise data including links, forwarding symbols and usernames in the unannotated emoticon-containing text data and the annotated social text data is filtered; The text containing multiple emoticons in the data after filtering the noise data is copied so that each copied text contains only a single non-repeated emoticon in the original text and the original text is deleted; The emoticons with a frequency of N in all text data are counted and selected, and the corresponding pure text data is reserved as pre-training text data; and the emoticons of the pre-training text data are extracted as unique pre-training labels of the corresponding text.

3. The method of claim 1, wherein the method further comprises: The neural network model comprises an Embedding layer, two layers of bidirectional long short-term memory neural network layers, two layers of self-attention mechanism layers, a feature connection layer, a single-layer attention mechanism layer and a Softmax layer, the Embedding layer receives the pre-training text data, the output end of the Embedding layer is connected with the input ends of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers respectively, the output end of the Embedding layer, the output ends of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers are all connected with the input end of the feature connection layer, the output end of the feature connection layer is connected with the input end of the single-layer attention mechanism layer, and the output end of the single-layer attention mechanism layer is connected with the Softmax layer.

4. The method of claim 1, wherein the method further comprises: The expression of the first loss function is wherein N represents the number of non-repeated emojis selected as pre-training class labels in the S1 pre-processing stage, represents the presence or absence of the i-th emoji that is predicted to be contained in the text, and is 1 if the i-th emoji is contained in the text, and 0 otherwise; represents the presence or absence of the i-th emoji that is predicted to be contained in the text, and is 1 if the i-th emoji is contained in the text, and 0 otherwise; represents the probability that the i-th emoji is predicted to be contained in the text, and the sum of the prediction probabilities of the N emoji labels is 1.

5. The method of social text sentiment analysis with fusion of emoticons as claimed in claim 1, wherein, The text representation vector is calculated in the following manner: wherein, is a representation of a word at time step t, is a weight matrix of the attention mechanism layer of the single layer, is an importance score for time step t; is the total number of time steps.

6. The method of social text sentiment analysis with fusion of emoticons as claimed in claim 1, wherein, The expression of the second loss function is wherein N represents the number of non-repeated emojis selected as pre-training class labels in the S1 pre-processing stage, represents the presence or absence of the prediction that the text actually contains the i-th emoji, and is 1 when the text actually contains the i-th emoji and 0 otherwise; represents the presence or absence of the prediction that the text actually contains the i-th emoji, and is 1 when the text actually contains the i-th emoji and 0 otherwise; represents the probability that the text is predicted to contain the i-th emoji, and the sum of the probabilities of the N labels is 1.

7. A device for sentiment analysis of social text with fusion of emoticons, characterized in that, The device comprises: A data preprocessing module for preprocessing the unannotated emoticon-containing text data and the annotated social text data, obtaining pre-training text data, and extracting emoticons therein as pre-training labels; A pre-training module for building a neural network model based on long short-term memory sub-neural networks and self-attention mechanism sub-neural networks to pre-train the pre-training text data in the data preprocessing module, so as to obtain a pre-training model with emoticons as prediction labels; the pre-training module is also used for inputting the pre-training text data into an Embedding layer of the pre-training model to obtain an input vector sequence of the pre-training text data; The input vector sequence is input into two layers of bidirectional long short-term memory neural network layers and two layers of self-attention mechanism layers respectively to obtain syntax-related features and semantic-related features of the pre-training text data; The output feature vectors of each layer of the two layers of bidirectional long short-term memory neural network layers and the two layers of self-attention mechanism layers are vector-spliced with the input vector sequence output by the Embedding layer to obtain a connection feature vector of the pre-training text data; The connection feature vector is input into a single-layer attention mechanism layer, which is used for allowing the model to focus on key features of the pre-training text data during training, giving higher weights to the key features, and obtaining a text representation vector of the pre-training text data. The text representation vector is input into a Softmax layer for normalization to obtain a predicted expression category label of the pre-training text data sentence; A first loss function corresponding to the pre-training model is calculated based on a difference between the predicted expression category label and an actual expression contained in the corresponding text, parameters of each module in the pre-training model are updated, and a final pre-training model is trained and completed; The transfer learning module is configured to input the pre-processed labeled social text data in the data preprocessing module into the pre-training model, update parameters of the pre-training model through a chain unfreezing method, and train a social text sentiment classifier; the transfer learning module is further configured to: Adjust an output dimension of a Softmax layer in the pre-training model based on a labeled sentiment category of the labeled social text data, so that the output dimension of the Softmax is a number of sentiment categories * a corresponding category probability distribution; Input the labeled social text data into the pre-training model in the pre-training module to obtain a classification result of the social text; A second loss function corresponding to the neural network model is calculated based on a difference between an actual sentiment label of the labeled social text data and a model predicted label result, parameters of each module in the neural network model are updated through the chain unfreezing method, and a final social text sentiment classifier is trained. The chain unfreezing method is that, in the training process, first, the topmost parameters of the model are fine-tuned, then the parameters are adjusted layer by layer from the Embedding layer of the model upwards to the optimal result, and finally, parameters of all layers of the model are trained and updated; The sentiment prediction module is configured to input social text that needs to be classified into the social text sentiment classifier to obtain a predicted sentiment category.

8. The social text sentiment analysis apparatus of claim 7, wherein, The data preprocessing module is further configured to: Text data containing emojis are used as unlabeled text data containing emojis, and social text data with sentiment labels are used as labeled social text data; Noise data in the unlabeled text data containing emojis and the labeled social text data is filtered, and the noise data includes links, forwarding symbols, and user names; Text containing multiple emojis in the data after filtering the noise data is copied so that each copied text contains only a single non-repeated emoji in the original text and the original text is deleted; The frequency of the emojis in all text data is counted, and the top N emojis are selected to reserve their corresponding pure text data as pre-training text data; and the emojis of the pre-training text data are extracted as unique pre-training labels of the corresponding text.

Images