A multi-granularity sentiment analysis method for takeout user reviews

By preprocessing the takeaway review data and building a text classification model based on LSTM, combining the self-attention mechanism and the multi-head attention mechanism, the end-to-end model and the BERT-MRC model are used to integrate multiple models for sentiment analysis, and the current solution is solved There are problems in the art that it is difficult to extract user emotional information, and multi-grained emotion analysis and efficient emotion recognition are achieved.

CN113094502BActive Publication Date: 2024-11-05BEIJING UNIV OF TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202110303408.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-22
Publication Date
2024-11-05
Estimated Expiration
2041-03-22

AI Technical Summary

Technical Problem

The existing technology is difficult to accurately extract user emotional information from massive takeaway platform data. Traditional sentiment analysis algorithms are time-consuming and difficult to apply to big data scenarios, and it is difficult to achieve multi-grained sentiment analysis.

Method used

By preprocessing the takeaway review data, including traditional Chinese conversion, deduplication, emoji character processing, and word segmentation processing, and building a text classification model based on LSTM, combining the self-attention mechanism and the multi-head attention mechanism, the end-to-end model is adopted and The BERT-MRC model is trained and finally integrated multiple models for sentiment analysis.

Benefits of technology

It realizes multi-grained emotional analysis of takeaway review data, improves the accuracy and efficiency of emotional recognition, can more accurately judge user emotions, and is suitable for big data scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113094502B_ABST
    Figure CN113094502B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-granularity takeaway user comment sentiment analysis method. The sentiment analysis method comprises 1) preprocessing the comment data, including traditional-simplified conversion, data deduplication, expression data segmentation, etc.; 2) training the comment data with an end-to-end training network, and using this network as the baseline network, and the result of the network training is the baseline model; 3) adding an attention mechanism to the training network to improve the network's ability to extract data features; 4) training the data with multiple models, and fusing the multiple models to complement each other. The sentiment analysis of the takeaway comment data is finally achieved through the above operations. The relevant method designed by the present invention is oriented to real user data, takes into account various problems existing in the data, and can achieve better training and classification effects in a short time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention discloses a multi-granularity takeaway user review sentiment analysis method, which is mainly used for sentiment analysis of takeaway platform user reviews, and belongs to the technical field of text sentiment analysis. Background Art

[0002] Since the advent of platforms such as "Ele.me" in 2009, the number of users of food delivery platforms has increased exponentially. Today, food delivery platforms have become one of the largest and most valuable types of online platforms in China, becoming the most important information exchange platform and an important means of life and communication for people. When consumers choose to buy goods on food delivery platforms, in addition to paying attention to the price and sales volume of the goods, they will pay special attention to the comments of the goods. Merchants will also know consumers' feelings about the goods through consumers' comments on the goods, and use comments to magnify the advantages of the target goods and improve the shortcomings of the goods. The comments on current food delivery platforms are often sorted by time series, and the classification of comments is often classified according to good reviews, bad reviews, or "one star" to "five stars". Under this classification model, it is very difficult for users to accurately obtain the information they need, and it is easy to get lost in the massive platform data. How to extract users' subjective emotions from massive platform data and identify and utilize them has always been one of the most important research topics in the field of artificial intelligence and natural language processing.

[0003] Sentiment analysis is one of the most popular applications in the field of Natural Language Processing (NLP), and its application is becoming more and more extensive and important. The concept of sentiment analysis was first proposed by Nasukawa et al., also known as opinion mining, which refers to the use of natural language processing, machine learning, text classification and other technologies to systematically identify the emotional state and subjective information of emotional texts. Sentiment classification is mainly processed from the definition of the task, the method used and the granularity of the text. In recent years, sentiment analysis has been widely used in public opinion monitoring, topic inference, comment analysis and decision-making, and has high commercial and social value.

[0004] With the continuous development and progress of science and technology, in today's big data society, the role of text sentiment analysis is becoming more and more important. In response to many problems such as users cannot quickly understand the merchant information and merchants do not know how to improve user satisfaction, sentiment analysis helps to analyze the user's emotional tendency towards online ordering by studying the sentiment of catering review data and combining data mining. On the one hand, it can help users quickly understand the information of the store where they are ordering, and on the other hand, it can help merchants take different measures to increase the order rate according to the customer's emotions.

[0005] In sentiment analysis tasks, different classification methods will have different levels of division: according to the output results, it can be divided into sentiment intensity classification and sentiment polarity classification; according to the granularity of text processing, it can be divided into three research levels: word level, sentence level, and chapter level; according to the research method, it can be divided into unsupervised learning, semi-supervised learning, and supervised learning. Most traditional sentiment analysis algorithms use manually designed feature selection methods for feature extraction, such as maximum entropy models, naive Bayes, support vector machines, etc., but these methods are time-consuming, difficult to train, and have high labor costs. It is difficult to apply them to application scenarios with large data sets today.

[0006] Early sentiment analysis research focused on the positive, neutral or negative sentiment of sentences or passages, which is generally considered to be coarse-grained sentiment analysis. In recent years, fine-grained sentiment analysis has attracted the attention of researchers, and multi-granular analysis methods have also emerged. Refining the types of sentiment labels is a typical multi-granular sentiment analysis research. This type of research no longer simply scores sentiment as positive, neutral or negative, but uses more categories for classification. A more representative one is to add the degree of sentiment, changing the task from three categories to multiple categories, such as five categories of sentiment. Another type of multi-granular analysis method is to multi-granularize the sentiment object. This type of research is concerned with the sentiment directed at a specific object. It can be further divided into aspect-level sentiment analysis, entity-level sentiment analysis, entity-aspect-level sentiment analysis, etc. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-granularity takeaway user review sentiment analysis method and system for real review data of a takeaway platform, thereby overcoming the technical defects in sentiment analysis at the current stage, and comprising:

[0008] 1) Preprocess the original real review data of the food delivery platform until it can be used to train and test the neural network.

[0009] According to the method, step 1) comprises:

[0010] The original real review data of the food delivery platform is converted into simplified Chinese. That is, there are users who use traditional Chinese to make comments on the food delivery platform. Therefore, the comments of these users need to be converted into simplified Chinese first for subsequent processing.

[0011] The special emoticons in the original real review data of the food delivery platform are processed so that they are not segmented. In food delivery reviews, users often use symbolic emoticons or graphic emoticons to evaluate. In the food delivery platform review data, special emoticons include symbolic emoticons and graphic emoticons. It is necessary to avoid segmenting these data during data preprocessing.

[0012] The same comments in the original takeaway platform real comment data within a similar period of time are deleted. This operation is mainly to avoid unnecessary time consumption for model training due to a large number of comments with the same sentiment trend caused by buyers deliberately posting multiple rounds of bad reviews or stores deliberately posting a large number of good reviews.

[0013] When processing the original real review data of the food delivery platform, in addition to deduplication, it is also necessary to correct the review data with incorrect formats in the data, automatically change the incorrect format to the correct format, so that the final review data is more complete and effective. For some noise data, such as stop words, null values, special symbols, etc., these data also need to be deleted one by one.

[0014] Only after data preprocessing can the authenticity and reliability of research results be guaranteed. Therefore, data preprocessing is a prerequisite for ensuring that research conclusions are accurate and reliable.

[0015] 2) The original real review data of the food delivery platform is segmented. In the method of the present invention, a segmentation tool combining jieba and char is mainly used to segment the review data.

[0016] In the problem of Chinese sentiment analysis, compared with English texts that naturally contain spaces to divide phrases, the distance between characters and words in Chinese texts is not so obvious. Therefore, in the present invention, it is necessary to use words as the smallest unit for analyzing Chinese understanding, and divide Chinese texts into words or phrases according to the rules of Chinese grammar and context, so that each segment has independent semantics.

[0017] According to the method, step 2) comprises:

[0018] First, the original real review data of the food delivery platform is processed by the Jieba word segmentation tool for the first round of word segmentation, and then 10% of the segmented results are extracted for evaluation. The review data with inaccurate word segmentation is marked and removed from the word segmentation results of the Jieba word segmentation tool, and finally the first stage word segmentation result set M1 is formed.

[0019] For the review data that is inaccurately segmented by the jieba word segmentation tool, the char word segmentation tool is used to re-segment the words, and the results after segmentation form the second-stage word segmentation result set M2.

[0020] Then the two-stage word segmentation result sets M1 and M2 are merged to form the final word segmentation result M 终 .

[0021] 3) After the data preprocessing and word segmentation stages are completed, the basic model of the present invention is constructed using a LSTM-based text classification model.

[0022] According to the method, step 3) comprises:

[0023] Construct convolutional layers and maximum pooling layers, that is, divide the text into K regions according to sentences. Each region contains the words of the sentence. Extract features from each region and assign weights. Some regions do not contain emotional color, so the weights will become very small. Each word is converted into a vector using the word vector matrix obtained by word2vec training, that is, region i can be represented as Such a matrix is ​​then convolved with L convolution filters, and the convolution window length is ω, that is, after a window convolution of filter l, a mapping value yn will be obtained, and finally the vector

[0024] Vector y l Then use the maximum pooling layer for pooling to get a new vector.

[0025]

[0026] In the above text, r i is the word vector matrix obtained by word2vec training, w is the element in the word vector matrix obtained by word2vec training, W l ·x n:n+ω+1 is the element of the word vector matrix after the pooling operation, b l To correct the parameters.

[0027] LSTM encoder and linear decoder

[0028] Since the takeaway review text has temporal relationships and long-distance dependencies, we can use LSTM to encode the above-obtained regional vector as a sequence, and the final network hidden state can be regarded as the representation of the text x t Because the values ​​in both dimensions of the VA space are continuous values, a regression model is needed here, so a linear decoder is used for regression to finally get the predicted value. That is:

[0029] y=W d x t +b d

[0030] Where W d is the linear decoder slope, b d is the linear decoder intercept, where the LSTM network structure is as shown in the attached Figure 1 shown.

[0031] 4) After the initial construction of the model is completed, the basic model described in the present invention is optimized using the self-attention mechanism.

[0032] In the self-attention mechanism, the multi-head attention mechanism enables the model to pay attention to corresponding information from different subspaces. Since the self-attention mechanism often ignores the position factor of words in a sentence, it can explicitly capture the semantic relationship between the current word and all words in the sentence, while the multi-head attention mechanism maps the input sequence to different subspaces, which use the self-attention mechanism respectively, further enhancing the performance of the machine translation model.

[0033] According to the method, step 4) comprises:

[0034] This method uses the self-attention mechanism to process each word (that is, each element in the input sequence), such as x i When performing calculations, the self-attention mechanism can associate it with all the words in the sequence and calculate the semantic similarity between them. The benefit of this is that it can help explore the semantic relationship between all the words in the sequence, thereby encoding the words more accurately.

[0035] Each attention head is fed with an input sequence of n-tuples Perform the operation and then calculate a set of n-tuple output sequences For element z in the output sequence Z i , is the input element x i , x j After linear transformation and calculation of its weighted sum, we get:

[0036]

[0037] In the softmax function, linear transformation of input elements enhances the expressive power. The softmax score determines the size of the attention score expressed by each word at the current position. Here, the value vector V j Multiplying by the softmax score is to keep the value of the currently focused word intact and drown out irrelevant words. These weighted value vectors are then summed to get the self-attention mechanism output, which is sent to the feedforward neural network layer for further calculation. The softmax function is calculated as follows:

[0038]

[0039] Q, K, and V represent query, key, and value, respectively. They are abstract representations useful for calculating attention scores. k is the dimension of the key, divided by It is the scaled dot product, which can make the gradient more stable. The calculation methods of Q, K, and V are as follows:

[0040] Q i =xi WQ

[0041] K j =x j W K

[0042] V j =x j W V

[0043] W Q , W K , are matrices learned during the training process, which are the weight matrices of Q, K, and V. Each attention head has its own unique weight matrix. It is worth mentioning that, unlike the traditional attention mechanism, in the self-attention mechanism, in the same attention head, W Q =W K =W V , so for the same input element x i , Q i =K i =V i

[0044] The self-attention mechanism uses l attention heads, and the output z of all attention heads h All are merged and then linearly transformed to obtain the output of each sublayer. The multi-head attention mechanism expands the model's ability to focus on different positions and provides multiple representation subspaces for the attention layer. The multi-head attention mechanism provides multiple sets of Query, Key, and Value sets, which are randomly initialized and generated. After training, each set will be used for input embedding and then invested in different representation subspaces. The output calculation formula of the multi-head attention mechanism is as follows:

[0045]

[0046] represents the output vector of the ith attention head. The function of Concat(·) is to merge the output vectors of all attention heads. O is the weight matrix generated during the model training process. Finally, the multi-head attention mechanism combines the outputs of each attention head and then performs a linear transformation to obtain the final output.

[0047] The calculation process of the attention mechanism is shown in the attached Figure 2 shown.

[0048] 5) After the model is constructed and optimized, training of the takeaway review data involved in the present invention begins.

[0049] According to the method, step 5) comprises:

[0050] Regarding the processing of loss during training, in the method of the present invention, it is chosen not to adjust the training loss.

[0051] Regarding the sampling of the training data, in the method of the present invention, it is chosen not to perform oversampling or undersampling processing on the training data.

[0052] During the training phase, only the training results are adjusted according to the distribution of categories in the training data, which includes the following steps:

[0053] The first step is to bucket the training data according to the length of the text. The purpose of this operation is to be able to truncate long texts without affecting the overall training speed.

[0054] In the present invention, the principle of length bucketing is to cluster the comment data according to the text length, and divide it into long text and short text. A smaller batch_size is used for long text, and a larger batch_size is used for short text. Based on this operation, training work can be carried out well on all comment data.

[0055] The second is to adjust and optimize the learning rate of the training process. The learning rate is an important hyperparameter for machine learning. It controls the speed of adjusting the weights of the neural network based on the loss gradient. Most optimization algorithms involve it. The smaller the learning rate, the slower the loss gradient decreases and the longer the convergence time.

[0056] Therefore, in order to reach the minimum loss value faster and ensure that the converged loss value is the global optimal solution of the neural network, in the present invention, the triangular learning rate adjustment method is used to adjust the learning rate. The triangular learning rate adjustment method uses the simplest linear function to dynamically adjust the size of the learning rate, as shown in the attached figure. Figure 3 As shown, the difference between the minimum and maximum learning rate is halved after each epoch.

[0057] 6) During the model training process, select UnknownWords.

[0058] According to the method, step 6) comprises:

[0059] UNK is the abbreviation of Unknown Words, which often appears in machine translation tasks and text summarization tasks. The UNK problem occurs when a certain word is generated in some language models used. The language model is essentially a multi-classifier, which generates words under the current conditions by calculating the probability of each word in the vocabulary. In order to improve computing efficiency, only the N words with the highest frequency of occurrence are often selected as the vocabulary, and the other words are replaced by UNK, which causes UNK to appear when the language model is used. Among them, many UNKs may be some infrequent but meaningful words, such as organization names and place names.

[0060] In the present invention, starting from the second round, the original word is set to UNK with a random probability of 0 to 0.02, as shown in the attached figure. Figure 4 shown.

[0061] 7) After the Unknown Words are selected, the data is trained again using an end-to-end model, and the result of the model training will be used as the baseline result of the present invention.

[0062] According to the method, step 7) comprises:

[0063] Compared with deep learning, the process of traditional machine learning is often composed of multiple independent modules. For example, in a typical natural language processing problem, there are multiple independent steps including word segmentation, part-of-speech tagging, syntactic analysis, semantic analysis, etc. Each step is an independent task, and the quality of its results will affect the next step, thereby affecting the results of the entire training. This is non-end-to-end. In the training process of deep learning models, a predicted result will be obtained from the input end (input data) to the output end, and an error will be obtained when compared with the actual result. This error will be transmitted (back propagation) in each layer of the model, and the representation of each layer will be adjusted according to this error until the model converges or achieves the expected effect. All operations in the middle are included in the neural network and are no longer divided into multiple modules for processing. From the input of raw data to the output of the result, from the input end to the output end, the neural network in the middle is self-contained, which is end-to-end. Compared with the two, end-to-end learning eliminates the need for data annotation before each independent learning task is executed. In artificial intelligence problems, the cost of labeling samples is expensive and prone to errors.

[0064] The present invention adopts the following Figure 5 The end-to-end model shown is used to train the data, and during the training process, the model parameters are adjusted until the optimal training result is achieved.

[0065] 8) After the end-to-end model trains the data again, the BERT-MRC model is used to train the data.

[0066] The reason why the BERT-MRC model can achieve better performance than other models is its special data processing method. It adds description information of entity types before the input text. These descriptions of entity types serve as prior knowledge to improve the model extraction effect. Therefore, in scenarios with scarce data, the BERT-MRC model can obtain certain prior information by splicing the query in front of the input text, thereby improving performance.

[0067] According to the method, step 8) comprises:

[0068] Model input: Unlike other models, a lot of data preprocessing is required before using the BERT-MRC model. For example, there is an entity recognition dataset with the following three types of entities:

[0069] Name Character Name Place name Location Organization Name Name of government, enterprise, or other institution or organization

[0070] In the data preprocessing stage, the description of the entity type needs to be used as a query and spliced ​​to the front of each input sentence, and then the original annotations need to be modified in a targeted manner. For example: for the input text "Zhang San ate spicy hot pot with his friend Li Si at Zhang Liang Spicy Hot Pot in 2021", since there are three types of entities in the dataset, three pieces of data need to be constructed for this sentence. First, the description of the person's name is used as the query. This data is annotated with two entities of the person's name type (Zhang San and Li Si); then the description of the place name is used as the query, and there is no entity in this data; the data with the institution name as the query has an entity (Zhang Liang Spicy Hot Pot). It can be found that the BERT-MRC model increases the scale of the data and adds prior information through the query, but at the same time it also causes fewer entities in a piece of data, which will bring about label imbalance problems to a certain extent.

[0071] Model structure: After using BERT encoding to obtain word vectors, three classifiers are trained to predict the start and end position labels and a probability matrix. The start and end position sequences are sequences of the same length as the input text, corresponding to whether each character is the start and end position of an entity segment. The probability matrix represents the probability of whether each segment in the text is an entity segment. The above two label sequences and one matrix and three classifiers can calculate three losses in total during the training process. The total loss of the model is the sum of the above three losses. The specific structure and process of the model are as follows. Figure 6 Shown

[0072] 9) After the end-to-end model trains the data again, use the fastText model to train the data.

[0073] FastText is a word vector calculation and text classification tool that Facebook open-sourced in 2016. It is not very innovative in academics. However, its advantages are also very obvious. In text classification tasks, fastText (shallow network) can often achieve accuracy comparable to that of deep networks, but it is many orders of magnitude faster than deep networks in training time. On a standard multi-core CPU, it can train word vectors for a corpus of 1 billion words within 10 minutes, and can classify more than 500,000 sentences with more than 300,000 categories within 1 minute.

[0074] According to the method, step 9) comprises:

[0075] Use character-level n-gram vectors to represent sentences: word2vec treats each word in the corpus as an atom, and it generates a vector for each word. This ignores the morphological features within the word, such as: "apple" and "apples", "Daguan data" and "Daguan", in these two examples, the two words have more common characters, that is, their internal forms are similar, but in traditional word2vec, this word internal morphological information is lost because they are converted into different IDs.

[0076] To overcome this problem, fastText uses character-level n-grams to represent a word. For the word "apple", assuming that the value of n is 3, its trigrams are:

[0077] “<ap”,“app”,“ppl”,“ple”,“le> ”

[0078] Among them, < represents a prefix and > represents a suffix. Therefore, we can use these trigrams to represent the word "apple". Furthermore, we can use the vector superposition of these five trigrams to represent the word vector of "apple".

[0079] The above method of the present invention brings the following benefits:

[0080] A. The word vectors generated for low-frequency words will be better because their n-grams can be shared with other words.

[0081] B. For words outside the training vocabulary, their word vectors can still be constructed and their character-level n-gram vectors can be superimposed.

[0082] The model structure is as follows Figure 7 shown.

[0083] 10) After training the data using the fastText model, optimize the model and integrate multiple models.

[0084] According to the method, step 10) includes:

[0085] Model optimization: Based on the baseline model, this paper follows the common practice of reading comprehension and adds a SelfAttention layer (calculates the attention weight of the text to the text itself), and fuses the output after Attention with the original LSTM output using Gate (RNet) or Semantic Fusion (MnemonicReader), thereby improving the model training effect.

[0086] Model integration: In order to achieve better model diversity, we adopted word segmentation methods with multiple granularities. On the basis of the main model of Jieba word segmentation, we also introduced word segmentation methods with multiple granularities based on SentencePiece. SentencePiece word segmentation can lead to shorter sentence lengths, but the word segmentation errors are slightly more than Jieba, and it is easy to overfit. Therefore, we adopted the strategy of using only Finetune Char vectors and fixed word vectors to avoid overfitting. Word segmentation with multiple granularities combined with the Word+Char modeling method brings good model diversity.

[0087] In addition, the diversity of model dimensions comes from the differences in the RNet structure, the MnemonicReader structure, and the BERT model structure. When selecting the model, the optimal round model in the baseline model training was selected, and when integrating, a weighted integration method based on the effects of multiple aspects was adopted. Based on the above diversity strategy, multiple single models were integrated to achieve better results. The training model used in the end is as shown in the attached figure. Figure 8 shown.

[0088] Compared with the prior art, the present invention has the following technical advantages:

[0089] 1. For real takeaway review data, preprocess the data according to its specific characteristics before conducting sentiment analysis, including converting traditional Chinese characters into simplified Chinese characters, deleting overly repeated review data, and performing targeted processing on expression comments. This data preprocessing work can effectively reduce the errors caused by subsequent sentiment analysis.

[0090] 2. At present, many sentiment analysis methods can only perform single-scale recognition on takeout review data, and it is difficult to carry out multi-scale recognition on takeout review data. The method of the present invention can carry out multi-scale recognition on takeout data, thereby helping businesses to judge and analyze user emotions more accurately.

[0091] 3. The present invention integrates multiple models, and each model plays its own advantages, so that the emotion recognition results can be greatly improved after the model fusion.

[0092] 4. The present invention introduces an attention mechanism into the model. The introduction of this mechanism can help the model better discover the data features in the training data, so that the network model can better learn the correlation and emotional features in the data, and ultimately achieve the purpose of improving the prediction accuracy.

[0093] 5. The model takes into account a variety of factors during the training process, such as traditional and simplified Chinese, long and short texts, expression data, etc., and uses a variety of methods to improve or accelerate the training rate, which makes this method have a strong application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 This is the internal structure diagram of LSTM-Cell.

[0095] Figure 2 Schematic diagram of the attention mechanism calculation process.

[0096] Figure 3 Schematic diagram of triangular learning rate.

[0097] Figure 4 Select the schematic for UNK.

[0098] Figure 5 Schematic diagram of the end-to-end model structure.

[0099] Figure 6 Schematic diagram of the BERT-MRC model structure.

[0100] Figure 7 This is a schematic diagram of the fastText model structure.

[0101] Figure 8 Schematic diagram of the model structure after adding the attention mechanism. DETAILED DESCRIPTION

[0102] The method of the present invention is tested by conducting experiments on real takeaway data. The specific implementation method is as follows:

[0103] The invention obtained a total of 120,000 real data after annotation, of which the data set is divided into four parts: training, verification, test A and test B, and the verification data is 15,000. The evaluation objects in the data set are divided into two levels according to different granularities. The first level is the coarse-grained evaluation objects, such as the service, location and other elements involved in the comment text; the second level is the fine-grained emotional objects, such as the "service staff attitude", "queue waiting time" and other fine-grained elements in the "service" attribute. The specific division of the evaluation objects is shown in the following table.

[0104]

[0105] The sentiment tendency of each fine-grained element has four states: positive, neutral, negative, and not mentioned. The sentiment tendency is described using four values ​​[1, 0, -1, -2]. The sentiment tendency value and its meaning comparison table are as follows:

[0106] Sentiment tendency value 1 0 -1 -2 meaning Positive emotions Neutral Emotion Negative emotions Emotional orientation not mentioned

[0107] First, the data is preprocessed, including removing a large amount of duplicate data, converting traditional Chinese comments to simplified Chinese comments, and performing special processing on emoticon comments. For example, a traditional Chinese comment data is "[Jingwu Duck Head Hubei Restaurant] is located at No. 1054, west side of the first floor of the Universal Tower, 88259300, located in the downtown area, not easy to park", and after simplified Chinese conversion, it becomes: "[Jingwu Duck Head Hubei Restaurant] is located at No. 1054, west side of the first floor of the Universal Tower, 88259300, located in the downtown area, not easy to park". According to statistics, in the data set used in the present invention, a total of 4.61% (4839 / 10500) of the data are traditional Chinese comments. For example, a merchant had 28 identical comment data "very delicious" between 11:30-11:42 noon on November 12, 2019. After processing, only one data was retained. For example, for high-frequency Emoji expressions and character expressions^_^, it is necessary to ensure that they are not segmented.

[0108] After data division, the present invention has a total of 10 million unsupervised data, 105,000 labeled data that can be used for training, and 15,000 data that can be used for verification. During the training process, the training loss is not adjusted, and the training data is not oversampled / undersampled. Only the training results are adjusted accordingly according to the distribution of categories in the training data. The data is then bucketed by length. The purpose of this operation is to avoid truncating long texts without affecting the overall training speed. The specific method of this operation is to cluster the comment data according to the length of the text, using a small batch_size for long texts and a large batch_size for short texts. The batch_size used in the present invention is 23, 16, 8. In order to enable the model to better carry out learning tasks, this paper uses a triangular learning rate to adjust the learning process accordingly. At the same time, starting from the second round of training, the original word is randomly set to UNK with a probability of 0 to 0.02.

[0109] After completing all the above operations, the end-to-end model parameters are set accordingly and adjusted appropriately according to the training results until the optimal training results are obtained.

[0110] In order to achieve better training and prediction results, an attention mechanism module is added to the end-to-end network to better extract the data features in the training data. At the same time, the training is combined with the fastText model, BERT-MRC model and other models. The purpose of combining training with other models is to fully explore the various data features in the training data and supplement the deficiencies of the main model.

[0111] Finally, after each single model achieves good results, multiple models are deeply integrated to jointly carry out emotion recognition tasks.

[0112] In the present invention, a total of 6 single models are integrated to achieve the best effect. The losses corresponding to these 6 models are shown in the following table;

[0113] Model Loss Jieba.mreader.elmo.lstm 0.31265 Sp20w.rnet.lstm 0.31723 Sp10w.mreader.elmo.lstm 0.31556 Jieba.rnetv2.elmo.gru 0.32677 Sp20w.rnet.gru 0.32123 Char.bert 0.34357

[0114] After the training, the present invention was tested, and the final test results showed that the present invention achieved an accuracy rate of 95.6% on the training set and an accuracy rate of 91.6% on the test set.

Claims

1. A multi-granularity takeaway user review sentiment analysis method, characterized by: The method comprises the following steps: 1) Preprocessing the original real review data of the food delivery platform to achieve the purpose of training and testing the neural network; converting the original real review data of the food delivery platform into simplified Chinese, that is, there are users who use traditional Chinese to make comments on the food delivery platform; 2) Performing word segmentation processing on the original real review data of the food delivery platform, using a word segmentation tool combining jieba and char to perform word segmentation processing on the review data; First, the original real review data of the food delivery platform is processed by the first round of word segmentation using the Jieba word segmentation tool, and then 10% of the results after word segmentation are extracted for evaluation; the review data with inaccurate word segmentation are marked and removed from the word segmentation results of the Jieba word segmentation tool, and finally the first stage word segmentation result set M1 is formed; For the review data that is inaccurately segmented by the jieba segmentation tool, the char segmentation tool is used to re-segment the words, and the segmentation results form the second-stage segmentation result set M2; Then the two-stage word segmentation result sets M1 and M2 are merged to form the final word segmentation result M 终 ; 3) After the data preprocessing and word segmentation phase, the basic model is constructed using the LSTM-based text classification model; 4) After the initial construction of the model is completed, the basic model is optimized using the self-attention mechanism; In the self-attention mechanism, the multi-head attention mechanism enables the model to pay attention to corresponding information from different subspaces; 5) After the model is built and optimized, start training on the food delivery review data involved; For the loss processing during training, choose not to adjust the training loss; For the sampling of training data, choose not to oversample or undersample the training data; During the training phase, only the training results are adjusted according to the distribution of categories in the training data; 6) In the process of training the model, select UnknownWords; According to the method, step 6) comprises: Generate the words under the condition by calculating the probability of each word in the vocabulary appearing under the current condition; starting from the second round, set the original word to UNK with a random probability of 0 to 0.02; 7) After the UnknownWords are selected, the data is trained again using the end-to-end model, and the results of the model training will be used as the baseline results; the data is trained using the end-to-end model, and during the training process, the model parameters are adjusted until the training results are obtained; 8) After the end-to-end model trains the data again, the BERT-MRC model is used to train the data; Model input: Perform data preprocessing before using the BERT-MRC model; Model structure: After using BERT encoding to obtain word vectors, three classifiers are trained to predict the start tag, end position tag, and a probability matrix. The start and end position sequences are sequences of the same length as the input text, corresponding to whether each character is the start and end position of an entity segment. The probability matrix represents the probability of whether each segment in the text is an entity segment. The above-mentioned predicted start tag, end position tag, two label sequences, and a matrix of three classifiers calculate three losses in total during the training process. The total loss of the model is the sum of the three losses of the three classifiers in the matrix. 9) After the end-to-end model trains the data again, use the fastText model to train the data; According to the method, step 9) comprises: Use character-level n-gram vectors to represent sentences, and fastText uses character-level n-grams to represent a word; 10) After training the data using the fastText model, optimize the model and integrate multiple models; Model optimization: Based on the baseline model, we added a SelfAttention layer to imitate reading comprehension, and fused the output after Attention with the original LSTM output using Gate or Semantic Fusion to improve the model training effect; Model integration: adopts multiple granularity word segmentation methods and introduces multiple granularity word segmentation based on SentencePiece; When selecting the model, the optimal round model in the baseline model training was selected, and when integrating, the method of weighted integration according to multiple aspects of the effect was adopted.

2. A multi-granularity takeaway user review sentiment analysis method according to claim 1, characterized in that: According to the method, step 3) comprises: Construct convolutional layers and maximum pooling layers, that is, divide the text into K regions according to sentences, each region contains the vocabulary of the sentence, extract features from each region and assign weights; each word is converted into a vector using the word vector matrix obtained by word2vec training, that is, region i is represented as Such a matrix is ​​then convolved with L convolution filters, and the convolution window length is ω, that is, after a window convolution of filter l, a mapping value y will be obtained n , and finally get the vector Vector y l Then use the maximum pooling layer for pooling to get a new vector; r i is the word vector matrix obtained by word2vec training, w is the element in the word vector matrix obtained by word2vec training, W l ·x n:n+ω+1 is the element of the word vector matrix after the pooling operation, b l is the correction parameter; LSTM encoder and linear decoder Since the takeaway review text has temporal relationships and long-distance dependencies, LSTM is used to encode the above-obtained regional vector as a sequence, and the final network hidden state is regarded as the representation of the text x t ; Because the values ​​in both dimensions of the VA space are continuous values, a regression model is needed here for regression, so a linear decoder is used for regression to finally get the predicted value; that is: y=W d x t +b d Where W d is the linear decoder slope, b d is the linear decoder intercept.

3. A multi-granularity takeaway user review sentiment analysis method according to claim 1, characterized in that: According to the method, step 4) comprises: When processing each word using the self-attention mechanism, i When performing calculations, the self-attention mechanism enables it to associate with all the words in the sequence and calculate the semantic similarity between them; Each attention head is fed with an input sequence of n-tuples Perform the operation and then calculate a set of n-tuple output sequences For element z in the output sequence Z i , is the input element x i ,x j After linear transformation and calculation of its weighted sum, we get: In the softmax function, linear transformation of the input elements enhances the expressive power; the softmax score determines the size of the attention score expressed by each word at the current position; here the value vector V j Multiplying by the softmax score is to keep the value of the currently focused word intact and drown out irrelevant words; then summing these weighted value vectors to get the self-attention mechanism output, which will be sent to the feedforward neural network layer for further calculation; the softmax function is calculated as follows: Q, K, and V represent query, key, and value, respectively. They are abstract representations useful for calculating attention scores. k is the dimension of the key, divided by It is the scaled dot product, which makes the gradient more stable. The calculation methods of Q, K, and V are as follows: Q i =x i W Q K j =x j W K V j =x j W V are matrices learned during the training process, which are the weight matrices of Q, K, and V respectively; each attention head has its own unique weight matrix; it is worth mentioning that, unlike the traditional attention mechanism, in the self-attention mechanism, in the same attention head, W Q =W K =W V , so for the same input element x i , Q i =K i =V i The self-attention mechanism uses l attention heads, and the output z of all attention heads h are merged, and then linearly transformed to obtain the output of each sublayer; the multi-head attention mechanism expands the model's ability to focus on different positions, providing multiple representation subspaces for the attention layer. The multi-head attention mechanism provides multiple sets of Query, Key, and Value sets, which are randomly initialized and generated. After training, each set will be used for input embedding and then invested in different representation subspaces; the output calculation formula of the multi-head attention mechanism is as follows: represents the output vector of the ith attention head. The function of Concat(·) is to merge the output vectors of all attention heads. O It is the weight matrix generated during the model training process; finally, the multi-head attention mechanism merges the outputs of each attention head and then performs a linear transformation to obtain the final output.

Citation Information

Patent Citations

  • A user comment emotion analysis system and method based on attention convolution neural network

    CN109284506A

  • Paid poster comment detection system and method based on attention mechanism

    CN110874397A

  • Aspect-level user product comment emotion analysis method and system

    CN111274398A