A dual-channel and attention-weighted method for aspect-level sentiment analysis of Mongolian language

Abstract

A Mongolian aspect-level sentiment analysis method with double channels and attention weighting extracts text information and aspect words of a Mongolian sentence to be analyzed; adopts a double-channel embedding form, embeds text information in a first channel to obtain a text vector E, and embeds an aspect word in the text information in a second channel to obtain an aspect word vector A; adopts an attention mechanism to calculate a word vector weight K of each word in the Mongolian sentence t ; adopts the attention mechanism to calculate an attention weight K a between the aspect word and each word in the Mongolian sentence; fuses the weights of K t and K a , weights the output result, obtains a weighted sentence representation q, and inputs a sentiment model to perform sentiment tendency discrimination. The present application can accurately capture Mongolian semantic information, the effective attention mechanism can help the model focus on the part related to sentiment analysis, and the multi-source information fusion can fully utilize the key information in the text, thereby improving the comprehensiveness and accuracy of sentiment analysis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, relates to sentiment analysis, especially sentiment analysis of Mongolian, and particularly relates to a Mongolian aspect-level sentiment analysis method with double channels and attention weighting. BACKGROUND

[0002] Sentiment analysis is one of the key branches of natural language processing in the field of artificial intelligence. The common scenario of text sentiment analysis is the research on the identification and analysis of the emotions expressed by users on the Internet. According to different text granularity research, sentiment analysis can be divided into document-level, sentence-level and aspect-level. With the continuous iteration and development of technology, the research focus of sentiment analysis gradually shifts from coarse-grained sentence-level sentiment analysis to more fine-grained aspect-level sentiment analysis. Aspect-level sentiment analysis refers to the process of subdividing sentiment analysis into different aspects or topics for analysis when performing sentiment analysis on text. Traditional coarse-grained sentiment analysis only focuses on the overall sentiment of a sentence, while aspect-level sentiment analysis can more specifically understand the emotional sentiment of different aspects or topics in a sentence, thereby achieving more detailed sentiment classification.

[0003] Aspect-level sentiment analysis focuses on mining fine-grained sentiment information, and aspect-level sentiment analysis generally includes three categories: dictionary-based method, traditional machine learning-based method and deep learning-based method.

[0004] Dictionary-based sentiment analysis solves the sentiment analysis task based on a sentiment dictionary with annotated sentiment polarity. After segmenting the input sentence, the words are compared and matched with the dictionary to obtain the sentiment polarity of the text. Compared with traditional manual operation, dictionary-based sentiment analysis can save certain manpower and resources.

[0005] The most commonly used machine learning-based sentiment analysis is supervised machine learning method. Its principle is to use manual methods to annotate the sentiment of the text. After constructing the feature information through manual means, a sentiment classifier is constructed using machine learning technology. Finally, the classifier is trained to achieve sentiment analysis discrimination. Common machine learning discrimination methods include Maximum entropy (ME), Support Vector Machine (SVM), Navie Bayes (NB) and other algorithms.

[0006] The related technology based on deep learning is relatively mature in various fields, and has made great progress in text sentiment analysis task. Deep learning network model can learn text feature information independently, which can save manpower and material resources. Commonly used neural network models include convolutional neural network (CNN), recurrent neural network (RNN) and the like. Compared with sentiment dictionary based and traditional machine learning based sentiment analysis, the neural network in deep learning can not only learn text features actively, but also retain the information of words in the text, more comprehensively extract the semantic information of the corresponding words, and thus realize more accurate sentiment analysis.

[0007] Aspect-level sentiment analysis has achieved certain results in a single field, but still has some defects. First, there is a certain limitation in feature extraction of text and aspect word information. Second, the dependency relationship between specific aspect words and context information is easily ignored, so that the text information related to the aspect words cannot be fully utilized for sentiment tendency judgment. Finally, the existing aspect-level sentiment analysis is only for mainstream languages. For some small languages such as Mongolian, the traditional method may have limitations in extracting text features, and it is difficult to fully capture the complex semantic information in Mongolian sentences, resulting in low accuracy of sentiment analysis. This may be due to the fact that the existing method ignores the dependency relationship between specific aspect words and context information, and cannot fully utilize the text information related to the aspect words for sentiment tendency judgment, resulting in inaccurate sentiment analysis results. Moreover, since Mongolian is an agglutinative language, traditional aspect-level sentiment analysis methods are mainly for non-agglutinative languages, and cannot accurately distinguish the sentiment information of different aspect words, which easily leads to confusion of sentiment analysis results, so that the existing method cannot be directly applied. Mechanically transplanting the existing method, it is found that the accuracy is very low in Mongolian sentiment analysis experiment. SUMMARY

[0008] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a dual-channel and attention-weighted Mongolian aspect-level sentiment analysis method, the main feature of which is to use a dual-channel embedding method in the input layer, and to strengthen the extraction of local features of the text, while introducing an attention mechanism to obtain the dependency relationship between the given aspect word and the context information, fully considering the influence of different aspect words on the text semantics, so as to improve the effect of Mongolian aspect-level sentiment analysis.

[0009] In order to achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0010] A dual-channel and attention-weighted Mongolian aspect-level sentiment analysis method, comprising the following steps:

[0011] Step 1, extract the text information and aspect words of the Mongolian sentence to be analyzed;

[0012] Step 2, adopt a double-channel embedding form, the first channel embeds the text information to obtain a text vector E, and the second channel embeds a certain aspect word in the text information to obtain an aspect word vector A;

[0013] Step 3, obtain a global feature vector h according to the text vector E;

[0014] Step 4, according to the global feature vector h, adopt an attention mechanism to calculate the word vector weight K of each word in the Mongolian sentence t ;

[0015] Step 5, according to the global feature vector h and the aspect word vector A, adopt an attention mechanism to calculate the attention weight K a between the aspect word and each word in the Mongolian sentence

[0016] Step 6, weight fusion is performed on the attention weights K t and K a , the output result is weighted to obtain a weighted sentence representation q, and a sentiment model is input to perform sentiment orientation discrimination.

[0017] Compared with the prior art, the Mongolian semantic information can be accurately captured by RoBERTa and GCNN, the effective attention mechanism helps the model to focus on the part related to sentiment analysis, and the multi-source information fusion can fully utilize various key information in the text, thereby improving the comprehensiveness and accuracy of sentiment analysis. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is the overall flowchart of the present application.

[0019] Figure 2 is the double-channel embedding flowchart of the present application.

[0020] Figure 3 is a GCNN schematic diagram.

[0021] Figure 4 is a text attention mechanism schematic diagram.

[0022] Figure 5 is an aspect word attention mechanism schematic diagram.

[0023] Figure 6 is a weight fusion layer schematic diagram. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and examples.

[0025] As Figure 1As shown, the application is a kind of dual-channel and attention weighted Mongolian aspect level sentiment analysis method, comprising the following steps:

[0026] Step 1, extract the text information and aspect words of the Mongolian sentence to be analyzed.

[0027] Step 2, adopt a dual-channel embedding form, the first channel embeds the text information to obtain a text vector E, and the second channel embeds a certain aspect word in the text information to obtain an aspect word vector A.

[0028] In the application, the first channel and the second channel are embedded by the RoBERTa model. That is, the text information and the aspect word are embedded by the RoBERTa pre-training model.

[0029] Therefore, the application needs to construct a RoBERTa pre-training model to process the text form in step 1, that is, to convert the text into "[CLS] Mongolian text [SEP]", and to construct the aspect word into the form of "[CLS] aspect word [SEP]".

[0030] The dual-channel embedding form of the application embeds the text information through the RoBERTa model in one channel, and embeds a certain aspect word in the text information in the other channel, such as Figure 2 As shown, the left channel embeds the text information "hotel service is very good", and the constructed style is , which means "hotel service is very good". The right channel embeds the aspect word "service", and the constructed style is which means "service" and is used for sentiment judgment of the aspect "service". By adopting dual channels, the model can focus on the aspect word, and the left channel obtains a text word vector embedding:

[0031] E = RoBERTa (text)

[0032] The right channel obtains an aspect word vector embedding:

[0033] A = RoBERTa (aspect word)

[0034] Step 3, obtain a global feature vector h according to the text vector E.

[0035] In the application, for the text vector E, a GCNN module is introduced to extract local features, so as to form the global features of the text with the embedding of the first channel RoBERTa model. The GCNN is a module composed of multiple convolution layers, as shown in Figure 3 After the text vector E obtained by the word embedding is extracted by the GCNN, a feature vector is obtained:​

[0036]

[0037] Where E i This represents the i-th word in the text vector E, which is the vector representation obtained by transforming the i-th word in the original Mongolian sentence. This represents the local features of the i-th word in the text vector E extracted by the j-th convolutional layer in GCNN.

[0038] Filter out the most obvious feature h from the feature vector i :

[0039]

[0040] Specifically, the selection can be achieved through max pooling layers, where n is the number of convolutional layers in GCNN; n is a hyperparameter, and the number of layers is determined based on the actual results.

[0041] The most prominent feature h corresponding to convolutional kernels of different sizes in the same hidden layer of RoBERTa is... i By concatenating the elements, a vector representation of the Mongolian sentence is formed, namely the global feature vector h:

[0042] h=(h 1 h 2 , ..., h c )

[0043] c represents the number of words in a Mongolian sentence.

[0044] In this step, the text vector E extracts local features for each word using GCNN, and the extracted features of each word are concatenated to form the global feature vector h of this invention.

[0045] Step 4: Based on the global feature vector h, calculate the word vector weight K of each word in the Mongolian sentence using the Attention mechanism. t .

[0046] like Figure 4 As shown, for text information, based on the calculation method of the self-attention mechanism, the hidden layer of the global feature vector h is first represented as u in the single-layer perceptron of the attention layer. That is, u is the implicit expression obtained after the global feature vector h is activated by the fully connected layer of the single-layer perceptron of the attention layer, which is also the implicit expression obtained after the text is activated by this layer. In order to calculate the importance of different words in the sentence, the implicit expression u and a randomly generated context information u are used. w The similarity is used to represent the similarity, and finally the attention weight K is obtained through the softmax mechanism. t The calculation steps are as follows:

[0047] u = tanh (wh + b)

[0048] K t = softmax (u T u w )

[0049] where tanh is a nonlinear activation function, w, u w and b are parameter matrices generated by the attention layer, and b is a bias.

[0050] Step 5, according to the global feature vector h and aspect word vector A, the attention mechanism is used to calculate the attention weight K a between aspect words and each word in the Mongolian sentence.

[0051] As Figure 5 shown, the global feature vector h and aspect word vector A of the sentence are found through the attention mechanism from the sentence and the most relevant words to the aspect word, and the calculation formula is as follows:

[0052]

[0053] where K a is the attention mechanism weight obtained by the softmax mechanism, and d is the dimension of the word vector.

[0054] Step 6, the attention weights K t and K a are fused, the output result is weighted, and the weighted sentence representation q is obtained, which is input into the sentiment model for sentiment orientation discrimination.

[0055] In this step, the Bi-LSTM is used for weight fusion, and the output result is weighted. As Figure 6 shown, in the weight fusion layer, the Bi-LSTM is used to fuse the word vector weight calculated by the text and the word weight combined with the aspect word, and the formula is as follows:

[0056]

[0057] q = K total *h

[0058] where K total is the comprehensive weight after weight fusion, denotes dot product, and q is the weighted sentence representation.

[0059] The emotional model adopted by the present application is a naive Bayes emotional model, and the vector obtained after weighting, i.e., the weighted sentence representation q, is input into the naive Bayes classifier model according to the size of a batch to perform emotional discrimination. The model divides the batch vector into a training set and a test set according to a proportion, assumes that the sentiment analysis task is a binary classification, and a batch vector is represented as Q={q1, q2,..., q m}, and the emotional category C={C1, C2}, wherein C1 corresponds to a positive emotional tendency, and C2 corresponds to a negative emotional tendency. m q is the mth Mongolian sentence in the batch, and the prior probability of the vector is calculated.

[0060]

[0061] The conditional probability of a single vector in the batch in the category C j is calculated.

[0062]

[0063] After obtaining the trained naive Bayes classifier, the polarity of each sample in the test set is judged.

[0064]

[0065] According to the above steps, the corresponding category of Mongolian can be obtained, and thus the final result of judging Mongolian is obtained.

[0066] The present application also provides a corresponding double-channel and attention-weighted Mongolian aspect-level sentiment analysis system, which comprises:

[0067] A data preprocessing module, when a Mongolian sentence is input into the system, the system will add [CLS] and [SEP] marks before and after the Mongolian sentence and the aspect word, respectively, to construct a sentence form required for subsequent tasks, and then the constructed sentence and aspect word will be sent into the two channels of the RoBERTa module, which will convert the Mongolian into a corresponding vector to achieve the effect of preprocessing. The DCNN module will further process the sentence to capture global features after the RoBERTa processing is completed. The DCNN module takes a convolutional neural network as the theme and has a unique advantage in extracting local features.

[0068] After data preprocessing, the aspect word and the Mongolian sentence will be converted into a corresponding vector representation, and then the weight weighting module is used. First, the Attention mechanism is added to the Mongolian sentence channel to determine the importance of the words in the sentence. Meanwhile, in the other channel, the vectorized sentence representation and the aspect word are jointly passed through the Attention mechanism. In this channel, the Attention mechanism will determine which words in the sentence are more important to the aspect word according to the aspect word.

[0069] Two channels respectively complete the key words in the Mongolian sentence and the important words of the opposite aspect words, and each word is completed after weighting, and is sent to the bidirectional long short-term memory network for weight fusion, and after fusion, the key points in the sentence and the key points of the opposite aspect words are considered, and the final weighted result is input, and then the vectorized Mongolian sentence and the weighted result are combined, so that the subsequent classification task model can find the Mongolian key point and improve the classification accuracy.

[0070] The processed text vector is sent to the Naive Bayes classifier, and the Naive Bayes classifier can obtain the sentiment polarity of each sentence through training to complete the sentiment analysis task.

[0071] The principle of the application comprises:

[0072] In the preprocessing process of data, the RoBERTa pre-training model is used for word embedding of Mongolian sentences and aspect words, so that more comprehensive Mongolian word embedding expression can be obtained. The advantage of the double channel is that it can be more targeted for different aspect words, so that the text "the food in this restaurant is very good, but the service attitude is very poor" can more clearly show the emotion of the aspect.

[0073] After obtaining the word embedding matrix, the application adopts the Gated Convolutinonal Neural Networks (GCNN) to extract the global features of the text part, the GCNN is a neural network composed of a convolutional neural network and a gated linear unit, the convolutional neural network is used to extract local context features, the gated linear unit can control the information transmission between network levels, so that useful information can pass through the network, and useless information can be suppressed or ignored. After the word embedding matrix is continuously learned by the GCNN network, the global semantic representation is obtained.

[0074] The application adopts the attention mechanism to calculate the attention size of each word in the text channel, and for the aspect word channel, the attention mechanism is used to give different weights to different words related to the aspect word, and the dependency relationship between the given aspect word and each word information in the text is obtained through the attention mechanism, and the sentiment words related to the aspect word are given higher weights.

[0075] After obtaining the weight, the weight is added by a bidirectional long short-term memory network (Bi-dirEctional Long Short-TErm MEmory, Bi-LSTM) layer to obtain a comprehensive weight containing text information and aspect word information. Finally, the sentiment tendency is obtained by aspect-level sentiment analysis of the Mongolian sentence through a naive Bayes classifier. The naive Bayes classifier is a simple and powerful algorithm, which is based on the premise of independence assumption and combines the probability classification method of Bayes theorem. By learning and summarizing the text features between input and output variables, the value of the output variable of new data is classified and predicted. First, a batch of vectors is divided into a training set and a test set according to a certain proportion. The vectors in the training set are trained, and then the prior probability of the vector is calculated. The conditional probability of the vector belonging to different polarity categories is calculated. The trained naive Bayes classifier will calculate the score of each vector in the test set to obtain the probability of belonging to different categories. Finally, the category with the highest score of each sentence is taken as the final sentiment category.

[0076] In one embodiment, the Mongolian aspect-level sentiment analysis dataset is the SemEvalABSA Datasets dataset after machine translation. The dataset includes the overall sentence text and the aspect word in the sentence and the sentiment label of the aspect word. For the binary sentiment analysis task, the negative label is usually coded as 0, and the positive label is coded as 1. In the multi-classification task, different sentiment tendencies are coded according to a certain threshold interval.

[0077] In one embodiment, the sentence in step 1 is sent into the pre-trained model RoBERTa according to the set batch size. The pre-trained model will change the sentence into the target data format [CLS] text containing aspect word [SEP] and [CLS] aspect word [SEP] two parts, for example: the input text is [CLS] [SEP] and aspect word [CLS] [SEP]. Among them means "the food in this restaurant is very delicious", means "food".

[0078] In one embodiment, according to step 2, the pre-trained model RoBERTa will find the position of the words in the text according to the pre-trained word table and encode it as [101, 9876, 5432, 8765, 101, 102], and then map it to the word embedding vector of each word. The left channel pre-training model will convert all word embeddings into a word embedding vector matrix E of the text in a certain way, E is a two-dimensional matrix, the matrix dimension is [seq_len, embed_size], where seq_len is the number of words in the sentence, in the example, seq_len length is 6, embed_size is 768, the dimension is [6, 768] and the word embedding of the right aspect word channel is A, the matrix dimension is [1, 768].

[0079] In one embodiment, according to step 3, after encoding by the pre-trained model, the GCNN module in the left channel extracts local features from the encoded word embedding matrix. The formula is:

[0080]

[0081] The GCNN is composed of multiple convolution layers, and the number of convolution layers can be regarded as a hyperparameter, which can be increased or decreased according to experiments. In this example, 3 convolution layers are used. When the text passes through the first layer of feature extraction:

[0082]

[0083] Similarly, the last two layers are:

[0084]

[0085]

[0086] The features extracted by each layer are then passed through a max pooling layer to obtain the most obvious features of each word vector in each convolution layer and perform feature fusion:

[0087]

[0088] Similarly, the remaining words in the sentence are also extracted locally.

[0089] When each word after local feature extraction is spliced to form a word vector matrix of the sentence.

[0090] h = (h 1 , h 2 ,..., h 6 )

[0091] In one embodiment, according to step 4, the processed text vector h is calculated by the Attention layer to calculate the word information weight in the text. When h enters the Attention layer, it is first input to a single-layer perceptron, which will activate the text vector h through an activation function to obtain a hidden representation h for subsequent weight calculation.

[0092] u = tanh(wh + b)

[0093] where w is a parameter matrix of the single-layer perceptron layer, and b is a bias.

[0094] The Attention will calculate the attention weight K according to the hidden representation u t :

[0095] K t = softmax(u T u w )

[0096] where u w is a parameter matrix.

[0097] In one embodiment, the above attention weight technique does not involve aspect words, but the Attention attention mechanism focuses on the key points of the text. According to step 5, the core of the Attention mechanism is set, that is, the aspect word, and the importance of the remaining words in the sentence to the aspect word is calculated according to the aspect word, and the formula is as follows:

[0098]

[0099] where K a is the attention mechanism weight obtained by the softmax mechanism, and d is the word vector dimension.

[0100] In one embodiment, according to step 6, when we obtain the weight of the importance of the text and the importance weight with the aspect word as the core, the Bi-LSTM layer is used for fusion. The Bi-LSTM adopts the form of dot product for weight fusion:

[0101]

[0102] After obtaining the fused weight, the weighted sentence representation q can be calculated:

[0103] q = K total *h

[0104] In one embodiment, according to step 6, the processed weighted sentence representation q is subjected to sentiment discrimination by Naive Bayes. Generally, we will output one batch of text at a time according to the batch size, so Q represents a batch:

[0105] Q = {q1, q2,..., q n}

[0106] For binary classification task, sentiment class {C1, C2}, where C1 corresponds to positive sentiment tendency, C2 corresponds to negative sentiment tendency, i.e. C1 = 1, C2 = 0, sentiment class is {1, 0}. The prior probability of the vector in the positive or negative field is calculated, and the prior probability of positive tendency is:

[0107]

[0108] The prior probability of negative:

[0109]

[0110] i represents a batch of samples.

[0111] And the conditional probability formula of a single text in a batch in a certain class is:

[0112]

[0113] j = 1 or j = 2.

[0114] 70% of the samples in the data set are used as the training set, and the remaining 30% of the samples are used as the test set. The naive Bayes classifier is trained by the training set, and the polarity of each sample in the test set is judged:

[0115]

[0116] When the probability is greater than 0.5, it is considered as a positive sample, less than 0.5 as a negative sample, and equal to 0.5 as a result.

Claims

1. A dual-channel and attention-weighted Mongolian aspect-level sentiment analysis method, characterized in that, Comprising the following steps: Step 1, extract the text information and aspect words of the Mongolian sentence to be analyzed; Step 2, using a double-channel embedding form, the first channel and the second channel are embedded through the RoBERTa model, the first channel embeds the text information to obtain a text vector E, and the second channel embeds an aspect word in the text information to obtain an aspect word vector A ; Step 3, obtain the global feature vector h according to the text vector E; wherein: Local features are extracted through the GCNN module, and the embedding of the first channel constitutes the global features of the text; Wherein, GCNN extracts features from the text vector E to obtain the extracted feature vector: wherein represents the i-th word in the text vector E, i.e. the vector expression obtained by converting the i-th word in the original Mongolian sentence; represents the local feature about the i-th word in the text vector E extracted by the j-th convolution layer in the GCNN; screening out the most significant features in the feature vector : n is the number of convolution layers of GCNN; The most obvious features corresponding to the convolution kernels of different sizes in the same hidden layer of RoBERTa The spliced vector representation of the Mongolian sentence, i.e. the global feature vector h, is obtained: c is the number of words in the Mongolian sentence; Step 4, according to the global feature vector h, the word vector weight of each word in the Mongolian sentence is calculated using an attention mechanism ; Step 5, calculating the attention weight of each word in the Mongolian sentence and the aspect word using the attention mechanism according to the global feature vector h and the aspect word vector A , adopting the attention mechanism to calculate the attention weight of the dependency relationship between each word in the aspect word and the Mongolian sentence ; Step 6, weight fusion is performed on the attention weights and the output results are weighted to obtain a weighted sentence representation , which is input into a sentiment model for sentiment orientation determination.

2. The dual channel and attention weighted Mongolian aspect-level sentiment analysis method according to claim 1, characterized in that, The step 4, The expression of the step 4 is as follows: wherein, is the implicit expression of the global feature vector h after being activated by a single-layer perceptron fully connected layer of the attention layer, , is a randomly generated context information, and is a parameter matrix generated by the attention layer, and tanh is a nonlinear activation function, is a bias.

3. The dual channel and attention weighted Mongolian aspect-level sentiment analysis method according to claim 2, characterized in that, The step 5, The expression of the step 5 is as follows: wherein, d is the word vector dimension.

4. The dual channel and attention weighted Mongolian aspect-level sentiment analysis method according to claim 3, characterized in that, The step 6, weight fusion is carried out through Bi-LSTM, and the formula is as follows: wherein is the integrated weight after weight fusion, denotes the dot product.

5. The dual channel and attention weighted Mongolian aspect-level sentiment analysis method of claim 1, wherein, The sentiment model is a naive Bayes sentiment model, and the sentiment model will be described below. According to the size of the batch, the naive Bayes sentiment model is input, the model divides the batch vector into a training set and a test set according to the proportion, the sentiment analysis task is binary classification, and the vector of one batch is represented as The sentiment category is , wherein corresponding to a positive emotional tendency, corresponding to a negative emotional tendency, is the i-th Mongolian sentence in the batch. m ​ Calculate the prior probability of the vector: Calculate the category of a single vector within a batch. The conditional probability in is: After obtaining the trained Naive Bayes classifier, the polarity of each sample in the test set is judged: The corresponding category of Mongolian is obtained.

Images