A method and system for mongolian sentiment analysis fusing prompt and SRU

Abstract

A Mongolian sentiment analysis method fusing Prompt and SRU, Prompt is introduced in BERT model training, and is trained by using a data set; Mongolian is input into corresponding word embedding, segmented embedding and position embedding of the trained BERT model to generate a processed vector T; the vector T is input into SRU with built-in Attention for global feature extraction, and is input into CNN for local feature extraction; features extracted by the SRU and the CNN are fused through a neural network; the fused features are subjected to a max-pooling operation; a vector obtained by the max-pooling operation is input into a naive Bayes classifier for sentiment discrimination. The application also provides a corresponding Mongolian sentiment analysis system, and the application trains the BERT model by using Prompt, so that the pretreatment model is adapted to a downstream task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and relates to sentiment classification, in particular to a Mongolian sentiment analysis method fusing Prompt and SRU. BACKGROUND

[0002] Sentiment analysis, also known as opinion mining, is to analyze the emotional color of the text with emotional color, and to mine the sentiment contained therein. The emotional attitude is divided.

[0003] Sentiment analysis can be roughly divided into three categories according to the size of the text granularity: three levels of document-level sentiment analysis, sentence-level sentiment analysis and aspect-level sentiment analysis. The article and the sentence are analyzed as a data sample respectively in the document-level and sentence-level sentiment analysis, which belongs to the coarse-grained sentiment analysis task, while the aspect-level sentiment analysis belongs to the fine-grained task, which can analyze the different emotional polarity expressed by different aspects in the sentence. All granularity sentiment analysis tasks contain three types of classification results, namely "positive", "negative" or "neutral", which are usually represented by the array [-1, 0, 1] respectively. The focus of aspect-level sentiment analysis is on the extraction and learning of aspect information, especially when dealing with sentences containing multiple aspects. Different aspects may express opposite emotional polarity, and the sentiment modifier of a specific aspect may have a negative impact on other aspects, resulting in algorithm errors, so the model needs to learn the local context information related to the aspect.

[0004] There are several main methods in the field of sentiment analysis at present. First, sentiment analysis based on sentiment dictionary, which means that the existing sentiment dictionary is used to extract sentiment words from the text to be analyzed, and the sentiment tendency of the text is calculated. The classification result is very dependent on the integrity of the sentiment dictionary. With the development of technology, sentiment analysis based on machine learning appears, which means that the sentiment words are used as feature words to matrix the text, and machine learning related technology is used for classification. However, in some fields, the text is not labeled or the sentiment dictionary is not perfect, which requires a lot of cost, for example, for Mongolian, its sentiment dictionary is not perfect. The current mainstream method is sentiment analysis based on deep learning. Deep learning has a deep nonlinear network structure, which can learn the characteristics of data from low dimension to high dimension. In the field of natural language processing, it reflects the ability to learn the internal relationship and dependency of data samples, and forms an abstract and high-level representation, thereby effectively overcoming the constraints of traditional machine learning.

[0005] The mainstream BERT model and the like adopt a downstream task adaptation pre-training model manner, which has relatively more parameters, and BERT relies heavily on labeled data. In recent years, a pre-training language model can obtain high-confidence relationships between entities by predicting the words that are MASKed. However, there is a gap between pre-training and fine-tuning, which cannot fully utilize all advantages of the pre-training model. Traditional RNN has the problems of slow training speed, gradient explosion and the like, and some shortcomings need to be overcome in the application field of sentiment analysis. SUMMARY

[0006] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a Mongolian sentiment analysis method and system fusing Prompt and SRU, using Prompt to convert the pre-BERT training mode into a pre-training model adapted to a downstream task, so that the pre-training language model can play a greater effect, and the speed of SRU is obviously improved compared with RNN, and has a good improvement effect in solving gradient explosion.

[0007] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is:

[0008] A Mongolian sentiment analysis method fusing Prompt and SRU, comprising the following steps:

[0009] Step 1, introducing Prompt in BERT model training, and training by using a data set;

[0010] Step 2, inputting Mongolian into corresponding word embedding, segment embedding and position embedding of the trained BERT model to generate a processed vector T;

[0011] Step 3, inputting the vector T into an SRU with built-in Attention for global feature extraction, and inputting the vector T into a CNN for local feature extraction;

[0012] Step 4, fusing the features extracted by the SRU and the CNN through a neural network;

[0013] Step 5, performing a maximum pooling operation on the fused features;

[0014] Step 6, inputting the vector obtained by the maximum pooling operation into a Naive Bayes classifier for sentiment discrimination.

[0015] The present application also provides a Mongolian sentiment analysis system fusing Prompt and SRU, comprising:

[0016] The data preprocessing module adopts a Prompt-BERT module for data preprocessing, and is used for converting input Mongolian into a data processable vector T; the Prompt-BERT module is introduced in BERT model training;

[0017] The feature extraction module includes a global feature extraction unit and a local feature extraction unit, the global feature extraction unit adopts an Attention SRU, and is used for global feature extraction on the vector T, and the local feature extraction unit adopts a CNN, and is used for local feature extraction on the vector T;

[0018] The feature fusion module is used for feature fusion of the global feature and the local feature extracted by the feature extraction module, and performs a maximum pooling operation;

[0019] The sentiment analysis module adopts a Naive Bayes classifier, inputs the vector obtained through the maximum pooling operation, and realizes Mongolian sentiment classification.

[0020] Compared with the prior art, the Mongolian sentiment classification method has the beneficial effects that:

[0021] The BERT model is trained by using the Prompt, so that the preprocessing model is adapted to the downstream task, the preprocessing model can play a stronger advantage, the SRU model and the CNN model of Attention are inserted to extract and fuse features, the task of feature extraction is more comprehensive, finally, the Naive Bayes model has stable classification efficiency and is not sensitive to missing data, and the selection of this mode can well judge the sentiment of Mongolian. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The present application is a general flowchart.

[0023] Figure 2 The Prompt-BERT model is a Prompt-BERT model.

[0024] Figure 3 The Prompt flowchart is a Prompt flowchart.

[0025] Figure 4 The BERT model is a BERT model.

[0026] Figure 5 The Attention-SRU model is an Attention-SRU model.

[0027] Figure 6 The CNN model is a CNN model. DETAILED DESCRIPTION

[0028] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and examples.

[0029] As Figure 1As shown, the application is a Mongolian sentiment analysis method combining Prompt and SRU, including the following steps:

[0030] Step 1: Introduce Prompt in BERT model training, and train using data set.

[0031] As Figure 2 In the application, Prompt is introduced, data is input into Prompt, Prompt adds MASK label to the input data, and then input into BERT model, and BERT model is trained by predicting MASK label.

[0032] By combining Prompt with BERT, the training method of BERT model is changed. Prompt can give a hint to BERT model to help it better understand the task. Prompt changes the input of downstream task into the form of pre-training task, i.e. the form of MLM (Masked Language Model), and does not need to add a classifier for downstream task because the task form is suitable for the pre-training model itself, and only a simple mapping needs to be established. The purpose of Prompt is to better exploit the capabilities of pre-training language model, which is achieved by adding text at the input end.

[0033] In one embodiment, the specific steps can refer to Figure 3 , which is divided into four steps:

[0034] First, construct the Prompt template, define a template containing two slots to be filled in: [z] and [y], and slots: [z] and [y] refer to the operation of adding MASK label. For example, input: It means: I like this book very much. Template: It means: [z] in general, it is a [y] book, and then it is a Prompting operation, the sentence becomes: I like this book very much, in general, it is a [y] book. Mongolian is:

[0035] Second, for Prompt, the relationship between predicted word y and dictionary label label needs to be known, but it is impossible for y to correspond to any word, so a corresponding mapping function needs to be set to map the output word of BERT model to dictionary label label. As in the current example, according to the sentence meaning, the output label is assumed to be two, one is interesting One is boring

[0036] Third, the slots to be filled in: [y] are predicted by the BERT model, and the result (in the above example, the result is interesting ) is obtained through model training, and the result is brought into [y].

[0037] Fourth, the result Answer of [y] predicted is mapped back to the label of the original Prompt, and it is observed whether the result and the label attribute correspond to each other to compare whether the prediction is accurate.

[0038] Through the four steps, the original downstream task of BERT can be changed to adapt to the characteristics of the model, and a new BERT pre-training model can be generated.

[0039] In addition, since the Mongolian language has the shortcomings of insufficient corpus and large corpus noise. While using Prompt, other language pre-training models can be used, and the trained model can be directly applied to Mongolian data preprocessing through fine-tuning.

[0040] Step 2, input the Mongolian language into the corresponding word embedding, segment embedding and position embedding of the trained BERT model to generate the processed vector T.

[0041] As Figure 4 , the input of the BERT model is composed of three parts, one is the position embedding of the word, which is used to represent the relative position of the word in the sentence; two is the segment embedding, which divides the sentence into two or more segments; three is the word embedding, that is, the word vector representation. When the Mongolian text is input, the trained BERT pre-training model adds the corresponding TokenEmbeddings, Segment Embeddings and Position Embeddings to the text, and outputs the processed vector T.

[0042] Step 3, input the vector T into the SRU with built-in Attention for global feature extraction, and input it into the CNN for local feature extraction.

[0043] In this step, the vector T is subjected to SRU and CNN feature extraction, SRU performs global feature extraction on the Mongolian text, and the Attention in SRU feature extraction calculates the attention distribution of the relevant content before the SRU feature extraction. SRU is easy to ignore some local features in global feature extraction, so CNN is used to extract local features of the text. Since the Mongolian corpus has relatively large noise, using two different neural networks to extract features of the Mongolian language can effectively avoid the deviation caused by noise.

[0044] The SRU model with built-in Attention is as follows Figure 5As shown, the steps for global feature extraction using the same are as follows:

[0045] The vector T is input into the SRU with built-in Attention, and the attention distribution is calculated first. The following additive model is used to score the attention:

[0046] s(T i ,q)=V T tanh(Wt i +Uq)

[0047] Where s(T i ,q) represents the attention score, T i represents the vector at the i-th index position in the vector T, t i is an output vector of the decoder layer in Attention, q is the vector to be queried related to the task, is the output of the encoder layer in Attention, W, U and V are network parameters learned by the neural network;

[0048] After the calculation is completed, the SRU model is entered, and the SRU model expression formula is:

[0049]

[0050] f n =sigmoid(w f (y*)X n +b f

[0051] r n =sigmoid(w r (y*)X t +b r )

[0052]

[0053]

[0054]

[0055] Where, is the feature X t at time t after Attention processing, (y*) is the operator of the Attention layer, ⊙ represents Hadamard product, X n represents the input feature at time n, w, w f , w r are weight parameters that can be automatically learned in the model training process, b f , b r are bias quantities that can be automatically learned in the model training process; fn , r n denotes the forget gate and the reset gate, denotes the cell state and the original hidden state, h n is the hidden state output by the SRU, wherein f b is an attention guiding network, which can adaptively adjust the attention focus of the SRU neural network, so that the SRU neural network has stronger robustness.

[0056] The CNN model is shown in FIG. 2, and the steps of local feature extraction using the CNN model are as follows: Figure 6

[0057] First, the vector T is mapped to a p-dimensional vector by a mapping function f: T→V p , and is converted into a vector matrix with a length of L;

[0058] Then, the vectors in the vector matrix are convolved in the convolution layer. The convolution layer contains e neurons, and each neuron corresponds to a convolution kernel K∈V l*p for convolution operation to extract features from the vectors. It is assumed that O 1:L is a word embedding matrix with a length of L, and the feature generated by the jth neuron is:

[0059] Z j = ReLU(O 1:L *K j +b j )

[0060] where b j is a bias term, * is a convolution operation, and ReLU is a nonlinear activation function.

[0061] Under the action of the sliding window, the features Z1, Z2,..., Z j generated by the jth neuron are obtained. j max(Z1,Z2,...,Z j ), which is used to capture the feature with the maximum value; the final output of the convolution layer is the concatenation of the outputs of the e neurons, denoted as E, which is defined as:

[0062] E = [G1,G2,...,G k ,...,G e ]

[0063] Finally, E enters the fully connected layer to obtain the following output:

[0064] H = ReLU(CE + I)

[0065] where C is a vector weight matrix, C∈V e*s , and I is a bias term, I∈R s ​, S is the dimension of the fully connected layer output.

[0066] The CNN local feature extraction work is completed.

[0067] Step 4, the features extracted by SRU and CNN are fused by neural network.

[0068] This step uses the concat method of neural network for feature fusion, the formula is as follows:

[0069]

[0070] The input is double channel, and the input features are h i and H i , respectively. i , where c represents the number of vectors in each channel, the input feature H i of the second channel is spliced after the input feature h i of the first channel, K represents the vector fusion index, the first channel to K i , and the second channel feature is fused and spliced, the index position is from K i+c to K concat .

[0071] Step 5, the fused features are subjected to max pooling (MaxPooling) operation.

[0072] Concat will concatenate the vectors, which will expand the dimensions of the two tensors. Therefore, after feature fusion, the features at different positions are aggregated and counted by max pooling operation to prevent overfitting phenomenon, and the pooling result is:

[0073] Pool = Maxpooling (Z concat )

[0074] Step 6, input the vector obtained by max pooling operation into naive Bayes classifier for sentiment discrimination.

[0075] Naive Bayes classifier is a simple and powerful algorithm. Naive Bayes classification method is based on the assumption of independence and combines the probability classification method of Bayes theorem. By learning and summarizing the text features between input and output variables, the value of new data output variable is classified and predicted. First, the vector is divided into training set and test set according to certain proportion. First, the vector is all used as training set Q = {Q1, Q2,..., Q n}, the above vectors belong to different categories, positive or negative, category C = {C1, C2}, where C1 corresponds to positive emotional tendency, and C2 corresponds to negative emotional tendency. The prior probability of vector is calculated as:

[0076]

[0077] Calculate a certain vector Q k In the category C j The conditional probability estimation:

[0078]

[0079] Output category: by the test set each feature vector in the text belongs to the category, finally according to the maximum posterior probability classification:

[0080]

[0081] According to the above steps, the corresponding category of Mongolian can be obtained, and the final result of judging Mongolian is obtained.

[0082] The present application also provides a corresponding sentiment analysis system, comprising:

[0083] The data preprocessing module adopts the Prompt-BERT module for data preprocessing, which is used for converting the input Mongolian into a vector that can be processed, facilitating subsequent feature extraction. Specifically, first, the data is input into Prompt, and Prompt will construct a template. The function of the template is to convert the input text into a text with MSAK and the subsequent output text, so that the pre-training model can be trained and the MASK can be completed. Define a label dictionary, input the constructed sentence with the MASK label into BERT, so that BERT can predict the MASK label, predict the content of the MASK, check which label in the label corresponds to the predicted content, and then map the corresponding label back to the sentence with the MASK, so that the sentence is complete. Prompt adds the MASK operation according to the input format of the BERT model, so that BERT saves one step of operation and improves the efficiency of the pre-training model. Through the trained model, the input content is finally converted into the required vector format for subsequent use. In practical application, the Prompt-BERT model can be pre-trained in Chinese or English first, so as to overcome the problem of insufficient Mongolian corpus that may cause insufficient training of the pre-training model. Then, the trained model is fine-tuned for the preprocessing task of Mongolian.

[0084] The feature extraction module mainly uses SRU and CNN to realize feature extraction of data, wherein the Attention is added in the SRU, the Attention firstly performs preliminary attention calculation on a vector, and then is input into the SRU. The SRU also contains corresponding neural gate control information flow to prevent gradient explosion, and the SRU gate calculation only depends on the current input cycle, so that the model only has point-by-point multiplication calculation which depends on the previous time step, thereby enabling the network to be easily parallelized. The SRU has some deficiencies in local feature extraction when extracting global features, therefore, a double channel is adopted, and a CNN is additionally used to extract local features. Finally, the features extracted by the SRU and the CNN are fused, and a maximum pooling operation is performed to reduce the dimension of the fused feature vector.

[0085] The feature fusion module is used for fusing the global features and the local features extracted by the feature extraction module, and performing a maximum pooling operation.

[0086] The sentiment analysis module is used for realizing a Mongolian sentiment discrimination function. Specifically, the processed vector is input into a naive Bayes model, and the naive Bayes classifier finally discriminates the sentiment of the text.

Claims

1. A method for sentiment analysis of Mongolian language by fusing Prompt and SRU, characterized in that, Comprising the following steps: Step 1, introduce Prompt in BERT model training, train with data set; Step 2, input Mongolian into the corresponding word embedding, segment embedding and position embedding of the trained BERT model to generate processed vector T; Step 3, input vector T into SRU with built-in Attention for global feature extraction, and input into CNN for local feature extraction; Step 4, fuse the features extracted by SRU and CNN through neural network; Step 5, perform maximum pooling operation on the fused features; Step 6, input the vector obtained by maximum pooling operation into Naive Bayes classifier for sentiment discrimination.

2. The method of claim 1, wherein the Mongolian sentiment analysis method of fusing Prompt and SRU is characterized by, In step 1, the data is input into Prompt, Prompt adds MASK label to the input data, and then input into BERT model, BERT model is trained by predicting MASK label.

3. The method of claim 1 or 2, wherein the Mongolian sentiment analysis method of fusing Prompt and SRU is characterized in that, The step 1 comprises the following steps: First, construct a Prompt template, define a template containing two slots to be filled in: [z] and [y], the slots: [z] and [y] refer to the MASK label operation; Second, set a corresponding mapping function to map the words output by the BERT model to the dictionary label; Third, predict the slots to be filled in: [y] through the BERT model, get the result through model training, and bring the result into [y]; Fourth, map the result Answer of [y] back to the original label of Prompt and observe whether the result and the label attribute correspond to each other to compare whether the prediction is accurate.

4. The method of claim 1, wherein the sentiment analysis of Mongolian language using Prompt and SRU is characterized by, The step 3 is as follows: Vector T is input into SRU with built-in Attention, first calculate the attention distribution through Attention, and use the following additive model to score the attention: s(T i , q) = V t tanh(Wt i + Uq) where s(T i q) represents an attention score, T i represents a vector at an i-th index position in the vector T, t i is an output vector of a decoder layer in the Attention, q is a vector to be queried in relation to a task, is an output of an encoder layer in the Attention, and W, U, and V are network parameters learned by a neural network. After the calculation is completed, enter the SRU network, and the SRU model expression formula is: f n = sigmoid(w f (y*)X n +b f r n = sigmoid(w r (y*)X t + b r ) wherein, is the feature X at time t t The vector after Attention processing, (y*) is the operator of the Attention layer, ⊙ represents Hadamard product, X n represents the input feature at time n, w, w f , w r is a weight parameter that can be automatically learned in the model training process, b f , b r is a bias that can be automatically learned in the model training process; f n , r n represent the forget gate and the reset gate, c n , represent the cell state and the original hidden state, h n is the hidden state output through the SRU, wherein f b is an attention-guided network that can adaptively adjust the attention focus of the SRU neural network, making it more robust.

5. The method of claim 1, wherein the sentiment analysis of Mongolian language using fusion of Prompt and SRU is characterized by, The steps of local feature extraction are as follows: First, the vector T is mapped to a p-dimensional vector by a mapping function f: T→V p and converted to a vector matrix of length L. Then, convolution operations are performed on the vectors in the vector matrix in the convolutional layer, and under the action of the sliding window, the features Z1, Z2, ..., Z generated by the j-th neuron are obtained. j The obtained features are subjected to max pooling, denoted as G. j =max(Z1,Z2,…,Z) j The final output of a convolutional layer is a structure E, which is a concatenation of the outputs of e neurons, and is defined as follows: E = [G1, G2,..., G j ,...,G e ] Finally, E enters the full connection layer to obtain the following output: H=ReLU(CE+I) where C is a vector weight matrix, C e V e*s , I is a bias term, I e R s , and S is the dimension of the fully connected layer output.

6. The method of claim 1, wherein the sentiment analysis of Mongolian language using fusion of Prompt and SRU is characterized by, The step 4 uses the concat method of neural network to fuse features, and the formula is as follows: Feature fusion is performed by Concat, the input is double channel, and the input features are h i and H i , where c represents the number of vectors in each channel, the input feature H i of the second channel is spliced after the input feature h i of the first channel, K represents the vector fusion index, the first channel to K i , and the feature of the second channel is spliced, the index position from K i to K i+c .

7. The method of claim 1, wherein the sentiment analysis of Mongolian language using fusion of Prompt and SRU is characterized by, After feature fusion, different position features are aggregated and counted through maximum pooling operation to prevent overfitting phenomenon, and the pooling result is: Pool = Maxpooling (Z concat ).

8. A Mongolian sentiment analysis system combining Prompt and SRU, comprising: A data preprocessing module using Prompt-BERT module for data preprocessing, for converting input Mongolian into data processable vector T; the Prompt-BERT module is introduced in BERT model training; A feature extraction module including a global feature extraction unit and a local feature extraction unit, the global feature extraction unit uses SRU with Attention for global feature extraction of the vector T, and the local feature extraction unit uses CNN for local feature extraction of the vector T; The feature fusion module is configured to fuse the global feature and the local feature extracted by the feature extraction module and perform a max-pooling operation. The sentiment analysis module is configured to input the vector obtained by the max-pooling operation into a naive Bayes classifier to realize sentiment classification of the Mongolian language.

Images