Method, device, processor and storage medium for implementing fine-grained sentiment analysis based on Transformer

Abstract

The present invention relates to a method for implementing fine-grained sentiment analysis based on Transformer, wherein the method comprises the following steps: (1) obtaining data sets of two fields, Laptop and Restaurant, of Sem‑Eval 2014 Task 4 for model verification; (2) performing data analysis and conversion processing on the corresponding data sets obtained; (3) constructing a sentiment analysis model based on the main text, subject and subject sentiment category; (4) adding preset initial parameters to perform model training of the sentiment analysis model; (5) obtaining a fine-grained sentiment analysis output result obtained by model training. The present invention also relates to a corresponding device, processor and storage medium thereof. The method, device, processor and storage medium for implementing fine-grained sentiment analysis based on Transformer of the present invention can accurately identify and judge the sentiment tendency of a certain aspect level in the text by specifically distinguishing the sentiment categories of different aspects, and has a more prominent recognition and prediction advantage compared with other models.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, in particular to the field of natural language processing technology, and specifically refers to a method, device, processor and computer-readable storage medium thereof for implementing fine-grained sentiment analysis based on Transformer. Background Art

[0002] Before the emergence of artificial intelligence, machines only process structured data (such as data in Excel). However, most of the data in real-world scenarios is unstructured, such as articles, pictures, audio, video, etc. Among unstructured data, text is the largest. Although it does not take up as much space as pictures and videos, it has the largest amount of information. In order to analyze and use this text information, we need to use NLP technology to enable machines to understand and use this text information.

[0003] Natural Language Processing (NLP) is a subfield of computer science, information engineering, and artificial intelligence that deals with the interaction between computers and human (natural) language, especially how to program computers to process and analyze large amounts of natural language data. Challenges in natural language processing typically involve two major tasks: natural language understanding (NLU) and natural language generation (NLG).

[0004] Natural language understanding is to make machines like humans, with normal language understanding ability. Natural language understanding (NLU) or natural language interpretation (NLI) is a sub-topic of natural language processing in artificial intelligence and machine reading comprehension. It is applied to many sub-fields such as entity extraction, relationship extraction, machine translation, intelligent question answering, text classification, sentiment analysis, etc.

[0005] Natural language generation is an important part of NLP. Its main purpose is to reduce the communication gap between humans and machines and convert non-language data into language formats that humans can understand. NLG is to bridge the communication gap between humans and machines and convert non-language data into language formats that humans can understand, such as articles and reports.

[0006] Early coarse-grained sentiment analysis methods could only obtain the opinion information of the text, but in fact the evaluation may contain evaluations of multiple objects and aspects, and the sentiment tendencies assigned to these expression objects may be different. For example, the following comment text: The mobile phone screen is very clear, but the battery life is too weak.

[0007] Obviously, from the perspective of text, we cannot directly judge whether the user's sentiment towards the phone is positive or negative, because the user expressed his opinions on the two aspects of the phone. Then we can roughly know the two tasks of ABSA, namely, sentiment object identification (Aspect Term Extraction) and sentiment object tendency analysis (Aspect Term Polarity Analysis). The former is an NLP labeling task (such as named entity recognition, etc.), and the latter is a classification task (targeted judgment of the sentiment category of Aspect), but the existing technology lacks targeted judgment of the sentiment category of Aspect. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, device, processor and computer-readable storage medium for implementing fine-grained sentiment analysis based on Transformer.

[0009] In order to achieve the above objectives, the method, device, processor and computer-readable storage medium for implementing fine-grained sentiment analysis based on Transformer of the present invention are as follows:

[0010] The method for implementing fine-grained sentiment analysis based on Transformer has the following main features:

[0011] (1) Obtain the datasets of Laptop and Restaurant from Sem-Eval 2014 Task 4 for model validation;

[0012] (2) Perform data analysis and conversion on the corresponding data sets obtained;

[0013] (3) Constructing a sentiment analysis model based on the main text, subject, and subject sentiment category;

[0014] (4) adding preset initial parameters to perform model training of the sentiment analysis model;

[0015] (5) Obtain the fine-grained sentiment analysis output results obtained from model training.

[0016] Preferably, the step (1) comprises the following steps:

[0017] (1.1) Obtain the datasets of the two domains, Laptop and Restaurant, contained in the Sem-Eval 2014 Task 4 data source. Each dataset is in .xml file format and contains training data and test data.

[0018] (1.2) The accuracy of sentiment labeling for the Laptop and Restaurant datasets is used as an evaluation indicator to verify its effectiveness.

[0019] Preferably, the distribution of the emotion tags specifically includes:

[0020] The distribution of positive labels in the training and test sets of the Laptop and Restaurant datasets;

[0021] The distribution of neutral labels in the training and test sets of the Laptop and Restaurant datasets; and

[0022] The distribution of negative labels in the training and test sets in the Laptop and Restaurant datasets.

[0023] Preferably, the step (2) specifically comprises the following steps:

[0024] (2.1) Parsing the .xml data of the training set data and the test set data in the Laptop and Restaurant data sets respectively, extracting the text information data, the subject information data and the subject emotion category information data;

[0025] (2.2) Convert the text information data, subject information data and subject emotion category information data into UTF-8 encoding;

[0026] (2.3) The text information data is converted according to the data requirements of the BERT pre-training model.

[0027] Preferably, the step (3) specifically comprises the following steps:

[0028] (3.1) Encoding the body text to meet the data requirements of the BERT pre-training model;

[0029] (3.2) constructing a position encoding vector based on the position information of the subject in the main text;

[0030] (3.3) using replacement characters to hide the main information in the body text;

[0031] (3.4) Inputting the encoded text into the BERT pre-training model to obtain the last hidden layer information;

[0032] (3.5) Multiply the BERT hidden layer information with the subject's position encoding vector to obtain the subject's information in the BERT hidden layer;

[0033] (3.6) The information of the subject in the BERT hidden layer is processed through a linear layer to output the final sentiment category.

[0034] Preferably, the step (4) is specifically as follows:

[0035] The preset values ​​of the initial parameters are set using empirical values, and each initial parameter is optimized and adjusted using a grid search method. At the same time, the Adam optimization algorithm is used to calculate the minimized cross entropy loss function to obtain the optimal values ​​of the training parameters of the linear layer including but not limited to weights and biases, thereby completing the training of the sentiment analysis model.

[0036] The device for implementing fine-grained sentiment analysis based on Transformer has the following main features:

[0037] a processor configured to execute computer executable instructions;

[0038] The memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for implementing fine-grained sentiment analysis based on Transformer are implemented.

[0039] The processor for implementing fine-grained sentiment analysis based on Transformer has the main feature that the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the various steps of the method for implementing fine-grained sentiment analysis based on Transformer are implemented.

[0040] The main feature of this computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned method for implementing fine-grained sentiment analysis based on Transformer.

[0041] The method, device, processor and computer-readable storage medium for implementing fine-grained sentiment analysis based on Transformer of the present invention are adopted. Based on the acquired main text information, the sentiment object tendency is analyzed through classification tasks in a targeted manner. Compared with other classification models, the accuracy is used as the final evaluation indicator. The actual test result has a relatively obvious outstanding advantage compared with other models. It can provide a more effective training model for the current existing technology using natural language for text processing and has good applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1A schematic diagram of constructing a sentiment analysis model for the method of implementing fine-grained sentiment analysis based on Transformer of the present invention.

[0043] Figure 2 This is a schematic diagram of the data format for performing data identification in a specific embodiment of the method for implementing fine-grained sentiment analysis based on Transformer of the present invention. DETAILED DESCRIPTION

[0044] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0045] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0046] The method for implementing fine-grained sentiment analysis based on Transformer, wherein the method comprises the following steps:

[0047] (1) Obtain the datasets of Laptop and Restaurant from Sem-Eval 2014 Task 4 for model validation;

[0048] (2) Perform data analysis and conversion on the corresponding data sets obtained;

[0049] (3) Constructing a sentiment analysis model based on the main text, subject, and subject sentiment category;

[0050] (4) adding preset initial parameters to perform model training of the sentiment analysis model;

[0051] (5) Obtain the fine-grained sentiment analysis output results obtained from model training.

[0052] As a preferred embodiment of the present invention, the step (1) comprises the following steps:

[0053] (1.1) Obtain the datasets of the two domains, Laptop and Restaurant, contained in the Sem-Eval 2014 Task 4 data source. Each dataset is in .xml file format and contains training data and test data.

[0054] (1.2) The accuracy of sentiment labeling for the Laptop and Restaurant datasets is used as an evaluation indicator to verify its effectiveness.

[0055] As a preferred embodiment of the present invention, the distribution of the emotion tags specifically includes:

[0056] The distribution of positive labels in the training and test sets of the Laptop and Restaurant datasets;

[0057] The distribution of neutral labels in the training and test sets of the Laptop and Restaurant datasets; and

[0058] The distribution of negative labels in the training and test sets in the Laptop and Restaurant datasets.

[0059] As a preferred embodiment of the present invention, the step (2) specifically comprises the following steps:

[0060] (2.1) Parsing the .xml data of the training set data and the test set data in the Laptop and Restaurant data sets respectively, extracting the text information data, the subject information data and the subject emotion category information data;

[0061] (2.2) Convert the text information data, subject information data and subject emotion category information data into UTF-8 encoding;

[0062] (2.3) The text information data is converted according to the data requirements of the BERT pre-training model.

[0063] As a preferred embodiment of the present invention, the step (3) specifically comprises the following steps:

[0064] (3.1) Encoding the body text to meet the data requirements of the BERT pre-training model;

[0065] (3.2) constructing a position encoding vector based on the position information of the subject in the main text;

[0066] (3.3) using replacement characters to hide the main information in the body text;

[0067] (3.4) Inputting the encoded text into the BERT pre-training model to obtain the last hidden layer information;

[0068] (3.5) Multiply the BERT hidden layer information with the subject's position encoding vector to obtain the subject's information in the BERT hidden layer;

[0069] (3.6) The information of the subject in the BERT hidden layer is processed through a linear layer to output the final sentiment category.

[0070] As a preferred embodiment of the present invention, the step (4) is specifically as follows:

[0071] The preset values ​​of the initial parameters are set using empirical values, and each initial parameter is optimized and adjusted using a grid search method. At the same time, the Adam optimization algorithm is used to calculate the minimized cross entropy loss function to obtain the optimal values ​​of the training parameters of the linear layer including but not limited to weights and biases, thereby completing the training of the sentiment analysis model.

[0072] In practical application, the specific implementation steps of this technical solution are as follows:

[0073] Step s1, 1SemEval-2014 Task 4 dataset is from the International Semantic Evaluation Conference SemEval-2014, which is mainly used for fine-grained sentiment analysis. It includes two fields, Laptop and Restaurant. The dataset of each field is divided into training data and test data, and the file format is .xml. The accuracy of the proposed method on the test set of Laptop / Restaurant is used as the evaluation indicator to verify its effectiveness.

[0074] Source data format please refer to Figure 2 As shown, “aspectTerm term” is the identified subject and polarity is the identified emotion category.

[0075] Based on the distribution of sentiment labels in the Laptop / Restaurant dataset, the statistical results are shown in Table 1:

[0076] Table 1: Laptop / Restaurant dataset statistics

[0077]

[0078]

[0079] Step s2, the Laptop / Restaurant dataset has been divided into training set and test set. First, the .xml data is parsed for the training set and the test set respectively, that is, the text, subject and corresponding sentiment category are extracted and uniformly converted into UTF-8 encoding, and then the text information is converted according to the data requirements of the BERT pre-training model.

[0080] Step s3, build the model framework. The basic architecture of the sentiment analysis model is as follows Figure 1 shown.

[0081] Among them, Context is the text, Term is the subject, and Polarity is the emotional category of the subject. First, the text needs to be encoded to meet the data requirements of the BERT pre-training model; secondly, based on the position information of the subject in the text, a position encoding vector, i.e., position_ids, is constructed; thirdly, the subject information in the text is hidden, i.e., the subject is replaced with the [MASK] character. This processing method mainly takes into account that we hope that the model only focuses on the semantic information given to the subject by the context of the subject, rather than the information of the subject itself. The subject can take any value, but the emotional information brought by its contextual semantics remains unchanged; then the encoded text is input into the BERT pre-training model to obtain the last hidden layer information, i.e., last_hidden_state; then the BERT hidden layer information and the subject's position encoding vector are multiplied to obtain the subject's information in the BERT hidden layer; finally, the subject's information in the BERT hidden layer passes through a linear layer, i.e., Linear Layer, and finally outputs the emotional category, i.e., Polarity.

[0082] exist Figure 1 In the above example, input_ids, attention_mask, and token_type_ids are the data inputs of the BERT pre-trained model, position_ids is the position encoding of the term in the context, and the context is represented as [w1,w2,w3,w1,…,w i ,w i+1 ,…,w i+h -1,…,w n ], where [w i ,w i+1 ,…,w i+h -1] is a term with a length of h. position_ids is represented as [0,0,0,...,1,1,...1,...0].

[0083] Step s4, the initial parameters of model training are shown in Table 2.

[0084] Table 2: Initial parameters for model training

[0085]

[0086]

[0087] At present, the initial parameter setting is mainly based on empirical values, and the grid search method is used in subsequent optimization and adjustment. The Adam optimization algorithm is used to minimize the cross entropy loss function to obtain the optimal values ​​of the training parameters such as the linear layer weights and biases.

[0088] Step s5, comparison of model results, as shown in Table 3.

[0089] Table 3: Comparison of model results

[0090]

[0091] In order to evaluate the effect of the model more comprehensively, we list some benchmark methods for comparison. These benchmark models are introduced as follows. TD-LSTM is based on the LSTM network structure. It builds sub-networks containing the left / right context of the Target respectively, then merges the two sub-networks, and finally outputs the sentiment polarity value for the Target. AE-LSTM first builds a contextual semantic model based on LSTM, fuses the embedding vector of the Target and the hidden vector H of the context model, and generates an attention weight vector α based on the fusion vector. αH outputs the final sentiment polarity value. ATAE-LSTM is based on the AE-LSTM model. The main optimization method is to add the embedding vector of the Target to the embedding layer. IAN proposes an interactive attention network (IAN) model based on the long short-term memory network (LSTM) and the attention mechanism. It models the Target and Context respectively, and adds Attention to both. After that, let the two interact.

[0092] The above model method mainly relies on the LSTM network architecture, which has long-term memory function. It adopts a gating mechanism, which can solve the gradient explosion and gradient vanishing problems to a certain extent, but its performance is still affected by its inability to be parallel. Secondly, the main optimization point of the IAN model is the interaction between Target and Context, but the Target is often only a few characters long. There is no practical significance in building an LSTM network for it to learn its contextual semantics.

[0093] In order to more effectively compare the effects of this model with other models, we use accuracy as the final evaluation indicator, that is, Where T is the number of samples predicted correctly in the test sample, and N is the total number of test samples.

[0094] The Acc effect of this model on the Restaurant dataset is significantly better than the other four models, 3% higher than the IAN model, and 2% higher than the IAN model on the Laptop dataset.

[0095] Some test samples are shown in Table 4:

[0096] In the text, $T$ is the location of the subject, the actual label column is the real sentiment label in the data, and the predicted label column is the sentiment label predicted by the model in this article. In the label, 1 represents positive sentiment, 0 represents neutral sentiment, and -1 represents negative sentiment.

[0097] Table 4: Comparison of some test sample results

[0098]

[0099] The device for implementing fine-grained sentiment analysis based on Transformer, wherein the device comprises:

[0100] a processor configured to execute computer executable instructions;

[0101] The memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for implementing fine-grained sentiment analysis based on Transformer are implemented.

[0102] The processor for implementing fine-grained sentiment analysis based on Transformer, wherein the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the various steps of the method for implementing fine-grained sentiment analysis based on Transformer are implemented.

[0103] The computer-readable storage medium stores a computer program thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned method for implementing fine-grained sentiment analysis based on Transformer.

[0104] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0105] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0106] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0107] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0108] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0109] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0110] The method, device, processor and computer-readable storage medium for implementing fine-grained sentiment analysis based on Transformer of the present invention are adopted. Based on the acquired main text information, the sentiment object tendency is analyzed through classification tasks in a targeted manner. Compared with other classification models, the accuracy is used as the final evaluation indicator. The actual test result has a relatively obvious outstanding advantage compared with other models. It can provide a more effective training model for the current existing technology using natural language for text processing and has good applicability.

[0111] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method for fine-grained sentiment analysis based on Transformer, characterized in that: The method comprises the following steps: Step S1: Obtain the datasets of Laptop and Restaurant domains of SemEval-2014 Task 4 for model validation; Step S2: performing data analysis and conversion processing on the acquired corresponding data set; Step S3: constructing a sentiment analysis model based on the main text, subject and subject sentiment category; Step S4: adding preset initial parameters to perform model training of the sentiment analysis model; Step S5: Obtain the fine-grained sentiment analysis output result obtained by model training; The step S1 comprises the following steps: Step S1.1: Obtain the datasets of the two domains, Laptop and Restaurant, contained in the SemEval-2014 Task 4 data source, and the file format of each dataset is xml; Step S1.2: The accuracy of sentiment label determination for the datasets in the two fields of Laptop and Restaurant is used as the evaluation index to verify the effectiveness; The distribution of the emotion tags specifically includes: The distribution of positive labels in the training and test sets of the Laptop and Restaurant datasets; The distribution of neutral labels in the training and test sets of the Laptop and Restaurant datasets; and The distribution of negative labels in the training and test sets of the Laptop and Restaurant datasets; The step S2 specifically includes the following steps: Step S2.1: performing data analysis on the training set data and the test set data in the Laptop and Restaurant data sets, respectively, to extract text information data, subject information data, and subject emotion category information data; Step S2.2: converting the body text information data, subject information data and subject emotion category information data into UTF-8 encoding; Step S2.3: converting the text information data according to the data requirements of the BERT pre-training model; The step S3 specifically comprises the following steps: Step S3.1: Encode the body text to meet the data requirements of the BERT pre-training model; Step S3.2: constructing a position encoding vector according to the position information of the subject in the main text; Step S3.3: using replacement characters to hide the main body information in the body text; Step S3.4: input the encoded text into the BERT pre-training model to obtain the last hidden layer information; Step S3.5: multiply the BERT hidden layer information by the position encoding vector of the subject to obtain the information of the subject in the BERT hidden layer; Step S3.6: Process the subject's information in the BERT hidden layer through a linear layer and output the final emotion category.

2. The method for implementing fine-grained sentiment analysis based on Transformer according to claim 1, characterized in that: The step S4 is specifically as follows: The preset values ​​of the initial parameters are set using empirical values, and each initial parameter is optimized and adjusted using a grid search method. At the same time, the Adam optimization algorithm is used to calculate the minimized cross entropy loss function to obtain the optimal values ​​of the training parameters of the linear layer, including but not limited to weights and biases, thereby completing the training of the sentiment analysis model.

3. A device for implementing fine-grained sentiment analysis based on Transformer, characterized in that: The device comprises: a processor configured to execute computer executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, the steps of the method for implementing fine-grained sentiment analysis based on Transformer as described in any one of claims 1 to 2 are implemented.

4. A processor for implementing fine-grained sentiment analysis based on Transformer, characterized in that: The processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the various steps of the method for implementing fine-grained sentiment analysis based on Transformer as described in any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the method for implementing fine-grained sentiment analysis based on Transformer as described in any one of claims 1 to 2.

Images