A method and system for extracting aspect-viewpoint pairs by synchronously tracking entities and relationships

Abstract

The present invention discloses a method and system for extracting aspect-viewpoint pairs for synchronously tracking entities and relationships, belonging to the technical field of sentiment analysis. The method and system for extracting aspect-viewpoint pairs for synchronously tracking entities and relationships include the following steps: obtaining a comment sentence, and using a pre-trained BERT model to perform initial word vector encoding on the input comment sentence; further learning the representation of all entities in the comment sentence, and calculating the correlation between any two entities, to obtain a further encoded entity representation sequence and a correlation matrix; performing role identification on entities according to the entity representation; judging whether there is a relationship between entities according to the correlation matrix; and extracting aspect-viewpoint pairs according to the results of role identification and relationship identification, which has the advantages of multiple full interactions, mutual dependence, entity-specific scalability and strong applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a method and system for extracting aspect-viewpoint pairs for synchronously tracking entities and relationships. Background Art

[0002] The Aspect-Opinion Pair Extraction (AOPE) task was not formally proposed for a long time, so there is still a lot of room for exploration in this task. At present, most AOPE methods divide AOPE into two subtasks: entity recognition and relationship detection. Entity recognition is responsible for determining the role of each entity, that is, whether it is an aspect term / opinion term / neither. Relationship detection is responsible for determining the relationship between two entities, that is, the existence / non-existence of a pairwise relationship between aspects and opinions. These AOPE methods usually train the model in a joint learning manner, balancing the performance of the model on the two subtasks while allowing the two subtasks to influence each other.

[0003] The interaction between the two subtasks of AOPE is very valuable because entities judged as aspects or opinions by the entity recognition subtask are more likely to have pairwise relationships with certain entities, and two entities judged to have strong pairwise relationships by the relationship detection subtask are more likely to be aspects or opinions. In other words, the interaction between the two subtasks can bring inspiration to each other.

[0004] However, after research, it was found that although most existing methods train models in a joint learning manner, the two subtasks are still relatively independent, with few and insufficient interactions. For example, one method sets the entity recognition unit and the relationship detection unit as two channels. In order to achieve interaction, a synchronization unit is constructed to mutually influence the update of the next cycle input of the two channels. The experimental results prove the effectiveness of the interaction between the two subtasks, but from the perspective of the model structure, the connection between the two channels is still not tight enough, and the number of interactions is small. There is also a method that combines the entities after the comment sentence segmentation into spans of different lengths by enumeration, and designs two encoders to learn the representation of spans. Finally, entity recognition and relationship detection are performed on all candidate spans at the same time. In this method, the connection between the two subtasks of AOPE is only the shared span representation and joint learning. Although the above two methods take into account the interaction of the two subtasks, during execution, the entity recognition and relationship detection processes are still relatively independent and not highly correlated in terms of calculation.

[0005] Therefore, it is necessary to provide an aspect-viewpoint pair extraction method and system that simultaneously tracks entities and relationships, aiming to solve the above problems. Summary of the invention

[0006] In view of the deficiencies in the prior art, an embodiment of the present invention aims to provide a method and system for extracting aspect-viewpoint pairs for synchronously tracking entities and relationships, so as to solve the problems in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for extracting aspect-viewpoint pairs by synchronously tracking entities and relationships, the method comprising the following steps:

[0009] S1. Obtain the comment sentence and use the pre-trained BERT model to encode the input comment sentence into the initial word vector;

[0010] S2, further learning the representation of all entities in the comment sentence, and calculating the correlation between any two entities, to obtain the further encoded entity representation sequence and correlation matrix;

[0011] S3, identify the role of the entity according to the entity representation, and determine whether there is a relationship between the entities according to the association matrix;

[0012] S4. Based on the results of role identification and relationship identification, aspect-viewpoint pairs are extracted.

[0013] A system for extracting aspect-viewpoint pairs for synchronously tracking entities and relations, the system comprising:

[0014] The word embedding module is used to encode the initial word vector of the input comment sentence through the pre-trained BERT model;

[0015] The tracking network module is used to obtain the entity representation sequence and the relevance matrix after further encoding;

[0016] The entity annotation module is used to identify the role of entities based on their representations;

[0017] Relationship classification module: used to determine whether there is a relationship between entities based on the association matrix;

[0018] The reasoning module is used to extract aspect-opinion pairs based on the recognition results of the entity labeling module and the relationship classification module.

[0019] As a further solution of the present invention, the word embedding module uses a BERT model as an initial encoder, and the BERT model is used to learn the initial representation of entities containing context information. The input vector e of BERT i is the sum of three embedding feature vectors, including the token embedding Positional Embedding and segment embedding

[0020]

[0021] Token embedding is obtained by tokenizing sentences using WordPiece. BERT adds [CLS] and [SEP] at the beginning and end of a sentence, respectively, to obtain a complete token sequence, also known as an entity sequence. Token embedding is generated by converting each token into its corresponding id in BERT. Position embedding encodes the position information of the token into a feature vector. Segment embedding is used to distinguish different sentences in a paragraph.

[0022] As a further solution of the present invention, the tracking network module includes a tracking network composed of a group of gated RNNs, wherein each RNN chain shares trainable parameters at each time step, and all tokens in the comment sentence are set as tracking targets. At the same time, the input sequence of each time step is also set to the token sequence of the comment sentence, forming a network with an N×N grid shape after expansion, which is used to synchronously update the representation of all tokens in the computing unit A at each time step and calculate the correlation between two tokens.

[0023] As a further solution of the present invention, in the computing unit A, the updating process of the entity representation is highly dependent on the calculation process of the association degree. Specifically, at each time step, the representation of the tracked token is updated using the tracked token information, the token information input at the current time step, and the association degree between the two tokens:

[0024]

[0025] Use h i,t To store the tracked token x i In the updated representation at time step t, (3), h i,t-1 The token x to be tracked i In the previous time step, x t is the token input at time step t, r i,t It is token x i and token x t The correlation between them is calculated as follows:

[0026] r i,t =g(h i,t-1 ,x t ) formula (4).

[0027] As a further solution of the present invention, the entity labeling module selects CRF as the entity labeling model for the entity recognition subtask, and entity recognition assigns an entity label to each token. Among them, B means that the token is the beginning of an aspect item (A) or a viewpoint item (P), I means that the token is inside an aspect item (A) or a viewpoint item (P), and O means that the token does not belong to any aspect item or viewpoint item.

[0028] As a further solution of the present invention, the relationship classification module is for each token pair (x m ,x n ) Assign relationship labels Where 1 represents s m and n There is a pairwise relationship between aspects and opinions, while 0 indicates an unpaired relationship between two tokens. Using the association matrix R, we get the predicted relationship of each token pair and the real relationship label matrix The true relationship of each token pair is obtained, and the loss value of the relationship detection subtask is obtained by calculating the cross entropy between the predicted and true relationship distributions. The model is trained by joint learning, and the losses of the two subtasks are combined to obtain the final training goal.

[0029] As a further solution of the present invention, the reasoning module first uses CRF to identify aspect terms when formally reasoning about aspect-viewpoint pairs. and viewpoint items They are composed of one or more tokens. a and i o Indicate their starting subscripts in the token sequence S, and then use the association matrix R to calculate the probability that a and o are in a pair relationship according to the inference algorithm:

[0030]

[0031] When β exceeds a certain threshold, the current aspect item and opinion item are extracted as aspect-opinion pairs and put into the set AOP={(a k ,o l ),...}middle.

[0032] In summary, compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0033] 1. The present invention proposes an aspect-viewpoint pair extraction method that simultaneously tracks entities and relationships, and designs a tracking network that can calculate the probability that any two entities belong to an aspect-viewpoint pair while learning all entity representations in parallel. The two processes are interdependent in the calculation of each time step, and the obtained results are used for decoding of the two subtasks respectively. Therefore, in the present invention, the two subtasks can achieve multiple and sufficient interactions, and have the effect of multiple and sufficient interactions and interdependence;

[0034] 2. With the help of the characteristics of the tracking network, the present invention can learn more entity-specific information, including context information and aspect-viewpoint relationship information related to the tracked entity, and has an entity-specific effect;

[0035] 3. The present invention provides a general solution for the aspect-viewpoint pair extraction task and the quasi-aspect-viewpoint pair extraction task, namely, by utilizing an N×N grid structure, while learning the entity representation, an N×N association matrix can be obtained for detecting the relationship between entities. The computing units within the grid (i.e., the entity representation update algorithm) are easy to modify and replace. Therefore, the network design of the present invention has strong flexibility and scalability, and has the effects of relationship detection, scalability and strong applicability.

[0036] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention is a flowchart of an embodiment of the present invention.

[0038] Figure 2 This is a diagram of the tracking network structure in the embodiment of the invention.

[0039] Figure 3 This is a visualization diagram of the correlation in the embodiment of the invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0042] In one embodiment of the present invention, see Figure 1 , the method for extracting aspect-opinion pairs for synchronously tracking entities and relationships includes the following steps: S1, obtaining comment sentences, and using a pre-trained BERT model to perform initial word vector encoding on the input comment sentences; S2, further learning the representations of all entities in the comment sentences, and calculating the association between any two entities, to obtain a further encoded entity representation sequence and an association matrix; S3, performing role identification on entities based on entity representations, and judging whether there is a relationship between entities based on the association matrix; S4, extracting aspect-opinion pairs based on the results of role identification and relationship identification.

[0043] In one embodiment of the present invention, see Figure 1 and Figure 2 The invention discloses a system for extracting aspect-opinion pairs for synchronously tracking entities and relationships. The system comprises a word embedding module for encoding the initial word vector of the input comment sentence through a pre-trained BERT model; a tracking network module for obtaining a further encoded entity representation sequence and a correlation matrix; an entity labeling module for identifying the role of the entity according to the entity representation; a relationship classification module for determining whether there is a relationship between entities according to the correlation matrix; and a reasoning module for extracting aspect-opinion pairs according to the results of the entity labeling module and the relationship classification module.

[0044] In this embodiment, the word embedding module uses the BERT model as the initial encoder. The BERT model is used to learn the initial representation of entities containing context information. The input vector e of BERT is i is the sum of three embedding feature vectors, including the token embedding Positional Embedding and segment embedding

[0045]

[0046] Token embedding is obtained by tokenizing sentences using WordPiece. Specifically, words are divided into a limited set of common sub-word units, such as splitting "l earn i ng" into "l earn" and "i ng" to balance the effectiveness of words and the flexibility of characters. In the present invention, each token is an entity. BERT adds [CLS] and [SEP] at the beginning and end of the sentence respectively to obtain a complete token sequence, that is, an entity sequence. Token embedding is generated by converting each token into its corresponding id in BERT. Position embedding encodes the position information of the token into a feature vector. Segment embedding is used to distinguish different sentences in a paragraph.

[0047] In actual training, the vector sequence E = {e1, e2, ..., e N} will be fed into the pre-trained BERT model. BERT uses Transformer as its main framework, and Transformer can capture the bidirectional relationship in the sentence, so it can learn rich context information. Finally, the initial context representation of each token (that is, entity) can be obtained as follows:

[0048] x i =BERT(e i ) formula (2),

[0049] token sequence X = {x1, x2, ..., xi ,...,x N} will be used for the execution of subsequent tasks.

[0050] See also Figure 2 The tracking network module includes a tracking network composed of a group of gated RNNs, wherein each RNN chain shares trainable parameters at each time step, sets all tokens in the comment sentence as tracking targets, and at the same time, the input sequence of each time step is also set to the token sequence of the comment sentence, thus forming a network in the shape of an N×N grid after expansion, which is used to synchronously update the representation of all tokens in the computing unit A at each time step and calculate the correlation between two tokens.

[0051] In computation unit A, the entity representation update process and the correlation calculation process are highly dependent, which enables multiple interactions between the two subtasks of AOPE. Specifically, at each time step, we use the tracked token information, the input token information of the current time step, and the correlation between the two tokens to update the representation of the tracked token:

[0052] h i,t =f(x i ,h i,t-1 ,x t ,r i,t ) formula (3),

[0053] Use h i,t To store the tracked token x i The updated representation at time step t. In formula 3, h i,t-1 The token x to be tracked i In the previous time step, x t is the token input at time step t, r i,t It is token x i and token x t The correlation between them is calculated as follows:

[0054] r i,t =g(h i,t-1 ,x t ) formula (4),

[0055] It can be seen that r i,t The calculation process needs to use the token information being tracked and the token information input at the current time step.

[0056] Through the above algorithm, we can further learn the contextual information and relationship information related to the tracked entity to update the entity representation and calculate the correlation between two entities. By introducing the representation of the tracked entity itself, the learned information is more entity-specific. More importantly, by observing formulas (3) and (4), we can find that the correlation r i,t The calculation needs to be tracked token x i The representation h at the previous time step i,t-1 The participation of i,t The update of also depends on r i,t The participation of the token in the last time step represents the sequence H = {h 1,N ,h 2,N ,...,h i,N ,...,h N,N} and the correlation matrix R obtained from each time step will be used for token entity labeling and token relationship classification respectively, so the two AOPE subtasks of entity recognition and relationship detection can realize multiple and sufficient interactions in the present invention. In addition, there are many design methods for the functions f and g in formula (3) and formula (4), which can be specifically set according to actual needs, so the present invention is not only applicable to AOPE tasks but also to AOPE-like tasks.

[0057] The entity labeling module selects CRF as the entity labeling model for the entity recognition subtask. Entity recognition assigns an entity label to each token. Among them, B means that the token is the beginning of an aspect item (A) or a point of view item (P), I means that the token is inside an aspect item (A) or a point of view item (P), and O means that the token does not belong to any aspect item or point of view item. CRF needs to calculate the label sequence The probability of possibility is as shown in formula (5):

[0058]

[0059] Among them, Y E is the set of all candidate label sequences of sequence C, score is a comprehensive scoring function that includes state feature scores and transition feature scores, and its calculation formula is as follows:

[0060]

[0061] in, Indicates c i,N Labeled as y according to its own characteristics i E The state feature score of It means the tag Followed by a label The transfer feature score of . The trainable weight matrix and To build c i,N and the mapping between the five labels, is from The value extracted from .

[0062] Finally, the loss of the entity recognition subtask is calculated using the negative log-likelihood function:

[0063]

[0064] in, is the true label sequence of C. When CRF predicts The greater the probability of the possibility, the smaller the loss value of the task.

[0065] The relationship classification module is for each token pair (x m ,x n ) Assign relationship labels Where 1 represents s m and n There is a pairwise relationship between aspects and opinions, while 0 indicates a non-paired relationship between two tokens. Using the association matrix R, we can get the predicted relationship distribution of each token pair:

[0066]

[0067] in, Represents the true relationship label, using the true relationship label matrix Get the true relationship distribution of each token pair:

[0068]

[0069] The loss value of the relationship detection subtask is obtained by calculating the cross entropy between the predicted and true relationship distributions:

[0070]

[0071] The model is trained by joint learning, combining the losses of the two subtasks to obtain the final training goal:

[0072] L(θ)=λ E L E +λ R L R Formula (12),

[0073] In order to balance the two subtasks, two hyperparameters λ need to be set E and λ RAs the weight of the two task losses, to obtain the best joint learning performance of the model.

[0074] When formally reasoning about aspect-viewpoint pairs, the reasoning module first uses CRF to identify aspect terms. and viewpoint items They are composed of one or more tokens. a and i o Indicate their starting subscripts in the token sequence S, and then use the association matrix R to calculate the probability that a and o are in a pair relationship according to the inference algorithm shown in formula (13):

[0075]

[0076] From the relationship door i,t As can be seen from the calculation method, R is not a symmetric matrix, so it is necessary to obtain the two correlations of the two tokens in R respectively. When β exceeds a certain threshold, the current aspect item and opinion item will be extracted as aspect-opinion pairs and put into the set AOP={(a k ,o l ),...}middle.

[0077] Work results: The present invention has been experimented on the three datasets of 2014Res, 2014Lap, and 2015Res of the SemEval challenge tasks commonly used in the field of sentiment analysis. The performance of the method in the three tasks of aspect extraction, opinion extraction, and aspect-opinion pair extraction has surpassed several currently advanced aspect-opinion pair extraction methods;

[0078] The present invention provides a general solution for the aspect-viewpoint pair extraction task and the aspect-viewpoint pair-like extraction task.

[0079] For details, see Figure 3 ,When the comment sentence {The,potato,bal ls,were,not,dry,at,al,,,in,fact,it,was,buttery,.} is input, the entity annotation module of the method of the present invention will assign labels to all the word entities therein: {O,BA,IA,O,BP,IP,O,O,O,O,O,O,O,BP,O}. In addition, the correlation between the entities calculated by the present invention is visualized, as Figure 3As shown in the figure, different color intensities represent different orders of magnitude, and the darker the color, the larger the value. By observation, it can be found that the present invention accurately detects the highest correlation between aspect-opinion relationships, and can even capture the long-distance relationship between "potato ba lls" and "buttery". Finally, using the reasoning module and combining the prediction results of the two subtasks, the aspect-opinion pairs (potato ba lls, not dry) and (potato ba lls, buttery) can be extracted.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A system for extracting aspect-viewpoint pairs that simultaneously tracks entities and relationships, characterized in that: The system comprises: The word embedding module is used to encode the initial word vector of the input comment sentence through the pre-trained BERT model; The tracking network module is used to obtain the entity representation sequence and the relevance matrix after further encoding; The entity annotation module is used to identify the role of entities based on their representations; Relationship classification module: used to determine whether there is a relationship between entities based on the association matrix; The reasoning module is used to extract aspect-viewpoint pairs based on the results of the entity labeling module and the relationship classification module; The word embedding module uses the BERT model as the initial encoder. The BERT model is used to learn the initial representation of entities containing contextual information. The input vector e of BERT i is the sum of three embedding feature vectors, including the token embedding Positional Embedding and segment embedding Token embedding is obtained by tokenizing sentences using WordPiece. BER adds [CLS] and [SEP] at the beginning and end of a sentence to obtain a complete token sequence, i.e., an entity sequence. Token embedding is generated by converting each token into its corresponding id in BERT. Position embedding encodes the position information of the token into a feature vector. Segment embedding is used to distinguish different sentences in a paragraph. The tracking network module includes a tracking network composed of a group of gated RNNs, wherein each RNN chain shares trainable parameters at each time step, sets all tokens in the comment sentence as tracking targets, and at the same time, the input sequence of each time step is also set as the token sequence of the comment sentence, forming a network in the shape of an N×N grid after expansion, and the network is used to synchronously update the representation of all tokens in the calculation unit A at each time step and calculate the correlation between two tokens; In the calculation unit A, the entity representation update process is highly dependent on the association calculation process. Specifically, at each time step, the representation of the tracked token is updated using the tracked token information, the token information input at the current time step, and the association between the two tokens: h i,t =f(x i ,h i,t-1 ,x t ,r i,t ); Use h i,t To store the tracked token x i In the updated representation at time step t, (3), h i,t-1 The token x to be tracked i In the previous time step, x t is the token input at time step t, r i,t It is token x i and token x t The correlation between them is calculated as follows: r i,t =g(h i,t-1 ,x t )。 2. The aspect-viewpoint pair extraction system for synchronously tracking entities and relationships according to claim 1, characterized in that: The entity labeling module selects CRF as the entity labeling model for the entity recognition subtask. Entity recognition assigns an entity label to each token. Among them, B means that the token is the beginning of an aspect item (A) or a viewpoint item (P), I means that the token is inside an aspect item (A) or a viewpoint item (P), and O means that the token does not belong to any aspect item or viewpoint item.

3. The aspect-viewpoint pair extraction system for synchronously tracking entities and relationships according to claim 2, characterized in that: The relationship classification module is for each token pair (x m ,x n ) Assign relationship labels Where 1 represents s m and n There is a pairwise relationship between aspects and opinions, while 0 indicates an unpaired relationship between two tokens. Using the association matrix R, we get the predicted relationship of each token pair and the real relationship label matrix The true relationship of each token pair is obtained, and the loss value of the relationship detection subtask is obtained by calculating the cross entropy between the predicted and true relationship distributions. The model is trained by joint learning, and the losses of the two subtasks are combined to obtain the final training goal.

4. The aspect-viewpoint pair extraction system for synchronously tracking entities and relationships according to claim 3, characterized in that: When formally reasoning about aspect-viewpoint pairs, the reasoning module first uses CRF to identify aspect terms. and viewpoint items They are composed of one or more tokens. a and i o Indicate their starting subscripts in the token sequence S, and then use the association matrix R to calculate the probability that a and o are in a pair relationship according to the inference algorithm: When β exceeds a certain threshold, the current aspect item and opinion item are extracted as aspect-opinion pairs and put into the set AOP={(a k ,o l ),...}middle.

5. A method for extracting aspect-viewpoint pairs for synchronously tracking entities and relationships, implemented based on the system for extracting aspect-viewpoint pairs for synchronously tracking entities and relationships of claim 1, characterized in that: The following steps are involved: S1. Obtain the comment sentence and use the pre-trained BERT model to encode the input comment sentence into the initial word vector; S2, further learning the representation of all entities in the comment sentence, and calculating the correlation between any two entities, to obtain the further encoded entity representation sequence and correlation matrix; S3, identify the role of the entity according to the entity representation, and determine whether there is a relationship between the entities according to the association matrix; S4. Based on the results of role identification and relationship identification, aspect-viewpoint pairs are extracted.

Images