A triplet extraction method based on self-attention

Abstract

A triple extraction method based on self-attention includes the following steps: 1) presetting the entity type to be extracted from the case and the template of the triple describing the case according to the brief description of the case registered in the database; 2) in the brief description of the case, annotating the triples in the description according to the set template by using the annotation tool; 3) using the Transformer model as the core to build a pipeline extraction system; 4) using the self-attention structure of the model itself to model the triples in the case of entity redundancy; 5) using the method of sharing position information and biased self-attention to accelerate triple prediction; 6) evaluating and sorting the prediction results. The present invention extracts unstructured data in the data with a pipeline relationship extraction algorithm, thereby obtaining structured triple data, which provides powerful help for the subsequent analysis of the data.

Description

Technical Field

[0001] The present invention relates to information extraction, big data and deep learning, and provides a triplet extraction method based on self-attention. Background Art

[0002] With the continuous development of the Internet, the growth of information has entered an explosive era. Every day, thousands of structured and unstructured text data are constantly generated and updated. However, the huge amount of data is just a number. The reality is that a large amount of data is basically in a frozen state. After being stored in the database, there is no chance for secondary development.

[0003] In recent years, cloud computing and artificial intelligence have emerged. The powerful computing power provided by cloud computing and the large number of algorithms invented in the field of artificial intelligence have made it possible to reuse this type of data and rediscover its value.

[0004] At the Yunqi Conference on October 13, 2016, Hangzhou City announced a plan to install an artificial intelligence center in Hangzhou - Hangzhou City Data Brain. The goal is to rely on cloud computing and artificial intelligence to let data help the city think and make decisions. The main means of implementation is to integrate, analyze and calculate data from government affairs, public, corporate and Internet aspects, and then use information technology strategies to enable Hangzhou City Brain to have the ability to manage all aspects of the city. In the field of transportation, the city brain has achieved certain results. Preliminary experimental data show that with the intelligent adjustment of traffic lights, the road capacity has increased by about 5%. In recent years, the field of natural language processing has also seen the emergence of large-scale comprehensive technical projects such as knowledge graphs, which can enable the city brain to qualitatively improve its processing capabilities in police affairs, help the police summarize laws from historical cases, and improve the efficiency of solving cases. As the basis for building knowledge graphs, triple extraction naturally has important research significance.

[0005] There are a large number of relationships between people and places, between people and people, and between characters in the brief descriptions of historical cases, but it is time-consuming and laborious to manually label these three relationships, and it requires experienced professionals to label valuable triple data. However, deep learning methods can pre-learn the characteristics of such texts, and then automatically predict a large number of subsequent input texts. However, limited by the computing power bottleneck of current deep learning, only a small number of algorithm technologies have been successfully implemented, and the application scenario requirements are very simple. Once faced with complex situations, such as multi-relation triple extraction, long text relationship extraction, and entity overlapping relationship extraction, the existing pipeline triple extraction algorithm cannot accurately achieve the expected extraction goals, greatly reducing the data quality of the knowledge graph.

[0006] In this case, by introducing the self-attention mechanism and position sharing technology, the problems existing in the current pipeline extraction model can be greatly improved, and the speed of triple extraction can be accelerated, greatly improving the efficiency of the information extraction task. Summary of the invention

[0007] In order to solve the problem of triple data extraction in complex scenarios, the present invention proposes a data triple extraction method based on self-attention. In the existing case descriptions, the person-to-person relationships, person-to-person relationships and person-to-land relationships that are helpful for case reasoning are extracted, which plays a certain role in constructing high-quality knowledge graphs and accelerating case reasoning.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] A triplet extraction method based on self-attention, the method comprising the following steps:

[0010] 1) Obtain the description text of the case registered in the database, and pre-set the schema of the entities and triples that need to be annotated. The entities include names of people, places, time, crime, and method of committing the crime. The triples include the time of the crime, the location of the crime, the method of committing the crime, the crime, and the social relationship between the offender and the victim;

[0011] 2) According to the above annotation schema, the triple relationship in the description text is manually annotated on the web-based annotation platform, and annotated text in a certain format is generated as training data and verification data;

[0012] 3) After the description text and the annotation text are matched one by one, two sub-files are generated according to the annotation results;

[0013] 4) The training of the triple extraction method uses a pipeline strategy. First, the Transformer network model is used as an encoder and the pointer network is used as a decoder to train the entity extraction of the BIEO-annotated text. At the same time, another Transformer network model is used as an encoder and the fully connected network is used as a decoder to train the relationship classification of the text annotated with entities.

[0014] 5) After training is completed, save the entity model and relationship model separately;

[0015] 6) The prediction method is to input a text into the entity model, predict all potential entities in it, and pair them up and mark them after the text in the form of markers. The text marked with markers is then input into the relational model to obtain the relationship between each entity pair.

[0016] 7) Arrange the outputs of the two models to obtain the final triplet extraction results.

[0017] Furthermore, the process of step 3) is as follows: one of the sub-files is a named entity recognition annotation text that annotates all entities using the BIEO annotation method; the other sub-file is a relationship classification annotation text that annotates entity relationships at the entity span level using the marker tagging method.

[0018] Further, the process of step 4) is:

[0019] 4.1) Given a tokenized text X = [x1, x2…x n ], where x t is the token at position t in the text. After X is input into the Transformer model, the hidden representation of the text is obtained as H = [h1,h2…h n ], and the process is expressed as:

[0020] H = Transformer(X) (1)

[0021] 4.2) According to the annotated text of BIEO, we can construct the gold tag G for entity recognition = [g1,g2…g n ], and then input the hidden representation H obtained in 4.1) into the pointer network to obtain the labeled classification probability P = [p1, p2...p n ], and then use the cross entropy loss function to train the entity recognition task:

[0022]

[0023] Where N is the number of tokens, i is the position of the token in the sentence, L is the number of annotated categories, and c is the annotated category;

[0024] 4.3) The relation classification model needs to construct additional text vectors. Given a tokenized text X = [x1, x2…x n ], where x t is the token at position t in the text, and the entity marker is added after the text X, denoted as M = [m s1 ,m e1 ,m s2 ,m e2 ……m sk-1 ,m ek-1 ,m sk ,m ek], each pair of markers shares the position encoding with the starting and ending positions of the entity, which is used to mark the entity pairs with relationships. In addition, the self-attention layer of Transformer is used to extract the attention matrix A of the starting position markers of the head entity and the tail entity in each entity pair. s and A o , and make the following calculation to obtain the relational feature vector C for each entity pair used for classification i :

[0025] A=A s ·A o (3)

[0026]

[0027]

[0028] C=H T a (6)

[0029] Among them, h is the number of multi-head attention heads, and H is the hidden representation obtained by transformer;

[0030] 4.4) Then, the cross entropy loss function is used to calculate the loss of the locally annotated relationship and the predicted relationship:

[0031]

[0032] Among them, N is the number of relations, i is the position of a sentence in a batch of data, R is the total number of relation categories, and c is the category of the relation.

[0033] The beneficial effects of the present invention are as follows: 1. Using the pointer network as an entity extraction decoder optimizes the accuracy of entity extraction; 2. Using the self-attention layer mechanism to construct the relationship feature vector, the stability and reliability of pipeline triple extraction are improved, providing a guarantee for building a high-quality knowledge graph. 3. It solves the limitation that a sentence can only predict one type of relationship in the previous pipeline triple extraction, and accelerates the speed of relationship prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a logical flow chart of the triple extraction method based on self-attention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings.

[0036] Reference Figure 1 , a triplet extraction method based on self-attention, the method comprising the following steps:

[0037] 1) Obtain the description text of the case registered in the database, and pre-set the schema of the entities and triples that need to be annotated. The entities include names, places, time, crimes and methods of committing crimes. The triples include the time of committing crimes, the place of committing crimes, the methods of committing crimes, the crimes of committing crimes and the social relationship between the offender and the victim. The specific form can be expressed as (head entity: name, relationship: time of committing crimes, tail entity: time), (head entity: name, relationship: place of committing crimes, tail entity: place), (head entity: name, relationship: method of committing crimes, tail entity: method of committing crimes), (head entity: name, relationship: crime of committing crimes, tail entity: crime), (head entity: name, relationship: social relationship, tail entity: name);

[0038] 2) According to the above annotation schema, the triple relationship in the description text is manually annotated on the web-based annotation platform, and annotated text in a certain format is generated as training data and verification data;

[0039] 3) After the description text and the annotation text are matched one by one, two sub-files are generated according to the annotation results: one is the named entity annotation text annotated only by BIEO, and the other is the relationship classification annotation text annotated by the marker tagging method at the span level;

[0040] 4) The training of the triple extraction method uses a pipeline strategy. First, the Transformer network model is used as an encoder and the pointer network is used as a decoder to train the entity extraction of the BIEO annotated text. At the same time, another Transformer network model is used as an encoder and the fully connected network is used as a decoder to train the relationship classification of the text annotated with entities. The process is as follows:

[0041] 4.1) Given a tokenized text X = [x1, x2…x n ], where x t is the token at position t in the text. After X is input into the Transformer model, the hidden representation of the text is obtained as H = [h1,h2…h n ], the process can be expressed as:

[0042] H = Transformer(X) (1)

[0043] 4.2) According to the BIEO annotation text, construct the gold tag G for entity recognition = [g1, g2...g n ], and then input the hidden representation H obtained in 4.1) into the pointer network to obtain the labeled classification probability P = [p1, p2...p n]. Then the cross entropy loss function is used to train the entity recognition task:

[0044]

[0045] 4.3) The relation classification model needs to construct additional text vectors. Given a tokenized text X = [x1, x2…x n ], where x t is the token at position t in the text, and the entity marker is added after the text X, denoted as M = [m s1 ,m e1 ,m s2 ,m e2 ……m sk-1 ,m ek-1 ,m sk ,m ek ], each pair of markers shares a positional encoding with the start and end positions of the entity, which is used to mark entity pairs with relationships; in addition, using the self-attention layer of Transformer, we extract the attention matrix A of the start position markers of the head entity and the tail entity in each entity pair s and A o , and make the following calculation to obtain the relational feature vector C for each entity pair used for classification i :

[0046] A=A s ·A o (3)

[0047]

[0048]

[0049] C=H T a (6)

[0050] Among them, h is the number of multi-head attention heads, and H is the hidden representation obtained by transformer;

[0051] 4.4) Then, the cross entropy loss function is used to calculate the loss of the locally annotated relationship and the predicted relationship:

[0052]

[0053] Where N is the number of relations, i is the position of a sentence in a batch of data, R is the total number of relation categories, and c is the category of the relation;

[0054] 5) After training is completed, save the entity model and relationship model separately;

[0055] 6) The prediction method is to input a text into the entity model, predict all potential entities in it, and pair them up and mark them after the text in the form of markers. The text marked with markers is then input into the relational model to obtain the relationship between each entity pair.

[0056] 7) Arrange the outputs of the two models to obtain the final triplet extraction results.

[0057] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A triplet extraction method based on self-attention, characterized in that: The method comprises the following steps: 1) Obtain the description text of the case registered in the database, and pre-set the schema of the entities and triples that need to be annotated. The entities include names of people, places, time, crime, and method of committing the crime. The triples include the time of the crime, the location of the crime, the method of committing the crime, the crime, and the social relationship between the offender and the victim; 2) According to the above annotation schema, the triple relationship in the description text is manually annotated on the web-based annotation platform, and annotated text in a certain format is generated as training data and verification data; 3) After the description text and the annotation text are matched one by one, two sub-files are generated according to the annotation results; 4) The training of the triple extraction method uses a pipeline strategy. First, the Transformer network model is used as an encoder and the pointer network is used as a decoder to train the entity extraction of the BIEO annotated text. At the same time, another transformer network model is used as an encoder and the fully connected network is used as a decoder to train the relationship classification of the text annotated with entities. 5) After training is completed, save the entity model and relationship model separately; 6) When predicting, a text is first input into the entity model, all potential entities in it are predicted, and they are paired up and annotated with markers after the text. The text annotated with markers is then input into the relational model to obtain the relationship between each entity pair; 7) Arrange the outputs of the two models to obtain the final triple extraction results; The process of step 4) is: 4.1) Given a tokenized text X = [x1, x2…x n ], where x n is the token at position n in the text. After X is input into the Transformer model, the hidden representation of the text is obtained as H = [h1,h2…h n ], and the process is expressed as: H = Transformer(X) (1) 4.2) According to the BIEO annotation text, construct the gold tag G for entity recognition = [g1, g2...g n ], and then input the hidden representation H obtained in 4.1) into the pointer network to obtain the labeled classification probability P = [p1, p2…p n ], and then use the cross entropy loss function to train the entity recognition task: 4.3) The relation classification model needs to construct additional text vectors. Given a tokenized text X = [x1, x2…x n ], where x t is the token at position t in the text, and the entity marker is added after the text X, denoted as M = [m s1 ,m e1 ,m s2 ,m e2 ……m sk-1 ,m ek-1 ,m sk ,m ek ], each pair of markers shares position encoding with the start and end positions of the entity, which is used to mark entity pairs with relationships. In addition, using the self-attention layer of Transformer, we extract the attention matrix A of the start position markers of the head entity and the tail entity in each entity pair. s and A o , and make the following calculation to obtain the relational feature vector C for each entity pair used for classification i : A=A s ·A o (3) C=H T a (6) Among them, h is the number of multi-head attention heads, and H is the hidden representation obtained by transformer; 4.4) Then, the cross entropy loss function is used to calculate the loss of the locally annotated relationship and the predicted relationship: Among them, N is the number of relations, i is the position of a sentence in a batch of data, R is the total number of relation categories, and c is the category of the relation.

2. The method for extracting triples based on self-attention according to claim 1, characterized in that: The process of step 3) is as follows: one of the sub-files is a named entity recognition annotation text that annotates all entities using the BIEO annotation method; the other sub-file is a relationship classification annotation text that annotates entity relationships at the entity span level using the marker tagging method.

Images