A method for quickly constructing an overlapping relationship extraction text data

Abstract

The application provides a method for quickly constructing overlapping relationship extraction text data, comprising the following steps: step 1, constructing a field dictionary; step 2, obtaining text; step 3, dividing sentences and assembling data; step 4, adding anchor text to a candidate entity set; step 5, extracting non-anchor text entities in the text, and adjusting the data distribution of samples; step 6, inquiring the entity number of an entity in a Wikipedia knowledge graph, which is used as the unique identifier of the entity; and step 7, inquiring the relationship between entities, and marking the relationship of the divided sentences. The application redefines the type of overlapping relationship data, provides a data set that can be studied, and lays a data foundation for overlapping relationship research.

Description

Technical Field

[0001] This invention belongs to the field of relation extraction in natural language processing, and particularly relates to a method for rapidly constructing overlapping relations to extract text data. Background Technology

[0002] With the development of big data and the enhancement of machine computing power, more and more unstructured semantic data structures are beginning to attract attention, and knowledge graphs are one such form. In knowledge graph construction, it is necessary to extract structured semantic relationships from unstructured text; this is relation extraction. However, overlapping relationships are a relatively new problem encountered in relation extraction in recent years. How to simultaneously extract multiple semantic relationships where entities or relationships overlap from one or more text sequences has become a topic worthy of attention.

[0003] Currently, most commonly used relation extraction datasets lack research on overlapping relations. The only dataset currently addressing overlapping relations—NYT—contains a relatively small number of overlapping relations, has low complexity, contains a significant amount of dirty data, and suffers from unclear definitions. Deep learning-based machine learning tasks require a large amount of task data, and the lack of overlapping relation data objectively hinders the progress of research and application in these tasks.

[0004] Currently, commonly used methods for constructing relation extraction datasets mainly fall into two categories. The first type is based on manually labeled evaluation datasets. Typical datasets include SemEval, ACE2005, and ACE2003-2004. These datasets rely heavily on manual labeling, which is costly, and the ACE dataset is protected by copyright, making it unsuitable for large-scale and diverse industrial applications.

[0005] The second type is datasets built semi-automatically using Wikipedia corpora and a combination of remote supervision and minimal human intervention. These datasets are quick and inexpensive to build, but they offer relatively little research on overlapping relationships.

[0006]

Paper 1

[0007] The core idea of ​​this algorithm is to align text with entities in a large-scale knowledge graph and then annotate the text using the existing relationships between entities in the knowledge graph. The fundamental assumption of remote supervision is that if a triple R(E1, E2) (where R represents a relation, and E1 and E2 represent two entities) can be obtained from the knowledge graph, and E1 and E2 co-occur in sentence S, then S expresses the relation R between E1 and E2 and is labeled as a positive training example. Remote supervision algorithms are widely used in mainstream relation extraction systems and are one of the hot research topics in this field. While this algorithm effectively solves the problem of data annotation scale, its overly strong fundamental assumptions can introduce a large amount of noisy data.

[0008] [Paper 2] FewRel: A Large-Scale Supervised Few-Shot RelationClassification Dataset with State-of-the-Art Evaluation

[0009] This method combines manually labeled wiki data with remote supervision to semi-automatically construct the largest precision-labeled relation extraction dataset to date. Summary of the Invention

[0010] Objective: To overcome the difficulty of constructing data for extracting overlapping relationships, this invention provides a method for rapidly constructing text data for extracting overlapping relationships. This method can complete the construction of data for extracting overlapping relationships in a short time. The invention includes the following steps:

[0011] Step 1: Construct a domain dictionary;

[0012] Step 2, obtain the text;

[0013] Step 3: Segment and assemble the data;

[0014] Step 4: Add the anchor text to the candidate entity set;

[0015] Step 5: Extract non-anchor text entities from the text and adjust the data distribution of the samples.

[0016] Step 6: Query the entity ID of the entity in the Wikipedia knowledge graph, where the ID serves as the unique identifier of the entity;

[0017] Step 7: Query the relationships between entities and use the relationships to annotate the clauses.

[0018] Step 1 includes: using a named entity recognition tool to perform part-of-speech tagging, extracting words related to the dataset domain, wherein the dataset (the target dataset constructed in this invention) is the target domain overlap relationship extraction task dataset to be constructed.

[0019] Step 1 specifically includes: obtaining domain text of the target domain, segmenting the text into sentences using commas as delimiters, counting the frequency of n-grams (N-gram model) in the obtained sentences, and taking the n-grams with the highest frequency among all n-grams (X1) as the candidate set of entries for the dictionary; the value of X1 is generally 10%.

[0020] The obtained n-grams are fed into a BERT model pre-trained on Wikipedia to obtain the first type of word vectors. At the same time, the entries in the Wikipedia database are also fed into the BERT model to obtain the second type of word vectors. The two types of word vectors are clustered together. After selecting the Wikipedia database entries as the cluster centers, the n-gram distribution centered on the Wikipedia entries is obtained. The n-grams of the Wikipedia central entries and their surroundings are manually filtered to obtain the final domain dictionary. The domain dictionary contains a mapping relationship: a one-to-many relationship between the Wikipedia database entries and the aforementioned n-grams.

[0021] Step 2 includes:

[0022] Using the Wikipedia entries obtained in Step 1 as candidate words and the Wikipedia corpus from July 2020 as the corpus, the candidate words are traversed. For each word, the description and the first sentence of the Wikipedia page are extracted accordingly. The resulting set of text and symbols serves as the basic corpus. Since Wikipedia corpora possess certain textual features, such as a structure like [headword – description – segment description], overlapping relationships can be automatically constructed according to this grammatical structure. In this step, a data structure of <entry, entry description, segment title, segment text> will be obtained.

[0023] Step 3 includes: manually setting a syntactic template, the form of which is: <sentence; headword: {start position of headword, end position of headword}, ​​part of speech of headword, entity word 1: part of speech of entity word 1, entity word 2: part of speech of entity word 2…>. The descriptive portion and the first sentence of the main text from the basic corpus obtained in Step 2 are extracted as candidate texts. The texts are then tagged with part of speech using a part-of-speech tagging tool. Texts with high syntactic template matching scores are considered valid. The syntactic template matching score D is calculated using the following formula:

[0024] Where A0 and A1 are the start and end positions of the headword in the syntactic template, respectively; D0 and D1 are the start and end positions of the term name in the text, respectively; θ is the head entity setting ratio value; n is the number of entity words; and e i Let e′ be the category of the i-th entity.i Let D be the category of the i-th entity in the syntactic template. A value greater than 0.5 indicates a high syntactic template matching degree.

[0025] Step 4 includes:

[0026] Anchor text is a sequence of words in the Wikipedia corpus that is annotated in the form of hyperlinks. This part of the word sequence is directly used as candidate entities.

[0027] Step 5 includes extracting non-anchor text entities, which refer to entities that may exist in the word sequence outside the anchor text in a sentence. The method for extracting non-anchor text entities includes: labeling the sentences in the data obtained in Step 3 using a named entity recognition tool, statistically analyzing the frequency of entity types to show a long-tail distribution, ensuring that the benefit of each sample decreases as the number of samples increases; and adjusting the dataset using a statistical reweighting method to better facilitate subsequent training by ensuring a more balanced distribution of head and long-tail entities. Specifically, the samples are classified according to the combination of entity types in the sentence to obtain the sample categories, and then the data distribution is adjusted using a Box-Cox transformation (a generalized power transformation method).

[0028] Step 6 includes:

[0029] Step 6-1: Read the input string;

[0030] Step 6-2, mention detection: Decompose the input text according to the n-gram (n-gram model), and obtain mentions based on the entity word n-gram mapping obtained in step 1;

[0031] Step 6-3: Using Wikidata, Wikialias, and EDA (English Data Augmentation) tools, candidate entities are generated from the mentions generated in Step 6-2. The corresponding IDs of the entities are queried in the Wikidata knowledge graph. The generated data format is: [Correct entity and its ID; Candidate entity 1; Candidate entity 2; Candidate entity 3; ...];

[0032] Step 6-4: Entity disambiguation based on Wikidata knowledge graph: The candidate entities, correct entities, and sentences obtained from syntactic template matching in Step 3 are fed into a BERT-based entity disambiguation model in the character sequence format of [word sequence; correct entity; candidate entity 1; candidate entity 2; candidate entity 3; ...]. The input to the entity disambiguation model is [sentence word sequence; mention]. After passing through the BERT model (an existing large-scale pre-trained model), the first vector among the 512 output numerical vectors is taken as the mention vector representation. The first vector is matched with the vectors used in the clustering process in Step 1 and the numerical vectors of the obtained candidate entities to obtain the relationship matrix between the candidate entities and the input mentions.

[0033] Step 6-5: Based on the relation matrix, select the entity corresponding to the row with the highest probability score as the correct entity.

[0034] Step 7 includes: based on the assumption of remote supervision, labeling the relationships between the correct entities obtained in Step 6. When a relationship exists between two entities in the Wikidata knowledge graph, it is determined that the semantic relationship expressed by the sentence conforms to the corresponding relationship existing in the Wikidata knowledge graph. Specifically, it includes the following steps:

[0035] Step 7-1: Take the candidate entities obtained in step 4 and the sentence obtained in step 5 after the Box-Cox operation as the input sequence, and obtain the numerical vector representing the entity through a layer of neural network.

[0036] Step 7-2: Take the sentences as the input sequence and obtain the numerical vectors representing the sentences through a layer of neural network;

[0037] Step 7-3: Calculate the convex combination result of the numerical vectors representing entities and the numerical vectors representing clauses using dot product attention, and predict the probability of relation classification through a softmax layer.

[0038] Step 7-4: Take the line with the highest probability as the relationship between the entity and the clause.

[0039] The present invention also includes step 8:

[0040] Step 8-1: After steps 1 to 7, a batch of coarse data results are obtained;

[0041] Step 8-2: Design a query function to sample the labeled data. This invention uses the method based on the lowest confidence level to sample the data, that is, to select the sample with the lowest probability for labeling, and to sample 10% of the data to check the accuracy.

[0042] Step 8-3: Re-submit the manually labeled data into the process of steps 1 to 8. The process terminates when the accuracy of the sampled data is greater than 95%.

[0043] Beneficial effects: From a technical perspective, the present invention has the following advantages:

[0044] (1) The existing methods are integrated and coordinated in the entity linking model, disambiguation data is accumulated, and the accuracy of entity linking is improved.

[0045] (2) The introduction of human interaction into the relation extraction model improves data quality.

[0046] (3) It redefines the type of overlapping relationship data, provides a dataset that can be used for research, and lays the data foundation for the study of overlapping relationships.

[0047] From an application perspective, this invention has the following advantages:

[0048] (1) It can be used to construct relation extraction datasets for other scenarios.

[0049] (2) In the human-computer interaction process, the efficiency matching between automation technology and human labor is optimized efficiently.

[0050] (3) The method is simple, reproducible and scalable. Attached Figure Description

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0052] Figure 1 This is a flowchart of the method of the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the implementation of the entity linking scheme.

[0054] Figure 3 This is a schematic diagram of a relation classification (extraction) model.

[0055] Figure 4 This is a schematic diagram of the entity link model.

[0056] Figure 5 This is a schematic diagram combining model prediction and manual annotation.

[0057] Figure 6 This is a diagram illustrating the SEO relationship type structure.

[0058] Figure 7 This is a schematic diagram of the EPO relationship type structure.

[0059] Figure 8 This is a schematic diagram of the ELO relation type structure. Detailed Implementation

[0060] This invention includes the following steps:

[0061] Step 1: Construct a domain dictionary;

[0062] Step 2, obtain the text;

[0063] Step 3: Segment and assemble the data;

[0064] Step 4: Add the anchor text to the candidate entity set;

[0065] Step 5: Extract non-anchor text entities from the text and adjust the data distribution of the samples.

[0066] Step 6: Query the entity ID of the entity in the Wikipedia knowledge graph, where the ID serves as the unique identifier of the entity;

[0067] Step 7: Query the relationships between entities and use the relationships to annotate the clauses.

[0068] Step 1 includes: using a named entity recognition tool to perform part-of-speech tagging, extracting words related to the dataset domain, wherein the dataset (the target dataset constructed in this invention) is the target domain overlap relationship extraction task dataset to be constructed.

[0069] Step 1 specifically includes: obtaining domain text of the target domain, segmenting the text into sentences using commas as delimiters, counting the frequency of n-grams (N-gram model) in the obtained sentences, and taking the n-grams with the highest frequency among all n-grams (X1) as the candidate set of entries for the dictionary; the value of X1 is generally 10%.

[0070] The obtained n-grams are fed into a BERT model pre-trained on Wikipedia to obtain the first type of word vectors. At the same time, the entries in the Wikipedia database are also fed into the BERT model to obtain the second type of word vectors. The two types of word vectors are clustered together. After selecting the Wikipedia database entries as the cluster centers, the n-gram distribution centered on the Wikipedia entries is obtained. The n-grams of the Wikipedia central entries and their surroundings are manually filtered to obtain the final domain dictionary. The domain dictionary contains a mapping relationship: a one-to-many relationship between the Wikipedia database entries and the aforementioned n-grams.

[0071] Step 2 includes:

[0072] Using the Wikipedia entries obtained in Step 1 as candidate words and the Wikipedia corpus from July 2020 as the corpus, the candidate words are traversed. For each word, the description and the first sentence of the Wikipedia page are extracted accordingly. The resulting set of text and symbols serves as the basic corpus. Since Wikipedia corpora possess certain textual features, such as a structure like [headword – description – segment description], overlapping relationships can be automatically constructed according to this grammatical structure. In this step, a data structure of <entry, entry description, segment title, segment text> will be obtained.

[0073] Step 3 includes: manually setting a syntactic template, the form of which is: <sentence; headword: {start position of headword, end position of headword}, ​​part of speech of headword, entity word 1: part of speech of entity word 1, entity word 2: part of speech of entity word 2…>. The descriptive portion and the first sentence of the main text from the basic corpus obtained in Step 2 are extracted as candidate texts. The texts are then tagged with part of speech using a part-of-speech tagging tool. Texts with high syntactic template matching scores are considered valid. The syntactic template matching score D is calculated using the following formula:

[0074] Where A0 and A1 are the start and end positions of the headword in the syntactic template, respectively; D0 and D1 are the start and end positions of the term name in the text, respectively; θ is the head entity setting ratio value; n is the number of entity words; and e i Let e′ be the category of the i-th entity. i Let D be the category of the i-th entity in the syntactic template. A value greater than 0.5 indicates a high syntactic template matching degree.

[0075] Step 4 includes:

[0076] Anchor text is a sequence of words in the Wikipedia corpus that is annotated in the form of hyperlinks. This part of the word sequence is directly used as candidate entities.

[0077] Step 5 includes extracting non-anchor text entities, which refer to entities that may exist in the word sequence outside the anchor text in a sentence. The method for extracting non-anchor text entities includes: labeling the sentences in the data obtained in Step 3 using a named entity recognition tool, statistically analyzing the frequency of entity types to show a long-tail distribution, ensuring that the benefit of each sample decreases as the number of samples increases; and adjusting the dataset using a statistical reweighting method to better facilitate subsequent training by ensuring a more balanced distribution of head and long-tail entities. Specifically, the samples are classified according to the combination of entity types in the sentence to obtain the sample categories, and then the data distribution is adjusted using a Box-Cox transformation (a generalized power transformation method).

[0078] Step 6 includes:

[0079] Step 6-1: Read the input string;

[0080] Step 6-2, mention detection: Decompose the input text according to the n-gram (n-gram model), and obtain mentions based on the entity word n-gram mapping obtained in step 1;

[0081] Step 6-3: Using Wikidata, Wikialias, and EDA (English Data Augmentation) tools, candidate entities are generated from the mentions generated in Step 6-2. The corresponding IDs of the entities are queried in the Wikidata knowledge graph. The generated data format is: [Correct entity and its ID; Candidate entity 1; Candidate entity 2; Candidate entity 3; ...];

[0082] Step 6-4: Entity disambiguation based on the Wikidata knowledge graph: The candidate entities and correct entities generated in Step 6-3, along with the sentence obtained from syntactic template matching in Step 3, are fed into the BERT-based entity disambiguation model in the format of [word sequence; correct entity; candidate entity 1; candidate entity 2; candidate entity 3; ...] for computation. The computation process in the model is as follows: Figure 3 The input to the entity disambiguation model is the word sequence of the sentence and the mention. After passing through the BERT model (an existing large-scale pre-trained model), the first vector of the 512 output numerical vectors is taken as the vector representation of the mention. The first vector is matched with the vector used in the clustering process in step 1 and the numerical vector of the candidate entity to obtain the relationship matrix between the candidate entity and the input mention.

[0083] Step 6-5: Based on the relation matrix, select the entity corresponding to the row with the highest probability score as the correct entity.

[0084] Step 7 includes: based on the assumption of remote supervision, labeling the relationships between the correct entities obtained in Step 6. When a relationship exists between two entities in the Wikidata knowledge graph, it is determined that the semantic relationship expressed by the sentence conforms to the corresponding relationship existing in the Wikidata knowledge graph. The specific method is as follows: Figure 4 Specifically, it includes the following steps:

[0085] Step 7-1: Take the candidate entities obtained in step 4 and the sentence obtained in step 5 after the Box-Cox operation as the input sequence, and obtain the numerical vector representing the entity through a layer of neural network.

[0086] Step 7-2: Take the sentences as the input sequence and obtain the numerical vectors representing the sentences through a layer of neural network;

[0087] Step 7-3: Calculate the convex combination result of the numerical vectors representing entities and the numerical vectors representing clauses using dot product attention, and predict the probability of relation classification through a softmax layer.

[0088] Step 7-4: Take the line with the highest probability as the relationship between the entity and the clause.

[0089] The present invention also includes step 8:

[0090] Based on the use of relation prediction models to predict semantic relations in sentences, human interaction is introduced to improve annotation efficiency.

[0091] The specific process is as follows: Figure 5 :

[0092] Step 8-1: After steps 1 to 7, a batch of coarse data results are obtained;

[0093] Step 8-2: Design a query function to sample the labeled data. This invention uses the method based on the lowest confidence level to sample the data, that is, to select the sample with the lowest probability for labeling, and to sample 10% of the data to check the accuracy.

[0094] Step 8-3: Re-submit the manually labeled data into the process of steps 1 to 8. The process terminates when the accuracy of the sampled data is greater than 95%.

[0095] This solution, based on the current dataset for extracting overlapping relationships, proposes a more precise definition of overlapping relationships, introduces circular relationships for the first time, and quickly constructs a dataset for this purpose. As shown in Table 1:

[0096] Table 1

[0097] Relationship type illustrate SEO Entities overlap, but relationships do not. EPO Both the beginning and end entities overlap, and their relationships overlap. ELO The relationships overlap to form a ring.

[0098] The implementation scheme of this invention is as follows:

[0099] Using existing wiki data as seed data, entities and entity tags from the wiki data are stored in a MongoDB database as key-value pairs. Entity tagging is performed on the text using an entity linking model, with all entities originating from the wiki entities stored in the database. Subsequently, relationships already existing in the wiki data are mined from the data. Since a wiki page itself is an entity, as long as the page's entity is used as the head entity, the tagged entities will have a high probability of forming relationships with the head entity. The implementation of circular relationship extraction is based on this assumption. The entire technical solution involves technological innovations in two key areas.

[0100] Specific procedures (such as) Figure 1 ):

[0101] (1) Based on the selected vertical domain, a lexicon is constructed using both automated and manual methods. The automated approach involves using NER (Neural Evidence Recognition) tools for part-of-speech tagging to extract words with domain-related parts of speech. The extracted data structure is a structured triple of [sentence; part of speech; <tag start identifier, tag end identifier>]. First, the target domain text is obtained and segmented into sentences using commas as delimiters. The frequency of n-grams in the resulting sentences is statistically analyzed, and the n-grams with higher frequencies are selected as candidate entries for the lexicon. The obtained n-grams are then fed into a pre-trained BERT model on Wikipedia to obtain word vectors. Simultaneously, Wikipedia entries are also fed into the BERT model to obtain word vectors. The two types of word vectors are clustered together, with Wikipedia entries as the center. The resulting n-grams are then manually filtered to obtain the final lexicon. This lexicon contains a mapping relationship between Wikipedia entries with existing manually annotated knowledge and n-grams derived from statistical methods in the specific domain text.

[0102] (2) Some words in the dictionary exist on Wikipedia pages, while others do not. All words with existing Wikipedia pages are filtered out to obtain the Wikipedia page text content, which is stored in the database according to the structure of [id; word; text]. Furthermore, using the Wikipedia entries obtained in step 1 as terms, and the Wikipedia corpus from July 2020 as the corpus, the descriptions and the first paragraph of the main text of Wikipedia pages are extracted as the basic corpus. Since Wikipedia corpora possess certain textual characteristics, such as a structure like [headword – description – segment description], overlapping relationships can be automatically constructed according to this grammatical structure. In this step, a data structure of <term, term description, segment title, segment text> will be obtained.

[0103] If the search reveals that the term does not exist on a wiki page, then it is necessary to determine whether there is a fuzzy match through search query technology. Here, this solution uses Elasticsearch.

[0104] (3) Since the Wikipedia page corresponding to each entry is quite long, which is not conducive to model training, text compression is required in this step. In this scheme, the text compression method is as follows: a syntactic template is manually set, and the template form is as follows: <sentence; headword: {start position of headword, end position of headword}, ​​part of speech of headword, entity word 1: part of speech of entity word 1, entity word 2: part of speech of entity word 2…>. The descriptive part and the first sentence of the main text in the corpus are extracted as candidate texts, and the part of speech is tagged using a part-of-speech tagging tool. Texts with high template matching degree are considered valid texts. The matching degree here is represented by the following formula:

[0105] Template matching degree: Where A0 and A1 are the start and end positions of the central word in the template, D0 and D1 are the start and end positions of the word name in the text, θ is the head entity setting ratio, and n is the number of entity words. When the value of D is greater than 0.5, the sentence is considered to match the template. The sentences are added to a sequence stack of a set length. Matching stops when the sequence length exceeds the stack capacity.

[0106] (4) Anchor text is a page link in Wikipedia that has been manually annotated. It has a corresponding entity in wikidata. After extracting the anchor text, the corresponding entity number is found through the link and used as a candidate entity.

[0107] (5) Having only anchor text entities is far from sufficient; it is also necessary to extract effective entities from the text. This scheme adopts a sequence labeling + retrieval method to extract non-anchor text entities from the text, while adjusting the data distribution of the samples. Specifically, the named entity recognition tool is used to classify the sentence text, and the frequency of entity types is statistically analyzed to show a long-tail distribution, so that the benefit of each sample decreases as the number of samples increases. In this step, a statistical reweighting method is used to adjust the dataset so that the distribution of head entities and long-tail entities can better contribute to subsequent training. Specifically, the samples are classified according to the combination of entity types in the sentence to obtain the category of the sample. Then, the Box-Cox transformation is used to adjust the data distribution.

[0108] (6) After obtaining the entity's literal token, it is necessary to query the corresponding entity ID in the wiki domain. This process is mainly accomplished using entity linking technology. The technology used in this part will be explained in detail in the following section on entity linking.

[0109] (7) After entity annotation is completed, the relationships between entities are annotated based on the assumption of remote supervision. Since Wikipedia-like text encyclopedias have high content and low ambiguity in describing facts, the accuracy of remote supervision is relatively high in the relationship annotation process, and it is easy to obtain large-scale effective data. The interaction with humans in remote supervision will be discussed in detail in (4.2.2).

[0110] Entity linking model:

[0111] In the entity linking model, a combination of disambiguation tables and the model is primarily employed. A disambiguation table is a method that uses a static table to establish a mapping between text mentions and entity symbols. Based on the method described in the paper, a basic disambiguation table was extensively searched in Wikipedia in July 2021 and used as the foundation for this approach. To enhance the linking model's error correction capabilities, this approach further innovates the disambiguation table by using EDA augmentation methods to improve its performance in mention detection.

[0112] The second step in the entity linking model is entity disambiguation, which involves selecting the correct entity based on the context and information of the input text when the same text mentions multiple entities. The experimental section of the paper "Zexuan Zhong, Danqi Chen. 2020. A Frustrating Easy Approach for Entity and Relation Extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies" suggests that the type information of the mention plays a crucial role in the disambiguation process. Inspired by this experimental result, this scheme introduces type information from Wikipedia (which is performed by a simple classifier with an accuracy of 0.98) into the entity disambiguation stage. Therefore, the entity disambiguation model in this scheme actually requires two models to work together.

[0113] Specific steps (in combination) Figure 2 ):

[0114] (1) Read the input string in the user command line (or file reading) interface.

[0115] (2) Combine named entity recognition tools, mention correction models and user dictionaries built in the early stage to perform mention detection, which serves as seed objects for candidate entity generation.

[0116] (3) Using wikidata, wiki aliases, and EDA tools, candidate entities are generated from the mentions generated in the previous step. The generated data format is [correct entity; candidate entity 1; candidate entity 2; candidate entity 3; ...].

[0117] (4) Entity disambiguation based on the wikidata knowledge base. The specific approach is as follows: The candidate entities and correct entities generated in the early stage are fed into the BERT model in the format of [word sequence; correct entity; candidate entity 1; candidate entity 2; candidate entity 3; ...]. After calculation by the parameters in the model, the score of each candidate entity is output.

[0118] (5) Rank the entities according to the model prediction results and select the entity with the highest ranking as the correct entity.

[0119] Efficiently incorporating human knowledge into remote supervision:

[0120] After entity linking is completed, the next task is relation labeling. Under the remote supervision hypothesis, when a relationship exists between two entities in the Wikidata knowledge base, the semantic relationship expressed in the sentence is determined to be the same as the relationship expressed in the Wikidata knowledge base. The advantage of this approach is its ability to quickly label relationships on a large scale, but the disadvantage is that the labeling results contain a significant amount of noise. In this scheme, the accuracy of remote supervision is relatively high when using Wikidata because Wikidata is manually annotated encyclopedic data, and the semantic relationships are mostly encyclopedic factual relationships with relatively low ambiguity. If other types of data are used as input, such as literary data, the ambiguity will increase significantly, requiring the introduction of a manual noise reduction process. To enhance the robustness of this scheme, a manual interactive noise reduction scheme based on the idea of ​​offline learning is designed at this stage.

[0121] This scheme defines a multi-stage zero-shot relation classification task. In this task, the model can automatically label the corpus, while combining this with a small amount of manual screening to ultimately form a high-quality dataset. As shown in Table 2:

[0122] Table 2

[0123] Relationship type Number of instances Limit the number of instances of 40 relations. SEO 71650 55139 EPO 74957 45323 ELO 5076 5076

[0124] SEO relationship type structure such as Figure 6 As shown; the EPO relationship type structure is as follows Figure 7 As shown; the ELO relation type structure is as follows Figure 8 As shown;

[0125] The specific implementation is as follows:

[0126] (1) A portion (10%) of the mislabeled data was manually selected and trained as a classifier in batches. The classifier mainly predicts the mislabeled samples.

[0127] (2) An uncertainty sampling method is adopted to select the smallest sample with the highest probability for labeling. The manually labeled data and the data automatically predicted by the model are adjusted according to a set ratio and iteratively trained.

[0128] (2) Calculate the accuracy rate based on the results of manual sampling. Once the accuracy rate reaches 0.95 or higher, the iteration of the above eight steps is considered to be over.

[0129] Example

[0130] The task requires building a dataset of overlapping relationships within specific sub-fields of sports. Follow the workflow:

[0131] (1) Use web crawlers to obtain a large amount of sports-related text.

[0132] (2) Perform n-gram statistics on the obtained text. For example: “Someone [anchor text] (date -), ancestral home in a certain place, born in a certain place [anchor text], is a retired famous professional football player in a certain country [anchor text], and former captain of a certain team.” Through n-gram statistics, entities such as “someone, a certain place, football player, a certain team” can be obtained.

[0133] (3) By constructing a dictionary, a mapping dictionary is established, such as [[professional football player]->football player, [a national team->a football team->a team]];

[0134] (4) Set the extraction template. In SEO relationships, the data template mainly consists of {head entity, [0,3], tail entity category [nationality|person|organization]}, etc. A suitable sentence is collected through a round of template matching calculation. In this sentence, the template matching degree is 0.66.

[0135] (5) At this point, the anchor text "a person", "a place", "a country" and its type are added to the candidate entity set. Then, the NER tool is used to obtain the entities and their types: athlete (person), football player (person), national team (organization), captain (person). Subsequently, the data is adjusted according to the entity types contained in the sentence. This problem is modeled as a data balancing problem in a multi-class scenario. Here, the Box-Cox transformation is directly used.

[0136] (6) Query the entity ID of a person in the Wikipedia knowledge graph: Q36582. The same logic applies to other entities. Two situations may occur: either no ID is found, or multiple IDs are found. The first situation is an N / A type entity determination problem, and the second situation is an entity disambiguation problem. This invention addresses this problem using an entity linking model. Taking the mention of "national team" as an example, the data input into the entity linking model is ["A person [anchor text] (date -), ancestral home in a certain place, born in a certain place [anchor text], is a retired famous professional football player from a certain country [anchor text], former captain of the national team.", National Team]. Candidate entities for "national team" could be "national football team of a certain country, national women's football team of a certain country, national table tennis team of a certain country, national team of a certain country".

[0137] After BERT calculations, a 256-bit numerical vector is obtained. This vector is matched with the numerical vectors generated during dictionary construction in step one. It is found that the dot product of the numerical vector corresponding to the entity "National Football Team of a Certain Country" is the highest, thus correctly linking it to this entity. For N / A types, the model can directly classify the N / A category label. 6. Relation Classification

[0138] After the previous prediction yielded the entity pair "[a ​​person, a national football team]", the Wikipedia knowledge graph already contains the relationships "[a person is from a certain country, a person is the captain of a certain national football team]". At this point, the data can be automatically labeled with the "nationality" and "captain" relationships. The next step is to ensure the relationship classification model learns these relationships, as it will be used to predict other data.

[0139] (7) Combining human and machine prediction. After learning from the data on players and national teams, the model has mastered the ability to predict the relationship between "player, country, national team". At this point, if a new batch of data or new annotations appear, the automated prediction will be biased. For example, if "Zhang went to watch a game of a certain country's national team", because Zhang's entity type and the national team's entity class are very similar to the sentence "someone", the model will tend to predict Zhang as a player of that country's national team, thus making an error. This invention uses the method of selecting the sample with the lowest probability for annotation to collect potentially erroneous data. If the probability of the first data being predicted as "national team member" is 0.9, and the probability of the second data being predicted as that class is only 0.5, then the data with the lower probability is selected as the annotation data.

[0140] (8) Iteration. After the above steps are completed, a batch of data will be sampled from the generated data to check the accuracy. When the accuracy is higher than 0.95, the entire process ends and the data production is complete.

[0141] This invention provides a method for rapidly constructing overlapping relationships to extract text data. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for quickly constructing an overlap relation extraction text data, characterized in that, The method comprises the following steps: Step 1, constructing a field dictionary; Step 2, obtaining text; Step 3, dividing sentences and assembling data; Step 3 includes: setting a syntax template, intercepting the description part and the first sentence of the text of the body text in the basic corpus obtained in step 2 as a candidate text, marking the text with a part-of-speech tagging tool, determining that it is a valid text if the syntax template matching degree is high, and calculating the syntax template matching degree D through the following formula: wherein A0, A1 are the start position and end position of the center word in the syntactic template, D0, D1 are the start position and end position of the occurrence of the word entry name in the text, θ is a proportion value set for the head entity, n is the number of entity words, e i is the category of the i-th entity, e ′ i is the category of the i-th entity in the syntactic template; Step 4, adding anchor text to the candidate entity set; Anchor text is a sequence of words annotated in the form of a hyperlink in the Wikipedia corpus, and this sequence of words is directly used as a candidate entity; Step 5, extracting non-anchor text entities in the text while adjusting the data distribution of the sample; Non-anchor text entities refer to the entities that may exist in the sequence of words outside the anchor text in the sentence, Step 6, querying the entity number of the entity in the Wikipedia knowledge graph, which is used as the unique identifier of the entity; Step 7, querying the relationship between entities and annotating the relationship between the divided sentences.

2. The method of claim 1, wherein, Step 1 includes: using a named entity recognition tool to perform part-of-speech tagging, and extracting the words related to the data set field.

3. The method of claim 2, wherein, Step 1 specifically includes: obtaining field text of the target field, dividing the text by commas, and counting the n-gram frequency in the obtained divided sentences; taking the n-gram with a frequency in the top X1 among all n-grams as the n-gram of the candidate set of the dictionary; The obtained n-gram is input into the bert model pre-trained on Wikipedia to obtain the first type of word vector, and the entries in the Wikipedia database are also input into the bert model to obtain the second type of word vector; The two types of word vectors are clustered together, and the n-gram distribution centered on the Wikipedia entries is selected after clustering, and the final field dictionary is obtained by filtering the n-gram around the Wikipedia center entries.

4. The method of claim 3, wherein, Step 2 includes: Using the Wikipedia entries obtained in step 1 as candidate words, and using the Wikipedia corpus as the corpus, traverse the candidate words, and correspondingly grab the description and the first sentence of the text of the Wikipedia page to obtain the text and symbol set as the basic corpus.

5. The method of claim 4, wherein, Step 5 includes: the method for extracting non-anchor text entities includes: using a named entity recognition tool to tag the sentences in the data obtained in step 3, counting the frequency of entity types, and presenting a long-tail distribution, so that with the increase of the number of samples in the data set, the benefit brought by each sample tends to decrease, and the data set is adjusted by using statistical reweighting method. The specific method is: classifying the samples according to the combination of entity types in the sentence to obtain the categories of the samples, and then adjusting the data distribution using Box-Cox transformation.

6. The method of claim 5, wherein, Step 6 includes: Step 6-1, reading the input string; Step 6-2, mention detection: the input text is disassembled in the form of n-gram, and the mentions are obtained according to the entity word n-gram mapping obtained in step 1; Step 6-3, combining the Wikidata Wikipedia knowledge graph, wiki alias Wikipedia alias and EDA English data enhancement tool to generate candidate entities for the mentions generated in step 6-2, and querying the entity number in the Wikipedia knowledge graph, the generated data format is 【correct entity and its number; candidate entity 1; candidate entity 2; candidate entity 3;...】; Step 6-4, entity disambiguation based on wikidata: the candidate entities, correct entities generated in step 6-3 and the sentences matched according to the syntactic template in step 3 are sent to the BERT-based entity disambiguation model in the format of 【character sequence of word sequence; correct entity; candidate entity 1; candidate entity 2; candidate entity 3;...】; the input of the entity disambiguation model is 【word sequence of sentence; mention】; after the bert model, the first vector of the 512 numerical vectors in the output is taken as the vector representation of the mention; the first vector is matched with the vector obtained in the clustering process in step 1 and the numerical vector of the candidate entity to obtain the relationship matrix between the candidate entity and the input mention; Step 6-5, according to the relationship matrix, the entity corresponding to the row with the highest probability score is selected as the correct entity.

7. The method of claim 6, wherein, Step 7 includes: based on the assumption of remote supervision, the relationship between the correct entities obtained in step 6 is labeled, and when there is a relationship between two entities in wikidata, it is determined that the semantic relationship expressed by the sentence conforms to the corresponding relationship existing in the wikidata, which specifically includes the following steps: Step 7-1, the candidate entities obtained in step 4 and the sentence after Box-Cox operation obtained in step 5 are input sequences, and the numerical vector representing the entity is obtained through a layer of neural network; Step 7-2, the sentence is taken as the input sequence, and the numerical vector representing the sentence is obtained through a layer of neural network; Step 7-3, the numerical vector representing the entity and the numerical vector representing the sentence are combined in the form of dot product attention, and the probability of relationship classification is predicted through a layer of softmax; Step 7-4, the row with the maximum probability is taken as the relationship between the entity and the sentence.

8. The method of claim 7, wherein, Also includes step 8: Step 8-1, after steps 1-7, a batch of rough labeled data results are obtained; Step 8-2, design a query function to sample the labeled data, use the method based on the lowest confidence to sample the data, that is, select the sample with the smallest maximum probability to label, and sample ten percent of the data to check the accuracy; Step 8-3, the labeled data is sent back to the process of steps 1-8, and when the accuracy of the sampled data is greater than 95%, the process is terminated.

Images