A bootstrap relation extraction method and system based on metric learning

Abstract

The application discloses a bootstrap relation extraction method based on metric learning, comprising the following steps: extracting all entity pairs contained in a seed set and extracting all instances containing the entity pairs in an unlabeled corpus to form a first candidate set; measuring the first similarity of each instance in the first candidate set and all instances in the seed set sharing the same entity pair with the instance based on a pre-trained multi-input network; adding the instance in the first candidate set with the first similarity higher than a first threshold to the seed set; predicting the relation of the unlabeled corpus based on a pre-trained relation binary classifier and generating the second similarity of each instance, extracting the instance with the second similarity higher than a second threshold to form a second candidate set; measuring the third similarity of each instance in the second candidate set and all instances in the seed set based on the pre-trained multi-input network; and adding the instance in the second candidate set with the third similarity higher than a third threshold to the seed set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing in intelligent manufacturing, and particularly relates to a self-boosting relation extraction method based on metric learning and a system thereof. BACKGROUND

[0002] Relation facts, as an important part of human knowledge, are an important way to describe the logical connection between entities. With the advent of the IT3.0 era, people have new requirements for entity relation extraction. Because unstructured text often contains a large amount of semantic knowledge, and is relatively efficient and real-time. However, the syntactic structure of unstructured text is also flexible and variable, and the regularity is not strong. Therefore, how to accurately and perfectly extract the triplets with specific semantic relations from an unstructured text has always been a research hotspot in the field of knowledge graph. Similar to the named entity recognition task, the research of the relation extraction task has also experienced three stages: the rule-based method, the statistical-based method and the deep learning-based method.

[0003] The rule-based method identifies the syntactic element features in the text by using a syntactic analysis tool on the basis of a large amount of corpus and templates, and then automatically constructs pattern rules according to these features. However, due to the errors in the automatically constructed patterns, most of these methods need further inspection and adjustment by language experts. The statistical-based method is further divided into two types according to the differences in feature extraction focus: the feature vector-based method and the kernel function-based method. The former focuses on extracting various shallow features from the text sequence and its syntactic structure. Compared with the rule-based method, this kind of method has better coverage and less need for human supervision. Unfortunately, the carefully designed and extracted features and kernel functions may not capture all the effective information, and the dependence on existing natural language processing tools may cause the error accumulation problem of the downstream modules.

[0004] Therefore, current researchers are committed to automatically constructing text features through deep learning, thereby breaking the bottleneck of manually constructing features. According to the number and openness of the corpus and the relationship set, the deep learning-based method is divided into the following four types: supervised relation extraction, distant supervision relation extraction, bootstrapping relation extraction and unsupervised relation extraction. ① Supervised relation extraction automatically learns the syntactic and semantic features in a large-scale corpus through deep neural networks, and then performs relation extraction of a limited target relationship set. Generally, it can achieve relatively high accuracy and recall rate, but it is difficult to expand to new fields due to the high cost of corpus annotation. ② Distant supervision relation extraction assumes that the same entity has consistent relationship information in different texts, and automatically mines new relationships through entity alignment between the corpus and the knowledge graph. This method reduces the dependence of the model on manually annotated corpora, and provides the possibility for large-scale cross-domain relation extraction. ③ Bootstrapping relation extraction focuses on using a small amount of labeled instances as an initial seed set, then learning new templates, and then extracting new triples based on old and new templates and expanding the seed set. Through continuous iteration, new potential relationship triples are found and discovered from unstructured data. ④ Unsupervised relation extraction extracts the relationship text between entities through clustering and other methods, and then quickly extracts a very large number of relationship pairs, but the extraction results are often difficult to align with the knowledge base.

[0005] It can be seen that at present, a large number of researches are excavated in the field of relation extraction to improve the performance of the model on this task or expand the corresponding application scenario. The current field has made certain progress, laying a good foundation for subsequent automatic construction of knowledge graph and user behavior analysis. However, when the traditional relation extraction method is applied to the field of intelligent manufacturing, due to the fact that the related labeled corpus for intelligent manufacturing is too scarce, it is difficult to meet the precision requirements of supervised relation extraction.

[0006] In summary, the traditional relation extraction method applied to the field of intelligent manufacturing has the problem of low relation extraction precision. SUMMARY

[0007] Therefore, the present application provides a bootstrapping relation extraction method based on metric learning, which improves the processing and extraction method of data, solves the problem of low relation extraction precision of the traditional relation extraction method applied to the field of intelligent manufacturing.

[0008] To solve the above problems, the technical scheme of the present application is to adopt a bootstrap relation extraction method based on metric learning, which comprises: extracting all entity pairs contained in a seed set, and extracting all instances containing the entity pairs in an unlabeled corpus to form a first candidate set; measuring the first similarity of each instance in the first candidate set and all instances in the seed set that share the same entity pair with the instance based on a pre-trained multi-input network; adding the instances in the first candidate set with a first similarity higher than a first threshold to the seed set; performing relation prediction on the unlabeled corpus based on a pre-trained relation binary classifier to generate a second similarity of each instance, extracting instances with a second similarity higher than a second threshold to form a second candidate set; measuring the third similarity of each instance in the second candidate set and all instances in the seed set based on a pre-trained multi-input network; adding the instances in the second candidate set with a third similarity higher than a third threshold to the seed set.

[0009] Optionally, the method for pre-training the multi-input network comprises: constructing an initialized multi-input network; obtaining a training data set and a test data set composed of a corpus containing annotations; training and testing the multi-input network based on the training data set and the test data set.

[0010] Optionally, the pre-trained relation binary classifier comprises: training the relation binary classifier to represent the probability that each instance in the seed set represents a given relation.

[0011] Optionally, the multi-input network is configured as a triple network.

[0012] Optionally, the input sample of the triple network is configured to include positive relation samples, negative relation samples and benchmark relation samples, and the similarity measurement function of the triple network is iteratively optimized by combining the feature distance of the benchmark relation samples and the positive relation samples, the feature distance of the benchmark relation samples and the negative relation samples, to output accurate similarity prediction results.

[0013] Optionally, measuring the first similarity of each instance in the first candidate set and all instances in the seed set that share the same entity pair with the instance based on the pre-trained multi-input network comprises: for any instance in the first candidate set, extracting all similar instances in the seed set that share the same entity pair with the instance; measuring a plurality of initial similarities between the instance and each similar instance based on the pre-trained multi-input network; obtaining the first similarity of the instance by weighting a plurality of the initial similarities; repeating the above steps until all instances in the first candidate set are traversed to obtain the first similarity of all instances in the first candidate set.

[0014] Optionally, the measuring, by the pre-trained multi-input network, of the third similarity of each instance in the second candidate set and all instances in the seed set comprises: for any instance in the second candidate set, measuring, by the pre-trained multi-input network, a plurality of initial similarities of the instance and each instance in the seed set respectively; obtaining the third similarity of the instance by weighting the plurality of initial similarities; and repeating the above steps until all instances in the second candidate set are traversed to obtain the third similarity of all instances in the second candidate set.

[0015] Correspondingly, the application provides a bootstrap relation extraction system based on metric learning, comprising: a data storage unit configured to store a seed set, a pre-trained multi-input network and a relation binary classifier; a data processing unit configured to extract all entity pairs contained in the seed set, extract all instances containing the entity pairs in an unlabeled corpus to form a first candidate set, measure, by the pre-trained multi-input network, a first similarity of each instance in the first candidate set and all instances in the seed set sharing the same entity pair with the instance, add instances in the first candidate set with the first similarity higher than a first threshold to the seed set, perform relation prediction on the unlabeled corpus by the pre-trained relation binary classifier to generate a second similarity of each instance, extract instances with the second similarity higher than a second threshold to form a second candidate set, and then measure, by the pre-trained multi-input network, a third similarity of each instance in the second candidate set and all instances in the seed set, and add instances in the second candidate set with the third similarity higher than a third threshold to the seed set.

[0016] Optionally, the data storage unit further stores a training data set and a test data set composed of the annotated corpus, for training the multi-input network.

[0017] Optionally, the multi-input network is a triple network, and an input sample of the triple network is configured to include a positive relation sample, a negative relation sample and a benchmark relation sample, a similarity measurement function of the triple network is iteratively optimized by combining feature distances of the benchmark relation sample and the positive relation sample and the benchmark relation sample and the negative relation sample, to output an accurate similarity prediction result.

[0018] The primary improvement of the application is to provide a bootstrap relation extraction method based on metric learning, which measures the relations between the annotated corpus, the unlabeled corpus and the seed set of a given new relation by the pre-trained multi-input network and the relation binary classifier, extracts instances of the new relation from a query set containing existing relations, new relations and hidden relations, completes the extraction of bootstrap relations, and solves the problem of low relation extraction accuracy of traditional relation extraction methods applied in the field of intelligent manufacturing. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a simplified flowchart of the bootstrap relation extraction method based on metric learning of the present application;

[0020] Figure 2 is a simplified unit connection diagram of the bootstrap relation extraction system based on metric learning of the present application. DETAILED DESCRIPTION

[0021] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0022] As shown in Figure 1 , the present application provides a bootstrap relation extraction method based on metric learning, comprising:

[0023] S1: Extract all entity pairs contained in the seed set, and extract all instances containing the entity pairs in the unlabeled corpus to form a first candidate set C1.

[0024] The present application uses the idea of remote supervision. If different instances share the same entity pair, they are likely to express the same relationship. By extracting all entity pairs (e h , e t ) on the seed set Sr, then matching all instances x containing (e h , e t ) on the unlabeled corpus T, and then aggregating into a first candidate set for expanding the seed set Sr.

[0025] S2: Based on the pre-trained multi-input network, measure the first similarity of each instance in the first candidate set and all instances in the seed set that share the same entity pair with the instance.

[0026] However, since remote supervision has certain annotation error problems, in order to alleviate the annotation error problems brought by remote supervision, the present application sets S2, for each instance x in the first candidate set C1, finds all instances x' in the seed set Sr that share the entity pair (e h , e t ) with x, and based on the initial similarity between x and all x', the first similarity of instance x is obtained by weighting, thereby effectively alleviating the annotation error problems brought by remote supervision and improving the accuracy of relation extraction.

[0027] Further, the method for pre-training the multi-input network comprises: constructing an initialized multi-input network; obtaining a training data set and a test data set composed of a corpus containing annotations; training and testing the multi-input network based on the training data set and the test data set.

[0028] Further, the multi-input network is configured as a triplet network. The input examples of the triplet network are configured to include positive relation examples, negative relation examples and benchmark relation examples, and a similarity measurement function of the triplet network is iteratively optimized by combining feature distances between the benchmark relation examples and the positive relation examples, and the benchmark relation examples and the negative relation examples, to output an accurate similarity prediction result.

[0029] To facilitate understanding of the working principle of the multi-input network, an example is given that when the multi-input network is configured as a triplet network, a weight-shared encoder uniformly extracts semantic features of three types of input relation examples. Let the encoded positive relation examples, negative relation examples and benchmark relation examples be denoted as Input e , then after activation by a softmax activation function, the positive-benchmark feature distance d + and the negative-benchmark feature distance d - can be respectively represented as: wherein the distance measurement between examples is an L-K norm distance. Therefore, it can be known that when d + → 0, i.e., d - → 1, the benchmark example is constantly close to the positive example set and constantly far away from the negative example set, at which time the benchmark relation example tends to be classified as having the same relation as the positive example. Also, d + and d - are themselves representations in the probability sense, therefore, the similarity calculation formula between examples can be represented as Meanwhile, to further optimize the representation ability of the feature distance, the feature distance can be further derived as: d and a bias parameter b d

[0030]

[0031] Further, in the network pre-training process, for a corpus containing annotations , a batch of relation examples containing the same relation or different relations can be randomly extracted, and the network is trained by a triplet loss function. If S′ N is defined as a small batch of relation examples randomly extracted and encoded in a training round, then by appropriately extracting an example triplet of “2 positive examples + 1 negative example” or “2 negative examples + 1 positive example” in S′ N , training data with a batch size of |S′ N | can be constructed. If Input i ​Input j Input k For positive instances, benchmark instances and negative instances, the formula of the round model loss can be expressed as: Wherein, a is a regularization term. If the triplet network is trained sufficiently, i.e. d + →0 and d - →1, at this time, the interval between positive and negative instances depends on margin(a). In an ideal case, let a = 1, at this time, the loss function tends to 0, which also shows that using the triplet loss function for knowledge learning is effective.

[0032] Further, based on the pre-trained multi-input network, the first similarity of each instance in the first candidate set and all instances in the seed set sharing the same entity pair with the instance is measured, including: for any instance in the first candidate set, all similar instances in the seed set sharing the same entity pair with the instance are extracted; based on the pre-trained multi-input network, the plurality of initial similarities of the instance and each similar instance are respectively measured; the first similarity of the instance is obtained by weighting the plurality of initial similarities; the above steps are repeated until all instances in the first candidate set are traversed, and the first similarities of all instances in the first candidate set are obtained.

[0033] S3: the instances in the first candidate set with the first similarity higher than the first threshold value are added to the seed set to complete the expansion of the seed set, wherein the first threshold value is set by the user according to the project demand and other factors that affect the relationship extraction accuracy, and the present application is not limited in detail.

[0034] S4: based on the pre-trained relationship binary classifier, the relationship of the unlabeled corpus is predicted and the second similarity of each instance is generated, and the instances with the second similarity higher than the second threshold value are extracted to constitute a second candidate set, wherein the second threshold value is set by the user according to the project demand and other factors that affect the relationship extraction accuracy, and the present application is not limited in detail.

[0035] Further, the pre-trained relationship binary classifier includes: for a given relationship r of the seed set, the relationship binary classifier is trained to represent the probability of each instance in the seed set representing the given relationship.

[0036] The present application completes the expansion of the seed set by setting S1-S3, but the set of entity pairs (e h , e t ) in the current seed set Sr has not been expanded, therefore, the present application sets S4 to mine new entity pairs for the relationship r to expand the entity pairs in the seed set Sr.

[0037] S5: measuring third similarities between each instance in the second candidate set and all instances in the seed set based on the pre-trained multi-input network.

[0038] Further, measuring the third similarities between each instance in the second candidate set and all instances in the seed set based on the pre-trained multi-input network comprises: for any instance in the second candidate set, measuring a plurality of initial similarities between the instance and each instance in the seed set based on the pre-trained multi-input network respectively; obtaining the third similarity of the instance by weighting the plurality of initial similarities; and repeating the above steps until all instances in the second candidate set are traversed to obtain the third similarities of all instances in the second candidate set.

[0039] S6: adding instances in the second candidate set with third similarities higher than a third threshold to the seed set to complete the expansion of the seed set, wherein the third threshold is set by a user according to factors influencing the relationship extraction accuracy such as project requirements, and the present application does not make specific limitations.

[0040] The present application measures the relationships between the seed set containing the new relationship, the unlabeled corpus and the labeled corpus by the pre-trained multi-input network and the relationship binary classifier, extracts the instances of the new relationship from the query set containing the existing relationship, the new relationship and the hidden relationship, completes the bootstrap relationship extraction, and solves the problem of low relationship extraction accuracy of the traditional relationship extraction method applied in the field of intelligent manufacturing.

[0041] Correspondingly, as shown in Figure 2 The present application provides a bootstrap relationship extraction system based on metric learning, characterized in that it comprises: a data storage unit for storing a seed set, a pre-trained multi-input network and a relationship binary classifier; a data processing unit for extracting all entity pairs contained in the seed set, extracting all instances containing the entity pairs in an unlabeled corpus to form a first candidate set, measuring first similarities between each instance in the first candidate set and all instances in the seed set sharing the same entity pair with the instance based on the pre-trained multi-input network, adding instances in the first candidate set with first similarities higher than a first threshold to the seed set, performing relationship prediction on the unlabeled corpus based on the pre-trained relationship binary classifier to generate second similarities of each instance, extracting instances with second similarities higher than a second threshold to form a second candidate set, and further measuring third similarities between each instance in the second candidate set and all instances in the seed set based on the pre-trained multi-input network, and adding instances in the second candidate set with third similarities higher than a third threshold to the seed set.

[0042] Further, the data storage unit also stores a training data set and a test data set composed of an annotated corpus, for training the multi-input network.

[0043] Still further, the multi-input network is a triplet network, the input example of the triplet network is configured to include positive relation examples, negative relation examples and benchmark relation examples, by combining the feature distance between the benchmark relation examples and the positive relation examples, the benchmark relation examples and the negative relation examples, iteratively optimizing the similarity measurement function of the triplet network to output accurate similarity prediction results.

[0044] The above describes the self-booting relation extraction method and system based on metric learning provided by the embodiment of the present application. The embodiments in the description are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part. It should be pointed out that, for those skilled in the art, without departing from the principle of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0045] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

Claims

1. A bootstrap relation extraction method based on metric learning, characterized in that, include: Extract all entity pairs contained in the seed set, and extract all instances of the entity pairs contained in the unlabeled corpus to form the first candidate set; For any instance in the first candidate set, extract all similar instances in the seed set that share the same entity pair with that instance; The pre-trained multi-input network measures the initial similarity between the instance and each nearby instance. The first similarity of the instance is obtained by weighting multiple initial similarities; Repeat the above steps until all instances in the first candidate set have been traversed, and obtain the first similarity of all instances in the first candidate set; Instances from the first candidate set whose first similarity is higher than a first threshold are added to the seed set; Based on a pre-trained relation binary classifier, relation prediction is performed on the unlabeled corpus and a second similarity is generated for each instance. Instances with a second similarity higher than a second threshold are extracted to form a second candidate set. For any instance in the second candidate set, multiple initial similarities between the instance and each instance in the seed set are measured based on a pre-trained multi-input network. The third similarity of the instance is obtained by weighting multiple initial similarities; Repeat the above steps until all instances in the second candidate set have been traversed, and obtain the third similarity of all instances in the second candidate set; Instances from the second candidate set whose third similarity is higher than the third threshold are added to the seed set.

2. The bootstrap relation extraction method according to claim 1, characterized in that, Methods for pre-training multi-input networks include: Construct and initialize the multi-input network; Obtain the training dataset and test dataset, which consist of a corpus containing annotations; The multi-input network is trained and tested based on the training dataset and the test dataset.

3. The bootstrap relation extraction method according to claim 1, characterized in that, Pre-trained relation binary classifiers include: For a given set of seeds for a relation, a relation binary classifier is trained to characterize the probability that each instance in the seed set represents the given relation.

4. The bootstrap relation extraction method according to claim 2, characterized in that, The multi-input network is configured as a triplet network.

5. The bootstrap relation extraction method according to claim 4, characterized in that, The input samples of the triplet network are configured to include positive relation samples, negative relation samples, and baseline relation samples. By combining the feature distances between the baseline relation samples and positive relation samples, and between the baseline relation samples and negative relation samples, the similarity measurement function of the triplet network is iteratively optimized to output accurate similarity prediction results.

6. A bootstrap relation extraction system based on metric learning, characterized in that, include: The data storage unit is used to store the seed set, the pre-trained multi-input network, and the relation binary classifier; The data processing unit extracts all entity pairs contained in the seed set, extracts all instances containing the entity pairs in the unlabeled corpus to form a first candidate set, and for any instance in the first candidate set, extracts all similar instances in the seed set that share the same entity pairs with that instance; measures multiple initial similarities between the instance and each similar instance based on a pre-trained multi-input network; obtains the first similarity of the instance by weighting the multiple initial similarities; repeats the above steps until all instances in the first candidate set have been traversed to obtain the first similarity of all instances in the first candidate set; adds instances in the first candidate set whose first similarity is higher than a first threshold to the seed set; performs relation prediction on the unlabeled corpus based on a pre-trained relation binary classifier and generates a second similarity for each instance; extracts instances whose second similarity is higher than a second threshold to form a second candidate set; and then, for any instance in the second candidate set, measures multiple initial similarities between the instance and each instance in the seed set based on a pre-trained multi-input network. The third similarity of the instance is obtained by weighting multiple initial similarities; the above steps are repeated until all instances in the second candidate set are traversed, and the third similarity of all instances in the second candidate set is obtained. Instances in the second candidate set whose third similarity is higher than the third threshold are added to the seed set.

7. The bootstrap relation extraction system according to claim 6, characterized in that, The data storage unit also stores a training dataset and a test dataset consisting of an annotated corpus, used to train the multi-input network.

8. The bootstrap relation extraction system according to claim 6, characterized in that, The multi-input network is a triplet network. The input samples of the triplet network are configured to include positive relation samples, negative relation samples, and baseline relation samples. By combining the feature distances between the baseline relation samples and positive relation samples, and between the baseline relation samples and negative relation samples, the similarity measurement function of the triplet network is iteratively optimized to output accurate similarity prediction results.

Images