System and method for relationship extraction using self-adaptive thresholds and local context pooling - Patents.com

Self-adaptive thresholding and local context pooling enhance document-level relationship extraction by addressing multi-entity, multi-label challenges, resulting in improved accuracy and efficiency in predicting relationships across diverse datasets.

JP7584642B2Active Publication Date: 2024-11-15BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023521628
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-10-21
Filing Date
2021-09-27
Publication Date
2024-11-15
Estimated Expiration
2041-09-27

AI Technical Summary

Technical Problem

Current methods for document-level relationship extraction face challenges with multi-entity, multi-label problems, as they often rely on graph structures that may not fully leverage pre-trained language models and result in context-irrelevant noise due to identical entity representations across different pairs.

Method used

The use of self-adaptive thresholding and local context pooling techniques, which replace global thresholds with learnable entity-dependent thresholds and enhance entity embeddings by focusing on relevant contexts, respectively, to improve relationship extraction accuracy.

Benefits of technology

The proposed approach significantly outperforms traditional models by accurately predicting relationships in document-level extraction tasks, as demonstrated by improved F1 scores on various datasets, including DocRED, CDR, and GDA, showcasing enhanced performance in capturing complex document contexts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584642000055
    Figure 0007584642000055
  • Figure 0007584642000056
    Figure 0007584642000056
  • Figure 0007584642000057
    Figure 0007584642000057
Patent Text Reader

Abstract

A system and method for relationship extraction using self-adaptive thresholds and local context pooling (ATLOP) is provided. The system includes a computing device having a processor and a storage device storing computer-executable code. The computer-executable code is configured to provide a document, embed a plurality of entities in the document into a plurality of embedding vectors, and predict relationships between the entity pairs using the embedding vectors of the entity pairs in the document. The relationship prediction is based on an improved language model. Each relationship has a self-adaptive threshold, and a relationship between the entity pair is determined to exist if the logit of the relationship between the entity pair is greater than the logit function corresponding to the self-adaptive threshold.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [Cross citation] This application claims priority to U.S. Patent Application No. 17 / 076014, filed October 21, 2020, the entire disclosure of which is incorporated herein by reference. In describing this disclosure, several references are cited and discussed, and may include patents, patent applications, and various publications. Citation and / or discussion of such references is used only to describe this disclosure, and no such reference is admitted to be "prior art" disclosed herein. All references cited and discussed herein are incorporated herein by reference in their entirety, and to the same extent as if each reference was incorporated by reference alone.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to relationship extraction, and more particularly to relationship extraction using self-adaptive thresholds and local context pooling. [Background technology]

[0003] The background description provided herein is intended to provide an overall context for the present disclosure. Within the scope of this background description, any description that does not conform to the work of the current inventors and to the prior art at the time of filing is not expressly or implicitly admitted to be prior art to the present disclosure.

[0004] Relation Extraction (RE) is an important task in Natural Language Processing (NLP) to extract relations between entity pairs in pure text. Relations can be extracted from sentences or documents. Compared with sentence-level RE, document-level RE offers new challenges, in that a document typically contains multiple entity pairs, and an entity pair may appear multiple times in a document, associated with multiple possible relations or multiple labels.

[0005] To solve the multi-entity problem, most current methods use dependency structure, heuristics or structured attention to build a document graph, and then use a graph neural model to infer. The built graph connects distant entities in the document, thereby solving the shortfall of RNN-based encoders in capturing remote information. However, because the transducer-based model can model remote dependencies implicitly, it does not reveal whether the graph structure still contributes to the pre-trained language model, e.g., BERT. There are also methods that do not introduce the graph structure and directly apply the pre-trained language model. They simply average the embeddings of entity tokens to obtain entity embeddings, and input them into a classifier to obtain relationship labels. However, each entity has the same representation in different entity pairs, which may introduce context-irrelevant noise. Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, there is a need in the field to solve the multi-entity, multi-label problem in document-level relationship extraction. [Means for solving the problem]

[0007] In some aspects, the present disclosure provides two novel techniques to solve the multi-label and multi-entity problem: self-adaptive thresholding and local context pooling. Self-adaptive thresholding replaces the global thresholding used in multi-label classification in conventional work with a learnable entity dependent threshold. Local context pooling locates relevant contexts that help transfer attention from a pre-trained language model to determine relationships.

[0008] Specifically, the present disclosure provides a local context pooling technique that does not introduce a graph structure. Local context pooling solves the problem that all entity pairs use the same entity embedding. It enhances the entity embeddings using additional context related to the current entity pair. The present disclosure does not need to train a new context attention layer from scratch, but directly transfers the attention head from a pre-trained language model to obtain entity-level attention. Then, for two entities of a pair, the present disclosure finds the context that is important to both of them by integrating their attentions by multiplication.

[0009] For multi-label problems, traditional methods simplify it to a binary classification problem. After training, a global threshold is applied to the class probabilities to obtain the relational labels. The method involves heuristic threshold adjustment and introduces decision errors when the adjusted threshold in the development data is not the optimal threshold for all examples. In contrast, the present disclosure provides a self-adaptive threshold technique, which replaces the global threshold with a learnable threshold class. The threshold class is learned by a self-adaptive threshold loss. The self-adaptive threshold loss is a ranking-based loss that pushes the logit (logistic regression) of the positive class above the threshold in model training and pulls the logit of the negative class below the threshold. When testing, the present disclosure returns the class with a higher logit than the threshold class as the predicted label, or NA if the class does not exist. The technique eliminates the need for threshold adjustment and can adjust the threshold based on different entities, thereby obtaining better results.

[0010] By combining self-adaptive thresholding and local context pooling, the present disclosure provides a simple, novel and effective relationship extraction model, called ATLOP (Adaptive Thresholding and Localized cContext Pooling), to fully utilize the capabilities of pre-trained language models. The model solves the multi-label and multi-entity problems in document-level RE. Experiments on three document-level relationship extraction datasets, DocRED, CDR and GDA, show that the ATLOP model significantly outperforms the most advanced methods. DocRED is a large-scale document-level relationship extraction dataset built on Wikipadia and Wikipadia data, CDR is a chemical-disease relationship dataset, and GDA is a gene-disease related dataset.

[0011] In some aspects, the present disclosure relates to a system. In some embodiments, the system includes a computing device, the computing device including a processor and a storage device storing computer executable code, the computer executable code, when executed by the processor, Provide documentation, Embedding entities in the document into embedding vectors; predicting a relationship among a plurality of relationships between a first entity in the document and a second entity in the document based on a first embedding vector and a second embedding vector, wherein the first embedding vector in the plurality of embedding vectors represents the first entity and the second embedding vector in the plurality of embedding vectors represents the second entity; Here, the computer executable code is configured to perform embedding and prediction using a language model stored in the computing device, each relation in the plurality of relations having a self-adaptive threshold, and for one relation among the plurality of relations, it is determined that the relation exists if the logit of the relation is greater than the logit function of a corresponding one of the self-adaptive thresholds of the plurality of relations.

[0012] In some embodiments, the computer executable code is configured to embed each entity of the plurality of entities by summing up at least one hidden representation of each entity of the plurality of entities that is mentioned at least once using LogSumExp LSE.

[0013] In some embodiments, the computer executable code is configured to predict a relationship among the plurality of relationships by computing a local context pooling of an entity pair selected from the plurality of entities using the following formula:

number

[0014] In some embodiments, the hidden state of the subject entity and the object entity is determined according to the following formula:

number

[0015] In some embodiments, the computer executable code is configured to predict a relationship between the subject entity and the object entity using the following formula:

number

[0016]

number

[0017] In some embodiments, the language model includes at least one of a bidirectional encoder representing a transformer BERT, a robust optimization BERT method roBERTa, SciBERT, a generative pre-trained model GPT, GPT-2, and a reparametric transformer XL network XLnet.

[0018] In some embodiments, the language model has a structure based on BERT, and a loss function for training the language model is determined by the following formula:

number

[0019]

number

[0020] In some embodiments, the computer executable code is further configured to construct a knowledge graph using the predicted relationships in the first entity, the second entity, and the plurality of relationships between the first entity and the second entity. The knowledge graph may be, for example, a general knowledge graph including human knowledge, a fashion graph including features of fashion products, a gene-disease graph including relationships between human genes and human diseases associated with those genes, or a chemical-disease graph including relationships between chemicals and diseases.

[0021] In some embodiments, if a question includes the first entity and the second entity and the document is intended to include a solution according to the question, the computer executable code is further configured to form the solution using a predicted relationship in the plurality of relationships.

[0022] In some embodiments, the present disclosure relates to a method, in some embodiments, the method comprising: Providing a document via a computing device; embedding, by the computing device, a plurality of entities in the document into a plurality of embedding vectors; predicting, by the computing device, a relationship among a plurality of relationships between a first entity in the document and a second entity in the document based on a first embedding vector and a second embedding vector, the first embedding vector in the plurality of embedding vectors representing the first entity and the second embedding vector in the plurality of embedding vectors representing the second entity; Here, the embedding and prediction steps are performed using a language model stored in the computing device, and each of the multiple relations has a self-adaptive threshold, and for one of the multiple relations, if the logit of the relation is greater than the logit function of a corresponding one of the self-adaptive thresholds of the multiple relations, it is determined that the relation exists.

[0023] In some embodiments, embedding each of the plurality of entities using LogSumExp LSE by summarizing at least one hidden representation of each of the plurality of entities that is mentioned at least once.

[0024] In some embodiments, the predicting step comprises computing a local context pooling for an entity pair selected from the plurality of entities using the formula:

number

[0025] In some embodiments, the hidden state of the subject entity and the object entity is determined according to the following formula:

number

[0026] In some embodiments, predicting a relationship between the subject entity and the object entity using the formula:

number

[0027] In some embodiments, the language model comprises a bidirectional encoder representation BERT or SciBERT from a transducer, and a loss function for training the language module is determined by the following formula:

number

[0028] In some embodiments, the method further includes constructing a knowledge graph using the predicted relationships among the first entity, the second entity, and the plurality of relationships between the first entity and the second entity.

[0029] In some embodiments, if a question includes the first entity and the second entity and the document is intended to include a solution according to the question, the method further includes forming the solution using a predicted relationship in the plurality of relationships.

[0030] In some aspects, the present disclosure relates to a non-transitory computer-readable medium storing computer-executable code that, when executed by a processor of a computing device, is configured to perform the methods described above.

[0031] These and other aspects of the present disclosure will become apparent in the following description of the preferred embodiments from the drawings and accompanying text, in which variations and modifications thereof can be effected without departing from the spirit and scope of the novel concepts of the present disclosure. [Brief description of the drawings]

[0032] The accompanying drawings illustrate one or more embodiment(s) of the present disclosure and, together with the written description, serve to explain the principles of the disclosure. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or similar elements of the embodiment. [Figure 1] FIG. 1 illustrates a schematic of a self-adaptive threshold and local context pooling (ATLOP) system according to some embodiments of the present disclosure. [Diagram 2] FIG. 2 shows a schematic of an example from the DocRED dataset. [Diagram 3] FIG. 3 illustrates a schematic diagram of a self-adaptive threshold loss according to some embodiments of the present disclosure. [Figure 4] FIG. 4 illustrates a schematic diagram of local context pooling according to some embodiments of the present disclosure. [Diagram 5] FIG. 5 illustrates a schematic diagram of a training process of the ATLOP relation extraction application according to some embodiments of the present disclosure. [Figure 6] FIG. 6 illustrates a schematic diagram of an inference process of the ATLOP relation extraction application according to some embodiments of the present disclosure. [Figure 7] In Figure 7, Table 1 shows the statistics of the dataset in the experiment, where Ent., Ment. and Doc. are abbreviations for entity, mention and document, respectively. [Figure 8] In FIG. 8, Table 2 shows hyper-parameters of the ATLOP application according to some embodiments of the present disclosure. [Figure 9]In Figure 9, Table 3 shows the results for the DocRED development and test sets. The table reports the mean and standard deviation of F1 on the development set after five training runs with different random seeds. The table reports the official test score of the best checkpoint on the development set. [Figure 10] In Fig. 10, Table 4 shows the test F1 scores (calculated in %) on the CDR and GDA datasets. The table reports the mean and standard deviation of F1 on the test set after five training runs with different random seeds. [Figure 11] In Figure 11, Table 5 shows the ablation study of ATLOP on DocRED. We close different components of one model at a time. We report the average devF1 score over five training runs with different seeds. [Figure 12] In Fig. 12, Table 6 shows the results of different threshold policies on DocRED. Our self-adaptive threshold always outperforms other policies on the test set. [Figure 13] Figure 13 shows the devF1 scores for documents with different numbers of entities in DocRED. When the number of entities is greater than 5, our local security context can achieve better results. When the number of entities increases, the improvement is more noticeable. [Figure 14] FIG. 14 illustrates example context weights from FIG. 2 using local context pooling according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The present disclosure will be described in more detail in the following examples, which are merely used for illustration, and many modifications and variations of which will be obvious to those skilled in the art. Hereinafter, the embodiments of the present disclosure will be described in detail. With reference to the drawings, in all drawings, the same numerals indicate the same elements. As used in the description and the claims that follow, the words "a", "one" and "said" include the plural unless the context clearly dictates otherwise. Also, as used in the description and the claims that follow, the meaning of "in" includes "in" and "on". Also, for the convenience of the reader, titles or subtitles may be used in the specification, which will not affect the scope of the present disclosure. Also, some terms used in the present specification have more specific definitions below.

[0034] Terms used herein generally have their common meaning in the art, in the context of this disclosure and in the specific context in which each term is used. Discussion of some terms in this disclosure below or elsewhere in the specification provides additional guidance to the practitioner in relation to the description of this disclosure. As can be understood, the same can be expressed in more than one way. Thus, alternative language and synonyms can be used for any one or more terms described herein, without any special meaning whether or not a term is described or discussed in detail herein. The use of examples anywhere in this specification, including examples of any terms described herein, is only illustrative and does not limit the scope and meaning of the disclosure or any exemplary term. Similarly, the disclosure is not limited to the various examples shown in this specification.

[0035] As used herein, the term "module" may refer to belonging to or including an Application Specific Integrated Circuit (ASIC), an electronic line, a combinatorial logic circuit, a Field Programmable Gate Array (FPGA), a processor (shared, dedicated, or group) that executes code, other suitable hardware components that provide the described functionality, or a combination of some or all of the above, such as an on-chip system. The term module may also include memory (shared, dedicated, or group) that stores code executed by a processor.

[0036] As used herein, the term "code" may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, and / or objects. As used above, the term "shared" means that some or all code from multiple modules may be executed using a single (shared) processor. Also, some or all code from multiple modules may be stored in a single (shared) memory. As used above, the term group means that some or all code from a single module may be executed using a set of processors. Also, some or all code from a single module may be stored using a set of memories.

[0037] As used herein, the term "interface" generally refers to a communication tool or device for effecting data communication between components at an interaction point between the components. Generally, an interface may be applied at both the hardware and software level and may be a unidirectional or bidirectional interface. Examples of physical hardware interfaces include, for example, electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. Components communicating with an interface may be, for example, multiple components of a computer system or peripheral devices.

[0038] The present disclosure relates to a computer system. As shown in the drawings, computer components may include physical hardware components shown as solid line blocks and virtual software components shown as dashed line blocks. As will be understood by those skilled in the art, unless otherwise stated, these computer components may be implemented in the form of, but are not limited to, software, firmware or hardware components or a combination thereof.

[0039] The devices, systems and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory, tangible, computer-readable medium. The computer programs may further include stored data. Non-limiting examples of non-transitory, tangible, computer-readable media are non-volatile memory, magnetic memory and optical memory.

[0040] The present disclosure will now be described more fully with reference to the drawings, in which examples of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the examples set forth herein. On the contrary, these examples are provided so that the disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0041] FIG. 1 is a schematic diagram of a self-adaptive threshold and local context pooling (ATLOP) system according to some embodiments of the present disclosure. As shown in FIG. 1, the system 100 includes a computing device 110. In some embodiments, the computing device 110 may be a server computer, a cluster computer, a cloud computer, a general-purpose computer, a headless computer, or a special-purpose computer that provides relationship prediction and evidence prediction. The computing device 110 includes, but is not limited to, a processor 112, a memory 114, and a storage device 116. In some embodiments, the computing device 110 may include other hardware and software components (not shown) to perform corresponding tasks. Examples of these hardware and software components include, but are not limited to, other necessary memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, peripheral devices, etc.

[0042] Processor 112 may be a Central Processing Unit (CPU) that is configured to control the operation of computing device 110. Processor 112 may execute an operating system (OS) or other applications for computing device 110. In some embodiments, computing device 110 may have multiple CPUs as processors, such as, for example, two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs.

[0043] The memory 114 may be a volatile memory, such as a random-access memory (RAM), that is used to store data and information during operation of the computing device 210. In some embodiments, the memory 114 may be a volatile memory array. In some embodiments, the computing device 110 may operate on multiple memories 114.

[0044] The storage device 116 is a non-volatile data storage medium that stores the OS (not shown) and other applications of the computing device 110. Examples of the storage device 116 may include non-volatile memory, such as flash memory, memory cards, USB drives, hard disk drives, floppy disks, optical disk drives, solid state drives, or any other type of data storage device. In some embodiments, the computing device 110 may have multiple storage devices 116, which may be the same storage device or different types of storage devices, and applications of the computing device 110 may be stored in one or more storage devices 116 within the computing device 110.

[0045] In this embodiment, the processor 112, memory 114, and storage device 116 are components of a computing device 110 (e.g., a server computing device). In other embodiments, the computing device 110 may be a distributed computing device, with the processor 112, memory 114, and storage device 116 being shared resources among multiple computing devices within a predefined domain.

[0046] Additionally, storage device 116 includes an ATLOP relationship extraction application 118, training data 130, and prediction data 132. The ATLOP relationship extraction application 118 is configured to train its model structure using the training data 130 and to make predictions based on the prediction data 132. The training data 130 and the prediction data 132 are selectable for computing device 110, and the ATLOP relationship extraction application 118 may access training and prediction data stored on other devices.

[0047] As shown in FIG. 1, the ATLOP relation extraction application 118 includes a document preparation module 120, an encoder 122, a classifier 124, a function module 126, and an interface 128. In some embodiments, the ATLOP relation extraction application 118 may include other applications or modules necessary for the operation of the ATLOP relation extraction application 118. It should be noted that each module 120-128 may be implemented as computer executable code or instructions, or a data table or database, or a combination of hardware and software, which together form an application. In some embodiments, each module may further include sub-modules. Or, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, the modules may be collectively referred to as a model, which may be trained using training data, and after being trained, may be used for prediction.

[0048]

number

[0049]

number

[0050] Figure 2 is a schematic diagram of an example from the DocRED dataset. As shown in Figure 2, the subject entity is "John Stearnstreet", the object entity is "Bendigo", and the relationships are "Born in" and "Died in". The "Born in" relationship is embodied in the previous two sentences, and the "Died in" relationship is embodied in the last sentence. Other entities in the document are also highlighted, but are unrelated to the "John Stearnstreet-Bendigo" entity tuple.

[0051]

number

[0052]

number

number

[0053]

number

[0054] The representation of one entity is the same in different entity pairs. To reduce the number of parameters in the bilinear classifier, the classifier 124 is arranged to use group bilinearity, which divides the embedding dimension into k equal-sized groups and applies bilinearity within the groups.

number

[0055]

number

[0056]

number

[0057]

number

[0058]

number

[0059] For classification of an entity pair to be accurate, the logit function (or log probability) of the positive label should be greater than the threshold, and the logit function of the negative label should be less than the threshold. The classifier 124 is configured to introduce a threshold class TH, which it automatically learns in a similar manner to other classes (see equation (5)). At test time, the classifier 124 returns as positive labels classes with higher logit than the TH class, or NA if these classes do not exist. This threshold class learning relies on an entity threshold. It is an alternative to a global threshold, so there is no need to tune the threshold for the development set. In some embodiments, as mentioned above, the classifier 124 uses the logit instead of probability. r and logit r and the logit of TH to determine whether the relationship r exists.

[0060] To learn the new model, the classifier 124 is arranged to define a special loss function that takes into account the TH class. Specifically, the classifier is arranged to design a self-adaptive threshold loss based on the standard class cross-entropy loss. The loss function is divided into two parts, as shown below:

number

[0061] The first part, L1, is for the positive label and the TH class. Since there can be multiple positive labels, we calculate the total loss as the sum of the class cross entropy losses over all positive labels. L1 forces the logit of all positive labels to be greater than the TH class. If there is no positive label, it is not used. The second part, L2, is for the negative class and the threshold class. It is the class cross entropy loss, and the TH class is the real label. This causes the logit of the negative label to be less than the TH class. The addition of these two parts is the total loss.

[0062]

number

[0063] To accurately locate contexts closely related to an entity, the present disclosure further improves pooling in the encoder 122, thereby influencing the hidden state in classification by the classifier 124. Specifically, the logsumexp pooling shown in equation (2) accumulates embeddings of all mentions of an entity across documents and generates a single embedding for the entity. The entity embedding is then used to classify all entity pairs. However, because some contexts may indicate unrelated relationships with the entity pair, preferably one focuses only on local representations of relevant contexts in the document, which helps determine the relationship of the entity pair.

[0064] Therefore, the present disclosure provides local context pooling, which enhances embedding of entity pairs by embedding additional context related to the two entities. In some embodiments, the present disclosure uses a model based on a pre-trained transformer as the encoder 122, which has already learned token-level dependencies by multi-head self-attention, and therefore the present disclosure considers using those attention heads directly to perform local context pooling. The method transfers the learned dependencies from the pre-trained language model and does not need to learn a new attention layer from scratch.

[0065]

number

number

number

[0066]

number

[0067]

number

[0068] Returning to FIG. 1, when the document preparation module 120, the encoder 122, and the classifier 124 make relationship predictions, the function module 126 is configured to perform a function using the relationship of the predictions. In some embodiments, the function is a structural knowledge graph, and the function module 126 is configured to integrate the entity pairs and their predictions into the knowledge graph. Each entity may be a node in the knowledge graph, and the relationship may be an edge linking the corresponding entities. In some embodiments, the function retrieves information from a database, and the function module 126 is configured to train the encoder 122 and the classifier 124 with a training dataset of the database, infer relationships from the database after training, and provide the entity pairs and their relationships to the user. In some embodiments, the function is a question answering system, and the function module 126 is configured to extract entities from a question, infer entity relationships from an answer database or a comment database, form an answer to the question using the entities extracted from the question and the inferred relationships, and provide the answer to the user who submitted the question.

[0069] The interface 128 is configured to provide an interface to an administrator of the ATLOP relation extraction application 118 to train the encoder 122 and classifier 124 and tune model parameters, or to provide an interface to a user to use the ATLOP relation extraction application 118 to obtain answers to problems and build or complete knowledge graphs using documents.

[0070] 5 is a schematic diagram illustrating a training process of the ATLOP relation extraction application according to some embodiments of the present disclosure. In some embodiments, the training process is implemented by the computing device 110 shown in FIG. It should be noted that unless otherwise stated in the present disclosure, the steps of the training process or method may be arranged in different orders and are therefore not limited to the order shown in FIG.

[0071] 5, in step 502, the document preparation module 120 retrieves training data 130 and provides the training data 130 to the encoder 122. The training data is documents of labeled entities and relationships.

[0072] In step 504, for each document, the encoder 122 adds a "*" symbol to the beginning and end of the entity mention, ie, adds a "*" symbol before and after the entity mention.

[0073] In step 506, the encoder 122 calculates the entity embedding using logsumexp with the symbol “*” at the beginning of the mention as the token indicating the mention, and sends the entity embedding to the classifier 124. Specifically, the encoder 122 has a basic encoder structure of a language model such as BERT, and obtains the embedding of each token in the training document, i.e.

number

[0074] Each token embedding is represented as a vector. Then, the encoder 122 uses the embeddings of the tokens corresponding to the entity mentions to obtain the embedding of the entity by logsumexp, i.e.

number

[0075] In step 508, upon receiving the embeddings of the entities from the encoder 122, the classifier 124 computes the local context pooling (local context embedding) of the entity pair by the following formula:

number

[0076] In step 510, the classifier 124 uses entity embeddings and local context pooling to compute the hidden states of the entities. Specifically, for relationship prediction of an entity pair including a subject entity and an object entity, the hidden states of the entities are computed according to the following formula:

number

[0077] In step 512, after obtaining the hidden states of the entities in the entity pair, the classifier 124 determines the logit between the entities using group bilinear.

number

[0078] In step 514, for the logit between the entity pairs corresponding to each relationship, the classifier 124 compares the determined logit with the logit of the self-adaptive threshold (TH class) corresponding to the relationship, and determines that the relationship exists if the logit is equal to or greater than the logit function of the threshold. If the probability is less than the threshold, it determines that the relationship does not exist. Since a document can contain multiple mentions of an entity pair and multiple relationships, an entity pair can have one or multiple determined relationships.

[0079] In step 516, the classifier 124 calculates a loss function based on the self-adaptive threshold using the following formula:

number

[0080] In step 518, the loss function is fed back into the model to adjust the parameters of the encoder 122 and the classifier 124, and another round of prediction is performed to optimize the model.

[0081] Thus, steps 506-518 are repeatedly performed on the same document until the loss L converges to a small value or a predetermined number of iterations is reached. Steps 502-518 are then performed on other documents in the training data. In some embodiments, each training round is performed in batches, and each patch contains multiple documents, e.g., four documents.

[0082] FIG. 6 is a schematic diagram illustrating an estimation process of the ATLOP relation extraction application according to some embodiments of the present disclosure after the ATLOP relation extraction application is trained. In some embodiments, the estimation process is implemented by the computing device 110 shown in FIG. 1. It should be noted that, unless otherwise described in this disclosure, the steps of the training process or method may be arranged in different orders and are not limited to the order shown in FIG. 6. It should be noted that the training data of ATLOP and the documents to be predicted using ATLOP should be in the same field. For example, training ATLOP using Wikipedia data can estimate common knowledge from texts, and training ATLOP using biomedical data can estimate the relationship between genes and diseases from biomedical papers.

[0083] 6, in step 602, the document preparation module 120 retrieves predicted data 132 and provides the predicted data 132 to the encoder 122. The predicted data is a document, entities in the document may or may not be provided, and no relationship labels are present.

[0084] In step 604, for each document, the encoder 122 identifies entities from the document by their naming entity identification, e.g., spaCy or Stanza, and appends a "*" to the beginning and end of mentions of the identified entities in the document. Preferably, a list of entities and labels is provided, and naming entity identification is not required.

[0085] In step 606, the encoder 122 uses the symbol “*” at the beginning of a mention to represent the token indicating the mention, calculates the entity embedding using logsumexp, and sends the entity embedding to the classifier 124. Specifically, the encoder 122 has a basic encoder structure of a language model such as BERT, and obtains the embedding of each token in the training document, i.e.

number

[0086] Each token embedding is represented as a vector. Then, the encoder 122 uses the embeddings of the tokens corresponding to the entity mentions to obtain the embedding of the entity by logsumexp, i.e.

number

[0087] In step 608, upon receiving the embeddings of the entities from the encoder 122, the classifier 124 computes the local context pooling (local context embedding) of the entity pair according to the following formula:

number

[0088] In step 610, the classifier 124 uses entity embeddings and local context pooling to compute the hidden states of the entities. Specifically, to predict the relationship of an entity pair including a subject entity and an object entity, the hidden states of the entities are computed according to the following formula:

number

[0089] In step 612, after obtaining the hidden states of the entities in the entity pair, the classifier 124 determines the logit between the entities using group bilinear.

number

[0090]

number

[0091] In step 616, after receiving the entity pairs and corresponding relationships, the functionality module 126 performs a function, such as building or completing a knowledge graph with entities as nodes and relationships as edges, or providing a solution to a problem, where entities are extracted from the problem and entity pair relationships are extracted from a database associated with the problem.

[0092] In some aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable code. In some embodiments, the computer executable code may be software stored in storage device 116 as described above. The computer executable code, when executed, may perform any of the methods described above.

[0093] experiment Datasets: Experiments conducted demonstrate the advantages of some embodiments of the ATLOP application of the present disclosure. The datasets used in the experiments include DocRED, CDR, and GDA, as shown in Table 1 in Figure 7. DocRED (Yao et al. 2019) is a large-scale general-purpose dataset for document-level RE built on Wikipedia sentences. It consists of 3053 artificially annotated training documents. For entity pairs that represent relationships, about 7% of entity pairs have multiple relationship labels. CDR (Li et al. 2016) is an artificially annotated dataset in the biomedical domain. It consists of 500 training documents. The task is to predict binary interactions between chemical and disease concepts. GDA (Wu et al. 2019b) is a large-scale dataset in the biomedical domain. It consists of 29192 training sentences. The task is to predict binary interactions between genes and disease concepts. For the experiments, we follow Christopoulou, Miwa, and Ananiadou (2019) and split the training set 80 / 20 into the training set and development set, respectively.

[0094]

number

[0095] For models using a global threshold, we search the threshold from {0.1, 0.2, ..., 0.9} and choose the value that maximizes devF1. All models are trained using a single Tesla V 100 GPU. On the DocRED dataset, training the BERT-base encoder takes about 1h 45m, while training the RoBERTa-large encoder takes about 3h 30m. On the CDR and GDA datasets, training the SciBERT-base encoder takes 20m and 3h 30m, respectively.

[0096] Main Results: We compare ATLOP with sequence-based, graph-based and transformer-based models on the DocRED dataset. The experimental results are shown in Table 3, Figure 9. Based on Yao et al. (2019), we evaluate using F1 and Ign F1. Ign F1 represents the F1 score, which does not include the relation facts of training and dev / test set sharing.

[0097] Sequence-based models: These models use neural architectures such as CNN (Goodfellow, Bengio, and Courville 2015) and bidirectional LSTM (Schuster and Paliwal 1997) to encode the entire document, then obtain entity embeddings and predict the relationship for each entity pair using a bilinear function.

[0098] Graph-based models. These models build document graphs by learning the latent graph structure of documents and use graph convolutional networks for inference (Kipf and Welling 2017). We incorporate the two most advanced graph-based models, AGGCN (Guo, Zhang, and Lu 2019) and LSR (Nan et al. 2020), for comparison. The results for AGGCN are obtained by reimplementing Nan et al. (2020).

[0099] These models are transformer-based models. These models can adapt pre-trained language models to document-level RE when they do not use graph structures. They can be further divided into pipeline models (BERT-TS (Wang et al. 2019a)), hierarchical models (HIN-BERT (Tang et al. 20a)) and pre-training methods (Coref BERT and CorefRoBERTa (Ye et al. 2020)). We further incorporate the BERT baseline (Wang et al. 2019a) in the comparison.

[0100] We show that the results obtained with our revised BERT baseline are clearly superior to those of Wang et al. (2019a) and 1.2% higher than the most advanced RNN-based model BiLSTM LSR. This indicates that a pre-trained language model can capture distant dependencies between entities when it does not explicitly use a graph structure. After integrating other techniques, our enhanced baseline BERT-EBASE obtains an F1 score of 58.52%, which is close to the currently most advanced model BERT-LSRBASE. Our BERT-ATLOPBASE model further improves the representation of BERT-EBASE by 2.6%, proving the effectiveness of the two proposed new techniques. With RoBERTa-large as the encoder, our ALTOP model obtains an F1 score of 63.40%, which is the new most advanced result on DocRED. By September 9, 2020, we rank first on Colab Ranking List 5.

[0101] Biomedical Dataset Results: The experimental results for the two biomedical datasets are shown in Table 4 in Figure 10. Verga, Strubell, and McCallum (2018) and Nguyen and Verspoll (2018) are both sequence-based models, which use self-attention networks and CNNs as encoders, respectively. Christopoulou, Miwa, and Ananiadou (2019) and Nan et al. (2020) use graph-based models, which build document graphs by heuristic or structured attention, and infer using graph neural networks. As is known, pre-training language models based on transducers have not been applied to document-level RE datasets in the biomedical domain. In the experiments, we use SciBERT BASE Replace the encoder with SciBERT BASE SciBERT is pre-trained on a multidisciplinary corpus of scientific publications. BASE Our SciBERTATLOP baseline already outperforms all previous methods. BASEThe model further improves the F1 scores of CDR and GDA by 4.3% and 1.4%, respectively, and produces new state-of-the-art results on these two datasets.

[0102] Ablation Study: To prove the effectiveness of our proposed technique, we perform two sets of ablation studies on ATLOP and enhanced baseline by turning off components one by one. As shown in Table 5 in Figure 11, we observe that all components are beneficial to the model representation. Self-adaptive threshold and local context pooling are equally important to the model representation, decreasing the devF1 score by 0.89% and 0.97%, respectively, when removed from ATLOP. It is worth noting that the self-adaptive threshold is only effective when using the self-adaptive threshold loss optimization model. We apply the self-adaptive threshold to the model with binary cross entropy training, and the resulting devF1 is 41.74%.

[0103] Our enhanced baseline model, BERT-E BASE In contrast, group bilinear and logsumexp pooling both result in an increase of about 1% in devF1. We find that the improvement in entity marking is small (0.24% in devF1), but we still use the technique in our model because it makes it easier to derive mention embeddings and mention-level attention.

[0104] Threshold Analysis: Global thresholding does not consider the variation in model confidence across different classes or instances, thereby producing suboptimal representations. An interesting problem is to determine whether we can improve the global thresholding by adjusting different thresholds for different classes. Therefore, we attempt to maximize the F1 score on the DocRED development set by adjusting class-dependent thresholds using a circular optimization algorithm (Fan and Lin 2007). The results are shown in Table 6 of Figure 12. We find that using each class threshold improves the devF1 score to 61.73%, which is even greater than the self-adaptive thresholding results. However, the gain does not transfer to the test set. The results for each class threshold are also worse than the global thresholding results. Our self-adaptive thresholding technique used learnable thresholds that can be automatically propagated to the test set.

[0105] Context Pooling Analysis: To demonstrate that our local context pooling (LOP) technique solves the multi-entity problem, we divide the documents in the DocRED development set into different groups according to the number of entities, and evaluate models trained with and without local context pooling on each group. The experimental results are shown in Figure 13. As we observe, for the two models, the more entities the documents contain, the worse their representation becomes. As long as the documents contain few entities (1-5), otherwise the model with LOP always outperforms the model without LOP, and such improvement becomes increasingly larger when the number of entities increases. However, the number of documents that contain only 1-5 entities is very small (there are 4 in the development set), and the document average in DocRED contains 19 entities. Therefore, our local context pooling still significantly improves the overall F1 score. This allows the local context pooling technique to capture the relevant context of entity pairs, thereby solving the multi-entity problem.

[0106] We further visualize the context weights illustrated in Figure 2. As shown in Figure 14, our local context pooling assigns high weights to birth and death, which are the two most relevant to the entity (John Stern Street, Bendigo). These two tokens are also evidence of two criteria relations, place of birth and place of death, respectively. The weights of tokens such as electing a politician are very small because they are only relevant to the subject entity, John Stern Street. The visualization shows that the local context is able to locate the contexts related to these two entities.

[0107] In summary, some embodiments of the present disclosure provide an ATLOP model for document-level relationship extraction, which includes at least two novel techniques: self-adaptive threshold and local context pooling. The self-adaptive threshold technique replaces the global threshold in multi-label classification with a learnable threshold class, which can determine the optimal threshold for each entity pair. The local context pooling utilizes the pre-trained attention head to locate the relevant context of the entity pair, thereby helping to solve the multi-entity problem. Experiments on extracting datasets from three common document-level relationships show that our ATLOP model significantly outperforms previous models and produces the new most advanced results on all datasets.

[0108] The ATLOP model has downstream applications in many other NLP tasks, e.g. knowledge graph construction, information querying, question answering, and dialogue systems.

[0109] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teachings.

[0110] The selection and description of the embodiments are intended to illustrate the principles of the present disclosure and its practical application, and allow those skilled in the art to utilize the present disclosure and various embodiments and make various modifications according to the specific applications envisioned. Alternative embodiments will become apparent to those skilled in the art without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is not limited by the above description and the illustrative embodiments described, but is limited by the appended claims.

Claims

1. a computing device, the computing device including a processor and a storage device storing computer executable code, the computer executable code, when executed by the processor, Provide documentation, Embedding entities in the document into embedding vectors; predicting a relationship among a plurality of relationships between a first entity in the document and a second entity in the document based on a first embedding vector and a second embedding vector, wherein the first embedding vector in the plurality of embedding vectors represents the first entity and the second embedding vector in the plurality of embedding vectors represents the second entity; wherein the computer executable code is configured to perform embedding and prediction using a language model stored on the computing device, wherein each relation in the plurality of relations has a self-adaptive threshold, and for the one relation in the plurality of relations, determine that the relation exists if the logit of the relation is greater than a logit function of a corresponding one of the self-adaptive thresholds of the plurality of relations. system.

2. The computer executable code is configured to embed each of the plurality of entities together with at least one hidden representation of at least one mention of each of the plurality of entities using LogSumExp LSE. The system of claim 1 .

3. The computer executable code is configured to predict a relationship among a plurality of relationships by computing a local context pooling for a pair of entities selected from the plurality of entities using the following formula: [0010] The system of claim 1 .

4. The hidden states of the subject entity and the object entity are [0025] The system of claim 3.

5. The computer executable code is configured to predict a relationship between the subject entity and the object entity using the following formula: [0030] The system of claim 4.

6. 【Number 4】 The system of claim 5.

7. The language model includes at least one of the bidirectional encoder representation BERT from the transducer, the robust optimization BERT method roBERTa, SciBERT, the generative pre-trained model GPT, GPT-2, and the reparametric transducer XL network XLnet. The system of claim 1 .

8. The loss function for training the language model is determined by the following formula: [0050] The system of claim 7.

9. The computer executable code is further configured to construct a knowledge graph using the first entity, the second entity, and the predicted relationships among the plurality of relationships between the first entity and the second entity. The system of claim 1 .

10. If a question includes the first entity and the second entity, and the document is destined to include a solution according to the question, the computer executable code is further configured to form the solution using a predicted relationship in the plurality of relationships. The system of claim 1 .

11. A processor-implemented method comprising: Providing a document via a computing device; embedding, by the computing device, a plurality of entities in the document into a plurality of embedding vectors; predicting, by the computing device, a relationship among a plurality of relationships between a first entity in the document and a second entity in the document based on a first embedding vector and a second embedding vector, the first embedding vector in the plurality of embedding vectors representing the first entity and the second embedding vector in the plurality of embedding vectors representing the second entity; wherein the embedding and prediction steps are performed by a language model stored in the computing device, and each of the relations among the plurality of relations has a self-adaptive threshold, and for one of the relations among the plurality of relations, if the logit of the relation is greater than the logit function of a corresponding one of the self-adaptive thresholds of the plurality of relations, then it is determined that the relation exists. method.

12. embedding each of the plurality of entities using LogSumExp LSE with at least one hidden representation of each of the plurality of entities that is mentioned at least once. The method of claim 11.

13. The predicting step includes computing a local context pooling for an entity pair selected from the plurality of entities using the formula: [006] The method of claim 11.

14. The hidden states of the subject entity and the object entity are determined by the following formula: [0070] The method of claim 13.

15. predicting a relationship between the subject entity and the object entity using the formula: [0080] The method of claim 14.

16. The language model includes a bidirectional encoder representation BERT or SciBERT from a transducer, and a loss function for training the language model is determined by the following formula: [0090] The method of claim 15.

17. constructing a knowledge graph using the predicted relationships among the first entity, the second entity, and the plurality of relationships between the first entity and the second entity; or and if a question includes the first entity and the second entity, and the document is expected to include a solution according to the question, forming the solution using a predicted relationship in the plurality of relationships. The method of claim 11.

18. A non-transitory computer-readable medium storing computer-executable code, the computer-executable code, when executed by a processor of an active computing device, Provide documentation, Embedding entities in the document into embedding vectors; predicting a relationship among a plurality of relationships between a first entity in the document and a second entity in the document based on a first embedding vector and a second embedding vector, wherein the first embedding vector in the plurality of embedding vectors represents the first entity and the second embedding vector in the plurality of embedding vectors represents the second entity; wherein the computer executable code is configured to perform embedding and prediction using a language model stored on the non-transitory computer readable medium, wherein each relation in the plurality of relations has a self-adaptive threshold, and for the one relation in the plurality of relations, determine that the relation exists if a logit of the relation is greater than a logit function of a corresponding one of the self-adaptive thresholds of the plurality of relations. Non-transitory computer-readable medium.

19. The computer executable code is configured to predict a relationship among the plurality of relationships by computing a local context pooling of an entity pair selected from the plurality of entities using the following formula: [0089] where the hidden states of the subject entity and the object entity are determined by the following formula: ##EQU00011## wherein the computer executable code is configured to predict a relationship between the subject entity and the object entity using the following formula: ##EQU00012## 20. The non-transitory computer readable medium of claim 18.

20. The language model includes a bidirectional encoder representation BERT from a transducer, and a loss function for training the language model is ##EQU00013## 20. The non-transitory computer readable medium of claim 18.

Citation Information

Patent Citations

  • Entity relationship extraction method and device, computer equipment and storage medium

    CN110059320A

  • Causal relation recognition device and computer program therefor

    JP2018060364A

  • Device and method for natural language processing

    US20180157643A1