Systems and methods for extracting relationships using adaptive thresholding and local contextual pooling

Abstract

A system and method for relation extraction using adaptive threshold and local context pooling (ATLOP). The system includes a computing device. The computing device has a processor and a storage device storing computer executable code. The computer executable code is configured to: provide a document; embed a plurality of entities in the document into a plurality of embedding vectors; and predict a relation between a pair of entities in the document using embedding vectors of the pair of entities. The relation prediction is based on an improved language model. Each relation has an adaptive threshold, and a relation between the pair of entities is determined to exist when a logit of the relation between the pair of entities is greater than a logit function of the corresponding adaptive threshold.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Patent Application No. 17 / 076014, filed October 21, 2020, which is incorporated herein by reference in its entirety.

[0003] References are cited and discussed in the description of this disclosure, which may include patents, patent applications, and various publications. The citations and / or discussions of such references are provided solely to clarify the description of this disclosure and do not imply that any such references constitute "prior art" as disclosed herein. All references cited and discussed in the specification are incorporated herein by reference in their entirety, to the same extent as each individual reference is incorporated by reference individually. Technical Field

[0004] This disclosure generally relates to relation extraction, and more specifically, to relation extraction using adaptive thresholding and local context pooling. Background Technology

[0005] The background description provided herein is intended to present the overall context of this disclosure. Within the scope of the description in this background section, the work of the currently identified inventors, and any descriptions of non-compliance with prior art at the time of filing, are not expressly or impliedly acknowledged as prior art to this disclosure.

[0006] Relation extraction (RE) is an important task in Natural Language Processing (NLP), which extracts relationships between entity pairs from plain text. Relations can be extracted from sentences or documents. Compared to sentence-level RE, document-level RE presents new challenges because a document typically contains multiple entity pairs, and an entity pair may appear multiple times in documents associated with multiple possible relations or tags.

[0007] To address the multi-entity problem, most current methods construct document graphs using dependency structures, heuristics, or structured attention, and then use graph neural models for reasoning. The constructed graph connects entities that are far apart in the document, thus overcoming the shortcomings of RNN-based encoders in capturing long-range information. However, since transformer-based models can implicitly model long-range dependencies, it is unclear whether graph structures still benefit pre-trained language models such as BERT. There are also methods that directly apply pre-trained language models without introducing graph structures. They simply average the embeddings of entity segments to obtain entity embeddings and feed them into a classifier to obtain relation labels. However, each entity has the same representation across different entity pairs, which can introduce noise from irrelevant contexts.

[0008] Therefore, there is a need in this field to solve the problem of multiple entities and multiple tags in document-level relationship extraction. Summary of the Invention

[0009] In some respects, this disclosure provides two novel techniques, adaptive thresholding and local context pooling, to address multi-label and multi-entity problems. Adaptive thresholding replaces the global thresholding used in previous work for multi-label classification with a threshold that depends on learnable entities. Local context pooling directly shifts attention from the pre-trained language model to locate relevant context that helps determine relationships.

[0010] Specifically, this disclosure provides a local context pooling technique instead of introducing a graph structure. Local context pooling addresses the problem of using the same entity embeddings for all entity pairs. It enhances the entity embeddings using additional context associated with the current entity pair. Instead of training a new context attention layer from scratch, this disclosure directly transfers the attention head from a pre-trained language model to obtain entity-level attention. Then, for the two entities in a pair, this disclosure merges their attentions through multiplication to find context that is important to both.

[0011] For multi-label problems, existing methods simplify them to binary classification problems. After training, a global threshold is applied to the class probabilities to obtain relation labels. This method involves heuristic threshold adjustment and introduces decision errors when the threshold adjusted in the development data may not be the optimal threshold for all instances. In contrast, this disclosure provides an adaptive thresholding technique that replaces the global threshold with a learnable threshold class. The threshold class is learned through an adaptive thresholding loss. This adaptive thresholding loss is a ranking-based loss that pushes the logit (logistic regression) of the positive class above the threshold and pulls the logit of the negative class below the threshold during model training. At test time, this disclosure returns the class with a higher logit than the threshold class as the predicted label, or NA if the class does not exist. This technique eliminates the need for threshold adjustment and can also adjust the threshold according to different entity pairs for better results.

[0012] By combining adaptive thresholding and local context pooling, this disclosure presents a simple yet novel and effective relation extraction model, called ATLOP (Adaptive Thresholding and Localized context pooling), to fully leverage the capabilities of pre-trained language models. This model addresses the multi-label and multi-entity problems in document-level relation extraction (RE). Experiments on three document-level relation extraction datasets—DocRED, CDR, and GDA—demonstrate that the ATLOP model significantly outperforms state-of-the-art methods. DocRED is a large-scale document-level relation extraction dataset constructed from Wikipedia and Wikidata; CDR is a dataset of chemical disease relationships; and GDA is a dataset of genetic disease associations.

[0013] In some aspects, this disclosure relates to a system. In some embodiments, the system includes a computing device having a processor and a storage device storing computer-executable code. The computer-executable code, when executed at the processor, is configured to:

[0014] Provide documentation;

[0015] Embed multiple entities from the document into multiple embedding vectors; and

[0016] Based on a first embedding vector and a second embedding vector, predict one of a plurality of relations between a first entity and a second entity in the document, wherein the first embedding vector represents the first entity and the second embedding vector represents the second entity.

[0017] The computer-executable code is configured to embed and predict using a language model stored in the computing device. Each of the plurality of relations has an adaptive threshold. For one of the plurality of relations, the relation is determined to exist when the logit of the relation is greater than the logit function of the corresponding adaptive threshold of the plurality of relations.

[0018] In some embodiments, the computer-executable code is configured to embed each of the plurality of entities by using LogSumExp LSE to summarize at least one hidden representation of each entity that is mentioned at least once in the plurality of entities.

[0019] In some embodiments, the computer-executable code is configured to predict one of a plurality of relations by calculating the local context pooling of a pair of entities selected from the plurality of entities using the following formula:

[0020]

[0021]

[0022] a (s，o) =q (s，o) / 1 T q (s，o) ,as well as

[0023] c (s，o) =H T a (s，o) ,

[0024] The entity pair comprises a subject entity and an object entity. It is the word-segmentation level attention of the main entity. It is the word-segmentation level attention of the object entity, A (s，o) yes and The product of In this context, H represents the number of attention heads, and A represents the number of attention heads. i (s，o) It is the i-th multiplication in H multiplications, a (s，o) It is q (s，o) Normalization to a sum of 1, H T a (s，o) H in the model is the last layer embedding of the pre-trained language model, and c(s,o) is the local context pooling of the entity pair.

[0025] In some embodiments, the hidden states of the subject entity and the object entity are determined by the following formula:

[0026] as well as

[0027]

[0028] in, It is the embedding of the main entity. This is the hidden state of the main entity. It is the embedding of the object entity. This is the hidden state of the object entity, W. s W o W C1 and W C2 These are model parameters.

[0029] In some embodiments, the computer-executable code is configured to use the following formula to predict the relationship between the subject entity and the object entity:

[0030]

[0031] Among them, logit r The main entity e s and the object entity e o The logit function for the relation r, where k is a positive integer. The dimension is divided by k to form multiple The dimension is divided by k to form multiple and br are model parameters. When the logit r When the logit function of the relation r is greater than the learnable threshold TH, the principal entity e s and the object entity e o The relationship r is present. In some embodiments, The dimensions and the stated The dimension is 768, and k is 12.

[0032] In some embodiments, the language model includes at least one of the following: a bidirectional encoder representation BERT from a transformer, a robust optimization method roBERTa, SciBERT, a pre-trained model GPT, GPT-2, and a reparameterized transformer XL network XLnet.

[0033] In some embodiments, the language model has a BERT-based architecture, and the loss function used to train the language model is determined by the following formula:

[0034]

[0035]

[0036] as well as

[0037] L = L1 + L2,

[0038] Among them, logit r The main entity e s and the object entity e o The logit function, where r represents the relation and k is a positive integer. The dimension is divided by k to form multiple The dimension is divided by k to form multiple and b r These are model parameters, TH is the learnable threshold of the relationship, and P is... T NT represents a positive relation class, and NT represents a negative relation class. In some embodiments, The dimensions and the stated The dimension is 768, and k is 12.

[0039] In some embodiments, the computer-executable code is further configured to construct a knowledge graph using predicted relationships from the plurality of relationships between the first entity, the second entity, and the first entity and the second entity. The knowledge graph may be, for example, a general knowledge graph containing human knowledge, a fashion graph containing features of fashion products, a genetic disease graph containing relationships between human genes and human diseases associated with those genes, or a chemical disease graph containing relationships between chemical substances and diseases.

[0040] In some embodiments, when the question includes the first entity and the second entity, and the document is pre-defined to include the answer to the question, the computer-executable code is further configured to form the answer using predicted relations among the plurality of relations.

[0041] In some aspects, this disclosure relates to a method. In some embodiments, the method includes:

[0042] Documentation provided by computing devices;

[0043] The computing device embeds multiple entities in the document into multiple embedding vectors; and

[0044] The computing device predicts one of a plurality of relationships between a first entity and a second entity in the document based on a first embedding vector and a second embedding vector, wherein the first embedding vector represents the first entity and the second embedding vector represents the second entity.

[0045] The embedding and prediction steps are performed by a language model stored in the computing device. Each of the plurality of relations has an adaptive threshold. For a given relation among the plurality of relations, the relation is determined to exist when the logit of the relation is greater than the logit function of the corresponding adaptive threshold among the plurality of relations.

[0046] In some embodiments, the step of embedding each of the plurality of entities is performed by summarizing at least one hidden representation of each of the plurality of entities by using LogSumExp LSE to summarize at least one mention of each of the plurality of entities.

[0047] In some embodiments, the prediction step includes calculating the local context pooling of a selected entity pair from the plurality of entities using the following formula:

[0048]

[0049]

[0050] a(s，o) =q (s，o) / 1 T q (s，o) ,as well as

[0051] c (s，o) =H T a (s，o) ,

[0052] The entity pair includes a subject entity and an object entity. It is the word-segmentation level attention head of the main entity. A is the word-segmentation level attention head of the object entity. (s，o) yes and The product of In this context, H represents the number of attention heads, and A represents the number of attention heads. i (s，o) It is the i-th multiplication in H multiplications, a (s，o) It is q (s，o) Normalization to a sum of 1, H T a (s，o) In this context, H represents the last layer of embedding in the pre-trained language model, and c... (s，o) It is the local context pooling of the entity pair.

[0053] In some embodiments, the hidden states of the subject entity and the object entity are determined by the following formula:

[0054] as well as

[0055]

[0056] in, It is the embedding of the main entity. This is the hidden state of the main entity. It is the embedding of the object entity. This is the hidden state of the object entity, W. s W o W C1 and W C2 These are model parameters.

[0057] In some embodiments, the step of predicting the relationship between the subject entity and the object entity is performed using the following formula:

[0058]

[0059] Among them, logit r The main entity e s and the object entity e oThe logit function for the relation r, where k is a positive integer. The dimension is divided by k to form multiple The dimension is divided by k to form multiple and b r These are model parameters. When the logit r When the logit of the relation r is greater than the learnable threshold TH, the principal entity e s and the object entity e o It has the relationship r.

[0060] In some embodiments, the language model includes a bidirectional encoder representation of BERT or SciBERT from the transformer, and the loss function for training the language module is determined by the following formula:

[0061]

[0062]

[0063] as well as

[0064] L = L1 + L2,

[0065] Among them, logit r The main entity e s and the object entity e o The logit function for the relation r, where k is a positive integer. The dimension is divided by k to form multiple The dimension is divided by k to form multiple and b r These are model parameters, TH is the learnable threshold of the relationship, and P is... T NT represents a positive relation class, and NT represents a negative relation class.

[0066] In some embodiments, the method further includes constructing a knowledge graph using the first entity, the second entity, and predicted relationships from the plurality of relationships between the first entity and the second entity.

[0067] In some embodiments, when the question includes the first entity and the second entity, and the document is pre-defined to include an answer to the question, the method further includes: using predicted relations from the plurality of relations to form the answer.

[0068] In some aspects, this disclosure relates to a non-transitory computer-readable medium storing computer-executable code. The computer-executable code, when executed at a processor of a computing device, is configured to perform the methods described above.

[0069] These and other aspects of this disclosure will become apparent from the following description of preferred embodiments taken in conjunction with the accompanying drawings and description therewith, although variations and modifications therein will be made without departing from the spirit and scope of the novel concept of this disclosure. Attached Figure Description

[0070] The accompanying drawings illustrate one or more embodiments of this disclosure and, together with the written description, serve to explain the principles of this disclosure. Where possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments.

[0071] Figure 1 An adaptive thresholding and local context pooling (ATLOP) system according to certain embodiments of the present disclosure is schematically depicted.

[0072] Figure 2 An example from the DocRED dataset is illustrated schematically.

[0073] Figure 3 An adaptive threshold loss according to certain embodiments of the present disclosure is schematically depicted.

[0074] Figure 4 Local context pooling according to certain embodiments of this disclosure is illustrated schematically.

[0075] Figure 5 The training process of an ATLOP relation extraction application according to certain embodiments of the present disclosure is illustrated schematically.

[0076] Figure 6 The inference process of an ATLOP relation extraction application according to certain embodiments of the present disclosure is illustrated schematically.

[0077] Figure 7 Table 1 shows the statistics of the dataset in the experiment, where Ent., Ment., and Doc. are abbreviations for entity, mention, and document, respectively.

[0078] Figure 8 Table 2 shows the hyperparameters of ATLOP applications according to certain embodiments of this disclosure.

[0079] Figure 9 Table 3 shows the results for the DocRED development and test sets. This table reports the mean and standard deviation of the F1 score on the development set after five training runs using different random seeds. The table also reports the official test scores for the best checkpoints on the development set.

[0080] Figure 10 Table 4 shows the test F1 scores (in percentage) for the CDR and GDA datasets. This table reports the mean and standard deviation of the F1 scores on the test set after five training runs using different random seeds.

[0081] Figure 11 Table 5 shows the ablation study of ATLOP in DocRED. We disable different components of a model one at a time. We report the average dev F1 score by performing five training runs using different seeds.

[0082] Figure 12 Table 6 shows the results for different threshold strategies in DocRED. Our adaptive threshold consistently outperforms other strategies on the test set.

[0083] Figure 13 The diagram shows the dev F1 scores of documents in DocRED with varying numbers of entities. Our local context pooling yields better results when the number of entities is greater than 5. This improvement becomes more significant as the number of entities increases.

[0084] Figure 14 This illustration demonstrates the use of local context pooling according to certain embodiments of the present disclosure. Figure 2 The example in the text shows the context weights. Detailed Implementation

[0085] This disclosure is described in more detail in the following examples, which are for illustrative purposes only, as many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of this disclosure will now be described in detail. Referring to the accompanying drawings, the same numerals denote the same parts in all views. As used herein and in the following claims, unless the context clearly specifies otherwise, the terms “a,” “an,” and “described” include plural references. Furthermore, as used herein and in the following claims, unless the context clearly specifies otherwise, “in…” includes both “in…” and “on…”. Additionally, headings or subheadings may be used in the specification for the reader's convenience, without affecting the scope of this disclosure. Furthermore, some terms used in this specification are given more specific definitions below.

[0086] In the context of this disclosure and the specific context in which each term is used, the terms used herein generally have their common meaning in the art. Certain terms used to describe this disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance regarding the description of this disclosure. It will be understood that the same thing can be expressed in more than one way. Therefore, alternative language and synonyms may be used for any one or more terms discussed herein, and there is no particular significance as to whether a term is elaborated or discussed herein. The use of examples anywhere in this specification, including examples of any terms discussed herein, is merely illustrative and in no way limits the scope and meaning of this disclosure or any exemplary term. Likewise, this disclosure is not limited to the various embodiments given in this specification.

[0087] As described herein, the term "module" can refer to or include application-specific integrated circuits (ASICs), electronic circuitry; combinational logic circuits; field-programmable gate arrays (FPGAs); processors (shared, dedicated, or grouped) that execute code; other suitable hardware components that provide the described functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip. The term "module" can include memory (shared, dedicated, or grouped) that stores code executed by the processor.

[0088] As described herein, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared" as used above means that some or all of the code from multiple modules can be executed using a single (shared) processor. Furthermore, some or all of the code from multiple modules can be stored in a single (shared) memory. The term "group of terms" used above means that some or all of the code from a single module can be executed using a set of processors. Furthermore, a set of memories can be used to store some or all of the code from a single module.

[0089] As described herein, the term "interface" generally refers to a communication tool or device used at the interaction point between components to perform data communication between components. Generally, interfaces can be applied at both the hardware and software levels, and can be unidirectional or bidirectional. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. Components communicating with the interface can be, for example, multiple components of a computer system or peripheral devices.

[0090] This disclosure relates to computer systems. As shown in the accompanying drawings, computer components may include physical hardware components, shown as solid line blocks, and virtual software components, shown as dashed line blocks. Those skilled in the art will understand that, unless otherwise stated, these computer components may be implemented as software, firmware, or hardware components, or combinations thereof, but are not limited to these forms.

[0091] The apparatus, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer program may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic memory, and optical memory.

[0092] This disclosure will now be described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are illustrated. However, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

[0093] Figure 1 An adaptive thresholding and local context pooling (ATLOP) system according to certain embodiments of this disclosure is schematically depicted. Figure 1 As shown, system 100 includes computing device 110. In some embodiments, computing device 110 may be a server computer, cluster, cloud computer, general-purpose computer, headless computer, or special-purpose computer that provides relation prediction and evidence prediction. Computing device 110 may include, but is not limited to, processor 112, memory 114, and storage device 116. In some embodiments, computing device 110 may include other hardware and software components (not shown) to perform their respective tasks. Examples of such hardware and software components may include, but are not limited to, other required memory, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0094] Processor 112 may be a central processing unit (CPU) configured to control the operation of computing device 110. Processor 112 may execute an operating system (OS) or other applications of computing device 110. In some embodiments, computing device 110 may have multiple CPUs as processors, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs.

[0095] Memory 114 may be volatile memory, such as random-access memory (RAM), used to store data and information during operation of computing device 210. In some embodiments, memory 114 may be an array of volatile memory. In some embodiments, computing device 110 may operate on multiple memories 114.

[0096] Storage device 116 is a non-volatile data storage medium for storing the operating system (not shown) and other applications of computing device 110. Examples of storage device 116 may include non-volatile memory such as flash memory, memory card, USB drive, hard disk drive, floppy disk drive, optical disk drive, solid-state drive, or any other type of data storage device. In some embodiments, computing device 110 may have multiple storage devices 116, which may be the same storage device or different types of storage devices, and applications of computing device 110 may be stored in one or more storage devices 116 of computing device 110.

[0097] In this embodiment, processor 112, memory 114, and storage device 116 are components of computing device 110 (e.g., a server computing device). In other embodiments, computing device 110 may be a distributed computing device, where processor 112, memory 114, and storage device 116 are shared resources from multiple computing devices in a predefined region.

[0098] Among other things, storage device 116 includes an ATLOP relation extraction application 118, training data 130, and prediction data 132. The ATLOP relation extraction application 118 is configured to train its model structure using the training data 130 and to make predictions based on the prediction data 132. The training data 130 and prediction data 132 are optional for computing device 110, provided that the ATLOP relation extraction application 118 has access to the training and prediction data stored on other devices.

[0099] like Figure 1As shown, the ATLOP relation extraction application 118 includes a document preparation module 120, an encoder 122, a classifier 124, a functional module 126, and an interface 128. In some embodiments, the ATLOP relation extraction application 118 may include other applications or modules required for the operation of the ATLOP relation extraction application 118. It should be noted that the various modules 120-128 are implemented by computer-executable code or instructions, or data tables or databases, or a combination of hardware and software, which together form an application. In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, the modules may also be collectively referred to as models, which can be trained using training data and, after training, can be used for prediction.

[0100] The document preparation module 120 is configured to prepare training samples or query samples and send the prepared training samples or query samples to the encoder 122. Given a training sample or query sample, such as document d and entity set... Document preparation module 120 is configured to define a relation set R and a relation {NA}. Relation {NA} represents no relation. For training samples, document preparation module 120 is also configured to provide ground truth labels corresponding to the relations of the entities. In some embodiments, when the format of the training samples or query samples conforms to the requirements of encoder 122 and classifier 124, document preparation module 120 can simply input the samples into encoder 122 and classifier 124. In some embodiments, when the format of the training samples or query samples differs slightly from the requirements of encoder 122 and classifier 124, document preparation module 120 can modify the format to conform to the requirements of encoder 122 and classifier 124.

[0101] Given document d, entity set The predefined set of relations R and relations {NA}, the task of document-level relation extraction is to extract relations from entity pairs (e s e o ) s，o＝1...n；s≠o Predict a subset of relations from R∪{NA}, where R is a predefined set of interest relations, and e s and e o These are identified as the subject entity and the object entity, respectively, where n is the total number of predefined entities, and n is a positive integer. Based on entity mentions... Entity e i It may appear multiple times in document d, where N is a positive integer representing entity e. i The number of times it is mentioned in document d, Represents entity e iIt is mentioned for the jth time in document d. If through entity (e) s e o If any pair of references is used to express a relation, then the entity (e) s e o There are relationships between entity pairs (e, e, d). Entity pairs that do not express any relationship are labeled NA. During training, the model needs to predict all entity pairs (e, e, d) in document d. s e o ) s，o＝1...n；s≠o The model generates labels and compares these predictions with baseline labels. During testing or querying, the model needs to predict all entity pairs (e) in document d. s e o ) s，o＝1...n；s≠o The tag.

[0102] Figure 2 An example from the DocRED dataset is illustrated schematically. Figure 2 As shown, the subject entity is "John Stanstreet," the object entity is "Bendigo," and the relations are "birthplace" and "deathplace." The "birthplace" relation is shown in the first two sentences, and the "deathplace" relation is shown in the last sentence. Other entities in the document are also highlighted, but are not related to the entity tuple "John Stanstreet-Bendigo."

[0103] Return to reference Figure 1 The encoder 122 and classifier 124 are based on language models such as BERT and are improvements upon those models. Upon receiving prepared document samples from the document preparation module 120, the encoder 122 is configured to encode the document as entity embeddings in vector form and send these entity embeddings to the classifier 124. For a given document d, the encoder 122 is first configured to identify entities using named entity recognition (NER), such as spaCy, Stanza, Unified Medical Language Systems (UMLS), or GeneOntology (GO), and to label entities, for example, by their span. Note that for the training data, entity and relation labels are provided without performing NER; during inference, possible entities and possible relations are also provided, and NER may not be necessary. Each entity may be mentioned multiple times in the document. The encoder 122 is then configured to mark the location of an entity mention by inserting a special symbol “*” at the beginning and end of each mention. Following the tag mention, encoder 122 is configured to convert documents containing the "*" tag into tokens, where each "*" is a token. Therefore, document d is now represented by tokens, i.e. Where l is a positive integer representing the total number of word segments, x t It is the t-th word out of l word segments. The document represented by the word segments is fed into a pre-trained language model such as BERT to obtain contextual embeddings:

[0104] [h1, h2, ..., h t , ..., h l ] = BERT([x1, x2, ..., x...) t , ..., x l (1)

[0105] Among them, h t It is a word segmentation x t Hidden vectors or embeddings.

[0106] After embedding the word segmentation, encoder 122 is also configured to use the embedding of the starting "*" before the entity reference as the embedding of that entity reference. In some embodiments, this disclosure may also use the ending "*" after the entity reference as the reference. Entity e i Encoder 12 is then configured to apply logsumexp pooling, a smoothed version of max pooling, to obtain entity embeddings.

[0107]

[0108] in, Represents entity e i The j-th mention in document d, It is a positive integer representing entity e. i The total number of entity mentions, It is entity e i The embedding of the j-th entity mentioned in document d. Pooling accumulates the signals mentioned in the document, and this pooling performs better than average pooling.

[0109] Classifier 124 is configured to predict the relationship between any two entities upon receiving an entity embedding and send that relationship to functional module 126. Given an entity pair e calculated by equation (2) s e o Embedded Classifier 124 is configured to map entities to hidden states z through linear layers and nonlinear activations, and then compute the probability of relation r through a bilinear function and sigmoid activation. This process is expressed as:

[0110]

[0111]

[0112]

[0113] in, These are the model parameters, and d is the dimension of the embedding vector. s It is the hidden state of the main entity, z o It represents the hidden state of the object entity, tanh is the hyperbolic tangent function, and W... s It is the embedding of the main entity. The weight, W o It is object entity embedding The weight, W r It is the weight of relation r, b r It is a learnable constant of relation r.

[0114] The representation of an entity is the same across different entity pairs. To reduce the number of parameters in the bilinear classifier, classifier 124 is configured to use grouped bilinearity, which splits the embedding dimension into k equal-sized groups and applies bilinearity within each group:

[0115]

[0116]

[0117]

[0118] For i = 1...k, These are model parameters. P(r|e s e o ) is the relation r and the entity pair (e) s e o The probability of association. In some embodiments, k = 12 and d = 768, therefore for z s Of the total 768 dimensions, 12 are... Each of them contains 64 dimensions. In this way, this disclosure can reduce the number of parameters from d 2 Reduce to d 2 / k. In some embodiments, the number of vector dimensions and k may have other values, depending on the circumstances.

[0119] In some embodiments, classifier 124 does not compute P(r|e s e o Instead, it calculates... Among them, logit r It is the main entity e s and object entity e o The logit function for relation r, where k is a positive integer. The dimension is divided by k to form multiple The dimension is divided by k to form multiple and b r These are model parameters. When the logit r When the learnable threshold TH of the relation r is greater than or the logit is greater than the learnable threshold TH, the principal entity e s and the object entity e o It has the relationship r.

[0120] Classifier 124 can be trained using binary cross-entropy loss. During inference, classifier 124 can be adjusted to maximize the global threshold θ of the evaluation metric (F1 score of RE) in the development set, and if P(r|e s e o If θ > 0, then r is returned as the association, or NA is returned if no association exists. The application of logsumexp pooling in encoder 122 and group bilinearity in classifier 124 enhances the performance of this disclosure, which significantly outperforms state-of-the-art language models such as BERT.

[0121] In some embodiments, the classifier 124 is further improved by replacing the global threshold θ in the model with an adaptive threshold. The classifier 124 outputs a probability P(r|e) in the range [0, 1]. s e o This requires converting a threshold into relation labels. Since the threshold has neither a closed-form solution nor is differentiable, a common practice for determining the threshold is to enumerate several values ​​in the range [0, 1] and choose the one that maximizes the evaluation metric (F1 score of RE). However, the model may have different confidence levels for different entity pairs or classes for which a global threshold is insufficient. The number of relations may vary (multi-label problem), and the model may not be globally calibrated, so the same probability does not mean that all entity pairs are the same. To address this issue, classifier 124 is configured to replace the global threshold with a learnable adaptive threshold, which reduces decision errors during inference.

[0122] For ease of explanation, this disclosure refers to entity pair T = (e s e o The tags are divided into two subsets: positive tags P T and negative label N T The definition is as follows:

[0123] Positive label P T ∈R represents a relation that exists between entities in T. If T does not represent any relation, then P T Empty.

[0124] Negative label N T∈R represents a relation that does not exist between entities. If T does not represent any relation, then N T =R.

[0125] If an entity is correctly classified, the logit function (or log odds) for a positive label should be greater than a threshold, while the logit function for a negative label should be less than a threshold. Classifier 124 is configured to introduce a threshold class TH, which learns automatically in the same manner as other classes (see Equation (5)). At test time, classifier 124 is configured to return classes with a higher logit than class TH as positive labels, or NA if such classes do not exist. This threshold class learning depends on the threshold of the entity. It is an alternative to a global threshold, thus eliminating the need to adjust the threshold in the development set. In some embodiments, as described above, classifier 124 is configured to compute the logit. r Instead of probability, logit r The relation r is determined by comparing it with the logit of TH.

[0126] To learn the new model, classifier 124 was configured to define a special loss function that takes into account the TH classes. Specifically, the classifier was configured to design an adaptive threshold loss based on the standard class cross-entropy loss. The loss function was split into two parts, as shown below:

[0127]

[0128]

[0129] L = L1 + L2.

[0130] The first part, L1, involves the positive labels and the TH class. Since there may be multiple positive labels, the total loss is calculated as the sum of the class cross-entropy losses on all positive labels. L1 ensures that the logit of all positive labels is greater than that of the TH class. If no positive labels exist, it is not used. The second part, L2, involves the negative class and the threshold class. It is the class cross-entropy loss, where the TH class is the true label. It ensures that the logit of the negative labels is less than that of the TH class. These two parts are simply added together to obtain the total loss.

[0131] Figure 3 An adaptive threshold loss according to certain embodiments of this disclosure is schematically depicted. Figure 3 As shown, the L1 loss considers the positive class P T For classes TH and N, L2 loss takes into account both classes TH and negative classes N. T Compared to the global threshold, classifier 124 achieved a significant performance gain.

[0132] To accurately locate contexts closely related to entity pair relationships, this disclosure further improves pooling in encoder 122, thereby influencing the hidden state in classification via classifier 124. Specifically, the logsumexp pooling shown in equation (2) accumulates the embeddings of all mentions of an entity throughout the document and generates an embedding for that entity. This entity embedding is then used in the classification of all entity pairs. However, since some contexts may represent relationships unrelated to entity pairs, it is preferable to have a local representation that focuses only on relevant contexts within the document, which helps in determining the relationships between entity pairs.

[0133] Therefore, this disclosure provides local context pooling, which enhances the embedding of entity pairs by adding additional contextual embeddings associated with the two entities. In some embodiments, since this disclosure uses a pre-trained transformer-based model as encoder 122, which has already learned word-level dependencies through multi-head self-attention, this disclosure considers directly using their attention heads for local context pooling. This method transfers the well-learned dependencies from the pre-trained language model without having to learn a new attention layer from scratch.

[0134] Specifically, this disclosure uses the word-segmentation level attention head A of the last transformer layer in a pre-trained language model, where attention A ijk，1≤i≤H，1≤j，k≤l This indicates the importance of segment k to segment j in the i-th attention head out of a total of H attention heads. For entity mentions starting from the j′-th segment (“*” symbol), this disclosure will A j＝j′ Attention is then given at the mention level, and then averaged among the attentions that mention the same entity to obtain entity-level attention. Each attention This indicates the importance of the context segmentation to the i-th entity in the H attention heads (e.g., in BERT, H can be 12). Then, for entity pairs (e... s e o This disclosure obtains contextual word segmentation that is important to both entities by multiplying the entity-level attentions of the two entities and then normalizing them:

[0135]

[0136]

[0137] a (s，o) =q (s，o) / 1 T q (s，o) , which means in q (s，o) The sum of the dimensions of the vectors in the middle is normalized to 1, that is, a (s，o) The sum of the dimensions of the vectors is 1.

[0138] c (s，o) =H T a (s，o) c (s，o) The number of dimensions can be, for example, 768.

[0139] Among them, c (s，o) Yes (e) s， e o The local context embedding of ) is obtained by modifying the original linear layer in equations (3) and (4) as follows: The context embedding is fused into the pooled entity embedding to obtain different entity representations for different entity pairs:

[0140]

[0141]

[0142] Among them, W C1 , These are model parameters.

[0143] Figure 4 Local context pooling according to certain embodiments of this disclosure is illustrated schematically. Figure 4 As shown, the word segmentation in the same column is the same word segmentation at different levels, and the tags are weighted and averaged to form entity pairs (e). s e o The local context c (s ，o) The weight of the word segmentation is determined by the weight of the main entity e from the last transformer layer. s Attention weights and object entity e o This is derived by multiplying the attention weights, so that only the word segments 402 and 404, which are important to both entities, will receive higher weights.

[0144] Please note that the word 'i' may have different meanings in different contexts within this disclosure. For example, e i In this context, i is a positive integer, representing the i-th entity; In this context, i is a positive integer representing the hidden representation z. s The i-th component among the k components; In this context, i is a positive integer representing the hidden representation z. o The i-th component among the k components; A ijk，1≤i≤H，1≤j，k≤l In this context, i is a positive integer between 1 and H, representing the i-th attention among H attentions; In this context, i is a positive integer between 1 and m, representing the attention of the i-th entity.

[0145] Return to reference Figure 1When the document preparation module 120, encoder 122, and classifier 124 perform relation prediction, function module 126 is configured to perform a function using the predicted relations. In some embodiments, this function is to construct a knowledge graph, and function module 126 is configured to incorporate entity pairs and their predicted relations into the knowledge graph. Each entity can be a node in the knowledge graph, and a relation can be an edge linking the corresponding entity. In some embodiments, this function is to retrieve information from a database, and function module 126 is configured to train encoder 122 and classifier 124 using a training dataset from the database, infer relations from the database after training, and provide entity pairs and their relations to the user. In some embodiments, this function is a question-answering system, and function module 126 is configured to extract entities from a question, infer entity relations from an answer database or a comment database, use the entities extracted from the question and the inferred relations to form an answer to the question, and provide the answer to the user who asked the question.

[0146] Interface 128 is configured to provide an interface to the administrator of ATLOP relation extraction application 118 to train encoder 122 and classifier 124 and adjust model parameters, or to provide an interface to users to use ATLOP relation extraction application 118 to obtain answers to questions, use documents to construct or complete knowledge graphs.

[0147] Figure 5 The training process of an ATLOP relation extraction application according to certain embodiments of the present disclosure is schematically depicted. In some embodiments, the training process is performed by... Figure 1 The computing device 110 shown is implemented. It should be noted that, unless otherwise stated in this disclosure, the steps of the training process or method may be arranged in a different order, and therefore are not limited to... Figure 5 The order shown.

[0148] like Figure 5 As shown, in step 502, the document preparation module 120 retrieves the training data 130 and provides the training data 130 to the encoder 122. The training data consists of documents with labeled entities and relationships.

[0149] In step 504, for each document, encoder 122 adds the symbol "*" at the beginning and end of the entity reference, or in other words, adds the symbol "*" before and after the entity reference.

[0150] In step 506, encoder 122 takes the asterisk "*" at the beginning of a mention as the word representing that mention, calculates the entity embedding using logsumexp, and sends the entity embedding to classifier 124. Specifically, encoder 122 has a basic encoder structure similar to BERT language models and obtains the embedding for each word in the training document, i.e.,

[0151] [h1, h2, ..., h t , ..., h l ] = BERT([x1, x2, ..., x...) t , ..., x l ]) (1).

[0152] The embedding of each word is represented by a vector. Encoder 122 then uses the embeddings of the words corresponding to entity mentions to obtain the entity embeddings via logsumexp, i.e.,

[0153]

[0154] In step 508, when the entity embedding is received from encoder 122, classifier 124 calculates the local context pooling (local context embedding) of the entity pair using the following equation:

[0155]

[0156]

[0157] a (s，o) =q (s，o) / 1 T q (s，o) ,

[0158] c (s，o) =H T a (s，o) .

[0159] In step 510, classifier 124 uses entity embedding and local context pooling to compute the hidden state of the entity. Specifically, for relation prediction of entity pairs containing a subject entity and an object entity, the hidden state of the entity is computed using the following equation:

[0160]

[0161]

[0162] In step 512, after obtaining the hidden states of the entities in the entity pair, classifier 124 uses group bilinearity to determine the logit between entities:

[0163]

[0164] In step 514, for the logit between entity pairs corresponding to each relation, classifier 124 compares the determined logit with the logit corresponding to an adaptive threshold (TH class) for that relation. If the logit is equal to or greater than the logit function of the threshold, the relation is determined to exist; if the probability is less than the threshold, the relation is determined not to exist. Because a document can include multiple mentions of an entity pair and multiple relations, an entity pair can have one or more defined relations.

[0165] In step 516, classifier 124 calculates the loss function based on the adaptive threshold using the following equation:

[0166]

[0167]

[0168] L = L1 + L2.

[0169] In step 518, the loss function is fed back to the model to adjust the parameters of encoder 122 and classifier 124, and another round of prediction is performed to optimize the model.

[0170] Therefore, steps 506–518 are iteratively performed on the same document until the loss L converges to a small value, or until a predetermined number of iterations has been reached. Then, steps 502–518 are performed on another document in the training data. In some embodiments, each training round is performed in batches, and each batch includes multiple documents, such as four documents.

[0171] Figure 6 The inference process of the ATLOP association extraction application, according to certain embodiments of the present disclosure, is schematically depicted after the ATLOP association extraction application has been trained. In some embodiments, the inference process is performed by... Figure 1 The computing device 110 shown is implemented. It should be noted that, unless otherwise stated in this disclosure, the steps of the training process or method may be arranged in a different order, and are therefore not limited to... Figure 6 The order shown is important. Note that the training data for ATLOP and the documents used for predictions with ATLOP should be in the same domain. For example, training ATLOP using Wikipedia data can be used to infer common sense from articles, while training ATLOP using biomedical data can be used to infer the relationship between genes and diseases from biomedical papers.

[0172] like Figure 6 As shown, in step 602, the document preparation module 120 retrieves the prediction data 132 and provides the prediction data 132 to the encoder 122. The prediction data is a document; entities in the document may or may not be provided, and there are no relation tags.

[0173] In step 604, for each document, encoder 122 identifies entities from the document using named entity recognition, such as spaCy or Stanza, and adds an asterisk "*" at the beginning and end of any mention of the identified entity in the document. Preferably, a list of entities and tags is provided, thus eliminating the need for named entity recognition.

[0174] In step 606, encoder 122 uses the symbol "*" at the beginning of a mention as the word segment representing that mention, calculates the entity embedding using logsumexp, and sends the entity embedding to classifier 124. Specifically, encoder 122 has a basic encoder structure such as BERT language models and obtains the embedding of each word in the training document, i.e.

[0175] [h1, h2, ..., h t , ..., h l ] = BERT([x1, x2, ..., x...) t , ..., x l ]) (1).

[0176] The embedding of each word is represented by a vector. Encoder 122 then uses the embeddings of the words corresponding to entity mentions to obtain the entity embeddings via logsumexp, i.e.

[0177]

[0178] In step 608, when the entity embedding is received from encoder 122, classifier 124 calculates the local context pooling (local context embedding) of the entity pair using the following equation:

[0179]

[0180]

[0181] a (s，o) =q (s，o) / 1 T q (s，o) ,

[0182] c (s，o) =H T a (s，o) .

[0183] In step 610, classifier 124 uses entity embedding and local context pooling to compute the hidden state of entities. Specifically, for relation prediction of entity pairs containing subject and object entities, the hidden state of an entity is computed using the following equation:

[0184]

[0185]

[0186] In step 612, after obtaining the hidden states of the entities in the entity pair, classifier 124 uses group bilinearity to determine the logit between entities:

[0187]

[0188] In step 614, for the logic between entity pairs corresponding to each relation, classifier 124 determines the probability and applies it to an adaptive threshold (TH class) corresponding to that relation, which is determined by... Figure 5 The training process shown is obtained from either the training process or TH (logit). TH Compare the logit of ) to the logit, if the logit r The logit function is greater than the threshold. TH If the probability is less than a threshold, the relationship is determined to exist; otherwise, it is determined not to exist. Because a document can include multiple mentions of an entity pair and multiple relationships, an entity pair can have one or more defined relationships. The classifier 124 then sends the entity pairs and corresponding relationships to the functional module 126. Thus, through inference, the entities in the document and the relationships between them can be obtained.

[0189] In step 616, after receiving entity pairs and their corresponding relationships, functional module 126 performs a function. This function may, for example, be to construct or complete a knowledge graph using entities as nodes and relationships as edges; or to provide an answer to a question, wherein entities are extracted from the question and entity pairs and relationships are extracted from a database related to the question.

[0190] In some aspects, this disclosure relates to a non-transitory computer-readable medium storing computer-executable code. In some embodiments, the computer-executable code may be software stored in storage device 116 as described above. When executed, the computer-executable code may perform one of the methods described above.

[0191] experiment

[0192] Datasets: The experiments conducted demonstrate the advantages of certain embodiments of the ATLOP application disclosed herein. The datasets used in the experiments include DocRED, CDR, and GDA, such as... Figure 7As shown in Table 1. DocRED (Yao et al. 2019) is a large-scale general dataset for document-level RE constructed from Wikipedia articles. It consists of 3053 manually annotated training documents. For entity pairs representing relations, approximately 7% of the entity pairs have multiple relation labels. CDR (Li et al. 2016) is a manually annotated dataset in the biomedical field. It consists of 500 training documents. The task is to predict binary interactions between the concepts of chemistry and disease. GDA (Wu et al. 2019b) is a large-scale dataset in the biomedical field. It consists of 29192 training articles. The task is to predict binary interactions between the concepts of gene and disease. The experiment followed Christopoulou, Miwa, and Ananiadou (2019) to split the training set into 80 / 20, serving as the training set and the development set, respectively.

[0193] Experimental Setup: The model disclosed in this publication is implemented based on PyTorch2 and Transformers3 using Huggingface. We use cased BERT-base (Devlin et al. 2019) or RoBERTa-large (Liu et al. 2019) as the encoder on DocRED, and cased SciBERT-base (Beltagy, Lo, and Cohan 2019) on CDR and GDA. We use mixed-precision training based on Apex library4 (Micikevicius et al. 2018). Our model is optimized using AdamW (Loshchilov and Hutter 2019) with learning rates ∈ {2e-5, 3e-5, 5e-5, 1e-4}, linearly warmed up for the first 6% of steps (Goyal et al. 2017), and then linearly decayed to 0. All hyperparameters were tuned on the development set. The hyperparameters for all datasets are as follows: Figure 8 As shown in Table 2.

[0194] For models using a global threshold, we searched for the threshold from {0.1, 0.2, ..., 0.9} and chose the value that maximized dev F1. All models were trained using a single Tesla V100 GPU. For the DocRED dataset, training the BERT-base encoder took approximately 1 hour and 45 minutes, and training the RoBERTa-large encoder took approximately 3 hours and 30 minutes. For the CDR and GDA datasets, training the SciBERT-base encoder took 20 minutes and 3 hours and 30 minutes, respectively.

[0195] Main results: We compared ATLOP with sequence-based models, graph-based models, and transformer-based models on the DocRED dataset. Experimental results are as follows. Figure 9 As shown in Table 3. Following Yao et al. (2019), we used F1 and Ign F1 for evaluation. Ign F1 represents the F1 score, which does not include the fact of shared training and development / test sets.

[0196] Sequence-based models. These models use neural architectures such as CNN (Goodfellow, Bengio, and Courville 2015) and bidirectional LSTM (Schuster and Paliwal 1997) to encode the entire document, then obtain entity embeddings, and use bilinear functions to predict the relations of each entity pair.

[0197] Graph-based models. These models construct document graphs by learning the latent graph structure of documents and use graph convolutional networks for inference (Kipf and Welling 2017). We incorporate two state-of-the-art graph-based models, AGGCN (Guo, Zhang, and Lu 2019) and LSR (Nan et al. 2020), for comparison. The results for AGGCN are obtained from a reimplementation of Nan et al. (2020).

[0198] Transformer-based models. These models can directly adapt pre-trained language models to document-level language processing (RE) without using graph structures. They can be further divided into pipelined models (BERT-TS (Wang et al. 2019a)), hierarchical models (HIN-BERT (Tang et al. 2020a)), and pre-trained methods (CorefBERT and CorefRoBERTa (Ye et al. 2020)). We also included the BERT baseline (Wang et al. 2019a) in our comparison.

[0199] We found that our reimplemented BERT baseline significantly outperforms Wang et al. (2019a) and surpasses the state-of-the-art RNN-based model BiLSTM LSR by 1.2%. This demonstrates that pre-trained language models can capture long-range dependencies between entities without explicitly using graph structures. After integrating other techniques, our enhanced baseline BERT-EBASE achieves an F1 score of 58.52%, approaching the state-of-the-art model BERT-LSRBASE. Our BERT-ATLOPBASE model further improves the performance of BERT-EBASE by 2.6%, demonstrating the effectiveness of the two proposed new techniques. Using RoBERTa-large as the encoder, our ALTOP model achieves an F1 score of 63.40%, a new state-of-the-art result on DocRED. As of September 9, 2020, we ranked #5 on the Colab leaderboard.

[0200] Results from biomedical datasets: Experimental results from two biomedical datasets are as follows Figure 10 As shown in Table 4, Verga, Strubell, and McCallum (2018) and Nguyen and Verspoll (2018) are sequence-based models that use self-attention networks and CNNs as encoders, respectively. Christopoulou, Miwa, and Ananiadou (2019) and Nan et al. (2020) use graph-based models that construct document graphs through heuristic or structured attention and use graph neural networks for inference. To our knowledge, transformer-based pre-trained language models have not yet been applied to document-level RE datasets in the biomedical field. In our experiments, we used SciBERT... BASE Instead of an encoder, SciBERT BAsE It was pre-trained on a multi-domain corpus of scientific publications. SciBERT BAsE Our baseline is already superior to all existing methods. Our SciBERTATLOP BAsE The model further improved the F1 scores of CDR and GDA by 4.3% and 1.4%, respectively, and produced new state-of-the-art results on both datasets.

[0201] Ablation Studies: To demonstrate the effectiveness of our proposed technique, we conducted two sets of ablation studies on ATLOP and enhanced baseline by shutting down one component at a time. Figure 11As shown in Table 5, we observed that all components contribute to model performance. Adaptive thresholding and local context pooling are equally important for model performance, causing a decrease in dev F1 score of 0.89% and 0.97%, respectively, when removed from ATLOP. Note that adaptive thresholding is only effective when optimizing the model using adaptive threshold loss. Applying adaptive thresholding to a model trained with binary cross-entropy yielded a dev F1 score of 41.74%.

[0202] For our enhanced baseline model BERT-E BAsE Both grouped bilinear and logsumexp pooling lead to an approximately 1% increase in dev F1. We found that the improvement in entity labeling is small (0.24% in dev F1), but we still use this technique in the model because it makes the derivation of mention embeddings and mention-level attention easier.

[0203] Threshold Analysis: The global threshold does not consider the variation of model confidence across different classes or instances, resulting in suboptimal performance. An interesting question is whether we can improve the global threshold by adjusting different thresholds for different classes. Therefore, we attempted to use a loop optimization algorithm (Fan and Lin 2007) to adjust the class-dependent thresholds to maximize the F1 score on the DocRED development set. The results are as follows... Figure 12 As shown in Table 6, we found that using per-class thresholds significantly improved the dev F1 score to 61.73%, even exceeding the results of adaptive thresholding. However, this gain did not carry over to the test set. The results of per-class thresholding were even worse than those of global thresholding. Our adaptive thresholding technique, on the other hand, uses a learnable threshold that can be automatically generalized to the test set.

[0204] Context Pooling Analysis: To demonstrate that our Local Context Pooling (LOP) technique solves the multi-entity problem, we divided documents in the DocRED development set into different groups based on the number of entities and evaluated the models trained using local context pooling or not for each group. Experimental results are as follows: Figure 13 As shown, we observed that for both models, their performance deteriorates as documents contain more entities. Unless a document contains very few entities (1 to 5), the model with LOP consistently outperforms the model without LOP, and this improvement becomes increasingly significant as the number of entities increases. However, the number of documents containing only 1 to 5 entities is very small (4 in the development set), and documents in DocRED contain an average of 19 entities. Therefore, our local context pooling still significantly improves the overall F1 score. This demonstrates that local context pooling captures the relevant context of entity pairs, thus addressing the multi-entity problem.

[0205] We also visualized Figure 2The example uses context weights. Figure 14 As shown, our local context pooling assigns high weights to birth and death, which are most relevant to the two entities (John Stanstreet, Bendigo). These two segmentations also provide evidence for the baseline relationships of birthplace and deathplace, respectively. Segments like election and politician receive much lower weights because they are only relevant to the subject entity John Stanstreet. The visualization demonstrates that local context can locate contexts relevant to these two entities.

[0206] In summary, certain embodiments of this disclosure provide an ATLOP model for document-level relation extraction, incorporating at least two novel techniques: adaptive thresholding and local context pooling. The adaptive thresholding technique replaces the global threshold in multi-label classification with a learnable threshold class that determines the optimal threshold for each entity pair. Local context pooling leverages pre-trained attention heads to localize the relevant context of entity pairs, thereby aiding in solving the multi-entity problem. Experiments on three public document-level relation extraction datasets demonstrate that our ATLOP model significantly outperforms existing models and produces new state-of-the-art results on all datasets.

[0207] The ATLOP model has many downstream applications in other NLP tasks, such as knowledge graph construction, information retrieval, question answering, and dialogue systems.

[0208] The foregoing description of exemplary embodiments of this disclosure is presented for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit this disclosure to its precise form. Many modifications and variations can be made in accordance with the above teachings.

[0209] The embodiments were chosen and described to explain the principles of this disclosure and its practical application, so that others skilled in the art can utilize this disclosure and various embodiments and make various modifications according to the particular purpose contemplated. Alternative embodiments will become apparent to those skilled in the art to which this disclosure pertains without departing from the spirit and scope of this disclosure. Therefore, the scope of this disclosure is defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

Claims

1. A system comprising a computing device comprising a processor and a storage device storing computer executable code, wherein, The computer executable code, when executed at the processor, is configured to: provide a document; embed a plurality of entities in the document into a plurality of embedding vectors; and predict, based on a first embedding vector and a second embedding vector, one of a plurality of relationships between a first entity in the document and a second entity in the document, the first embedding vector of the plurality of embedding vectors representing the first entity, the second embedding vector of the plurality of embedding vectors representing the second entity, wherein the computer executable code is configured to embed and predict using a language model stored in the computing device, each of the plurality of relationships having an adaptive threshold, for the one of the plurality of relationships, the relationship is determined to exist when a logit of the relationship is greater than a logit function of a respective one of the adaptive thresholds of the plurality of relationships; the computer executable code is configured to predict one of a plurality of relationships by computing a local contextual pooling of a pair of entities selected from the plurality of entities using the following equation: ， ， and ， wherein the entity pair comprises a subject entity and an object entity, is a word-level attention head of the subject entity, is a word-level attention head of the object entity, is and a product, wherein H is a number of attention heads, is an i-th multiplication in H multiplications, is normalized to sum 1, wherein is a last layer embedding of the pre-trained language model, is the local context pooling of the entity pair.

2. The system of claim 1, wherein, the computer executable code is configured to embed each of the plurality of entities by summarizing at least one hidden representation of at least one mention of each of the plurality of entities using a LogSumExp, LSE.

3. The system of claim 1, wherein, hidden states of the subject entity and the object entity are determined by the following equation: and , wherein, is an embedding of the subject entity, is a hidden state of the subject entity, is an embedding of the object entity, is a hidden state of the object entity, , , and are model parameters.

4. The system of claim 3, wherein, the computer executable code is configured to predict a relationship between the subject entity and the object entity using the following equation: ， wherein, is the subject entity and the object entity k is a positive integer, the dimension of is divided by k to form a plurality of , the dimension of is divided by k to form a plurality of , and are model parameters; and wherein the subject entity has the relation r with the object entity when the logit function of a learnable threshold TH of the relation r is greater than the relation r. and the object entity 5. The system of claim 4, wherein, dimensions of the dimensions of the 6. The system of claim 1, wherein, the language model comprises at least one of the following: Bidirectional Encoder Representations from Transformers, BERT, Robustly optimized BERT approach, roBERTa, SciBERT, Generative Pretrained Model, GPT, GPT-2, and Reparameterization Transformer XL network, XLnet.

7. The system of claim 6, wherein, a loss function used to train the language model is determined by the following equation: , , , and , wherein, is the subject entity and the object entity k is a positive integer, the dimension of is divided by k to form a plurality of , the dimension of is divided by k to form a plurality of , and is a model parameter, TH is a learnable threshold of the relation, denotes a positive relation class, NT denotes a negative relation class; is the subject entity and the object entity with respect to the relation , is a logit function of the learnable threshold TH, the relation denotes a relation belonging to the union of the negative relation class and the learnable threshold.

8. The system of claim 1, wherein, the computer executable code is further configured to construct a knowledge graph using the first entity, the second entity, and the predicted relationship of the plurality of relationships between the first entity and the second entity.

9. The system of claim 1, wherein, when a question includes the first entity and the second entity, and the document is predetermined to include an answer to the question, the computer executable code is further configured to form the answer using the predicted relationship of the plurality of relationships.

10. A method comprising: providing, by a computing device, a document; embedding, by the computing device, a plurality of entities in the document into a plurality of embedding vectors; and predicting, by the computing device, based on a first embedding vector and a second embedding vector, one of a plurality of relationships between a first entity in the document and a second entity in the document, the first embedding vector of the plurality of embedding vectors representing the first entity, the second embedding vector of the plurality of embedding vectors representing the second entity, wherein the embedding and predicting steps are performed using a language model stored in the computing device, each of the plurality of relations having an adaptive threshold, for the one of the plurality of relations, the relation is determined to exist when the logit of the relation is greater than the logit function of a respective one of the adaptive thresholds of the plurality of relations; the predicting step includes computing a local contextual pooling of a pair of entities selected from the plurality of entities using the following equation: , , and , wherein the entity pair comprises a subject entity and an object entity, is a word-level attention head of the subject entity, is a word-level attention head of the object entity, is and a product of, H in the above equation is the number of attention heads, is the i-th multiplication in H multiplications, is normalized to sum 1, in the above equation, is the last layer embedding of the pre-trained language model, is the local context pooling of the entity pair.

11. The method of claim 10, wherein, the step of embedding each of the plurality of entities is performed by using a LogSumExp, LSE, summary of at least one hidden representation of at least one mention of each of the plurality of entities.

12. The method of claim 10, wherein, the hidden states of the subject entity and the object entity are determined using the following equation: and , wherein, is an embedding of the subject entity, is a hidden state of the subject entity, is an embedding of the object entity, is a hidden state of the object entity, , , and are model parameters.

13. The method of claim 12, wherein, the step of predicting a relation between the subject entity and the object entity is performed using the following equation: , wherein, is the subject entity and the object entity k is a positive integer, the dimension of is divided by k to form a plurality of , the dimension of is divided by k to form a plurality of , and are model parameters; and wherein the subject entity has the relation r with the object entity when the logit function of a learnable threshold TH of the relation r is greater than the relation r. has the relation r.

14. The method of claim 13, wherein, the language model includes a Bidirectional Encoder Representations from Transformers, BERT, or SciBERT, a loss function used to train the language model is determined using the following equation: , , and , wherein, is the subject entity and the object entity is a logit function for the relation r, k is a positive integer, the dimension of is divided by k to form a plurality of , the dimension of is divided by k to form a plurality of , and is a model parameter, TH is a learnable threshold for the relation, denotes a positive relation class, NT denotes a negative relation class; is the subject entity and the object entity is a logit function for the relation , is a logit function for the learnable threshold TH, the relation denotes a relation belonging to the union of the negative relation class and the learnable threshold.

15. The method of claim 10, further comprising: constructing a knowledge graph using the first entity, the second entity, and the predicted relation of the plurality of relations between the first entity and the second entity; or when a question includes the first entity and the second entity, and the document is predetermined to include an answer to the question: forming the answer using the predicted relation of the plurality of relations.

16. A non-transitory computer-readable medium storing computer-executable code, wherein, the computer executable code, when executed at a processor of an active computing device, is configured to: provide a document; embed a plurality of entities in the document into a plurality of embedding vectors; and predict a relation of a plurality of relations between a first entity in the document and a second entity in the document based on a first embedding vector and a second embedding vector, the first embedding vector of the plurality of embedding vectors representing the first entity, the second embedding vector of the plurality of embedding vectors representing the second entity, wherein the computer executable code is configured to embed and predict using a language model stored in the non-transitory computer readable medium, each of the plurality of relations having an adaptive threshold, for the one of the plurality of relations, the relation is determined to exist when the logit of the relation is greater than the logit function of a respective one of the adaptive thresholds of the plurality of relations; wherein the computer executable code is configured to predict a relation of the plurality of relations by computing a local contextual pooling of a pair of entities selected from the plurality of entities using the following equation: , , and ， wherein the entity pair comprises a subject entity and an object entity, is a word-level attention head of the subject entity, is a word-level attention head of the object entity, is and a product of, wherein H is a number of heads, is an i-th multiplication in H multiplications, is normalized to sum 1, wherein is a last layer embedding of the pre-trained language model, is the local context pooling of the entity pair.

17. The non-transitory computer readable medium of claim 16, wherein, the hidden states of the subject entity and the object entity are determined using the following equation: and wherein, is an embedding of the subject entity, is a hidden state of the subject entity, is an embedding of the object entity, is a hidden state of the object entity, , , and are model parameters; and wherein the computer executable code is configured to predict a relation between the subject entity and the object entity using the following equation: wherein, is the subject entity and the object entity k is a positive integer, the dimension of is divided by k to form a plurality of , the dimension of is divided by k to form a plurality of , and are model parameters, when the logit function of the relationship r is greater than a learnable threshold value TH of the relationship r, the subject entity and the object entity have the relationship r.

18. The non-transitory computer-readable medium of claim 16, wherein, The language model comprises a bidirectional encoder representation from transformers (BERT) from a transformer, and a loss function used for training the language model is determined by the following formula: , , and , wherein, is the subject entity and the object entity is a positive integer, the dimension of is divided by k to form a plurality of , the dimension of is divided by k to form a plurality of , and is a model parameter, TH is a learnable threshold of the relation, denotes a positive relation class, NT denotes a negative relation class; is the subject entity and the object entity is a logit function of relation , is a logit function of a learnable threshold TH, relation denotes a relation belonging to the union of the negative relation class and the learnable threshold.

Images