Document-level relation extraction method integrating self-encoding and generative language model

Abstract

The invention provides a document-level relation extraction method integrating a self-encoding language model and a generative language model, which comprises the following steps of: firstly, extracting a data set by using a document-level relation, finely adjusting a document-level relation dichotomy model which takes a small self-encoding pre-training language model as an encoder, and carrying out dichotomy prediction on whether a relation exists or not on all entity pairs in a document; filling the cue word template with the document and the entity pairs with the relationship, finely adjusting the generative large language model, and generating the relationship category of the entity pairs; and finally, aligning a generation result of the large language model with a preset relation label by utilizing a natural language reasoning model so as to realize final document-level relation extraction. The invention provides an efficient and accurate document-level relationship extraction method, overcomes the defects of a traditional method in the aspects of long text information extraction and logical reasoning, and remarkably improves the performance and reliability of document-level relationship extraction.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a document-level relationship extraction method integrating autoencoding and generative language models. Background Art

[0002] The main goal of the document-level relation extraction task is to extract the relationships between specific entities from unstructured documents. This task can generate structured entity-relation triples, which can be used to build knowledge graphs to support downstream applications such as search engines, question-answering systems, and intelligent decision-making systems.

[0003] Compared with sentence-level relation extraction, document-level relation extraction faces more challenges. First, documents usually contain more entities, which increases the number of entity pairs that need to be enumerated exponentially. Second, the model needs to search for relevant information in longer texts and perform certain logical reasoning to accurately identify the relationship between entities. Therefore, in terms of quantity and difficulty, document-level relation extraction tasks are more complex and challenging than sentence-level relation extraction.

[0004] At present, the methods for solving document-level relationship extraction tasks are mainly based on technologies such as BERT or graph neural network (GNN). However, these methods still have difficulty in making effective logical inferences on the information in the document, resulting in the accuracy of relationship extraction being difficult to meet the requirements of practical applications.

[0005] In recent years, the rise of large language models such as GPT-4 and Llama3 has demonstrated their powerful dialogue understanding and generation capabilities. These models can demonstrate amazing task performance in various traditional natural language processing fields with only prompt words. However, the operation of such large language models requires extremely high computational costs, which makes them difficult to directly apply to the field of document-level relationship extraction.

[0006] In summary, document-level relation extraction tasks face huge challenges in processing a large number of entities and performing complex reasoning. The accuracy of existing methods still needs to be improved. Although generative large language models have shown great potential in natural language processing tasks, their high computational cost limits their application in document-level relation extraction. Therefore, a more efficient and accurate method is urgently needed to meet the practical application needs of document-level relation extraction tasks. Summary of the invention

[0007] To solve the above problems, the present invention proposes a document-level relationship extraction method that integrates autoencoder and generative language model, which has higher accuracy than traditional document-level relationship extraction methods and applies generative large language model to document-level relationship extraction tasks at a lower cost.

[0008] A document-level relation extraction method integrating autoencoder and generative language model comprises the following steps:

[0009] Step S1: Using a document-level relationship extraction dataset, which includes input text, entities, and relationships between entities, fine-tune a document-level relationship binary classification model with a small autoencoder pre-trained language model as an encoder, and perform binary classification prediction on all entity pairs in the document to determine whether they have a relationship or not;

[0010] Step S2: Fill the document and the entity pairs with the relationship into the prompt word template, fine-tune the generative large language model, and generate the relationship categories of the entity pairs;

[0011] Step S3: Based on the fine-tuned document-level relationship binary classification model and the generative large language model, the document-level relationship extraction task is performed, and the generation result of the large language model is aligned with the preset relationship label using the natural language inference model.

[0012] Furthermore, the step S1 comprises:

[0013] Step S11: Preprocess the document-level relation extraction dataset by inserting a special tag before each entity mention in the document, and then use a tokenizer to process the document into the input format required by the autoencoder pre-trained language model with tokens as the smallest unit. Where l is the input length of document D after processing, w i is the token at the i-th position;

[0014] Step S12: Obtain the token embedding sequence of document D through the encoder output and attention sequence

[0015] H, A=Encoder([w1,w2,...,w l ]) = [h1, h2, ..., h l ], [a1, a2, ..., a l ]

[0016] Where d is the hidden layer dimension of the encoder, and A is the average output of all attention heads in the last encoder layer;

[0017] If the document length exceeds the input limit of the pre-trained model, the document is split into multiple text sequences with overlapping parts and the maximum length, and each is input into the model separately, and the token embeddings of the overlapping parts are averaged;

[0018] Step S13: insert a special marker in front of the entity mention as the embedding of the mention, denoted as h m ;

[0019] For any entity e i And the corresponding mention in the documentation Use the LogSumExp function to obtain the embedding of the entity;

[0020] For any entity pair <e s , e o >, calculate its context embedding through the token embedding sequence H and the attention sequence A

[0021] Step S14: embed the head and tail entities through a fully connected neural network layer and the contextual embedding of the entity pair After fusion, a tanh activation function layer is used to obtain the final fusion representation.

[0022] Step S15: Predict entity pairs through a bilinear function <e s , e o >Probability P of a relationship existing s，o , P s，o Entity pairs greater than the threshold θ1 are predicted as positive examples, i.e., there is a relationship.

[0023] Step S16: According to the binary true label y and the probability P output in step S15 s，o The above document-level relationship binary classification model is trained, where the loss function is calculated as follows:

[0024]

[0025] Among them, α is a hyperparameter used to adjust the loss weight of positive and negative samples, and γ is a hyperparameter, which is 1-P s，o The exponent of jointly adjusts the loss weights of difficult-to-classify samples and easy-to-classify samples in positive samples.

[0026] Furthermore, the step S2 comprises:

[0027] Step S21, filling the input document and the entity pairs predicted as positive in step S15 into the prompt word template;

[0028] Step S22: input the filled prompt word template into the generative large language model, and output the corresponding relationship label r′. If there are multiple relationships, each relationship is separated by “,”;

[0029] Step S23: extract the real label r in the dataset according to the document-level relationship and fine-tune the generative large language model.

[0030] Furthermore, the step S3 comprises:

[0031] Step S31: <e s , e o > and the output r′ in step S22 is in accordance with [e s ],[r′],[e o ] are put together in order to form a complete sentence as a premise;

[0032] Step S32: <e s , e o >Fill in the preset hypothesis template to generate entity pairs <e s , e o >Assumptions about arbitrary relations;

[0033] Step S33: input the premise and hypothesis into the natural language reasoning model to obtain the probability P of the hypothesis being true, where the relationship corresponding to the hypothesis with a probability greater than the threshold θ2 is the final prediction result.

[0034] Furthermore, the LogSumExp function is used to obtain the embedding of the entity, specifically: in, For entity e i The number of mentions.

[0035] Furthermore, the context embedding of any entity pair is calculated through the token embedding sequence H and the attention sequence A Specifically:

[0036]

[0037] c s，o =H T q s，o

[0038] in, is the Hadamard product, is the average of the attention vectors of all mention embeddings of the head and tail entities, is the common context attention vector of the entity pair, is the contextual embedding of the entity pair.

[0039] Furthermore, the calculation method of step S14 is:

[0040]

[0041] Among them, [] represents the concatenation operation. is a trainable parameter.

[0042] Furthermore, the probability P s，o The calculation formula is: s，o =σ(zs W r z o +b r ),in, and b r is a trainable parameter and σ is the sigmoid activation function.

[0043] The present invention uses a document-level relationship extraction dataset to fine-tune a document-level relationship binary classification model with a small autoencoder pre-trained language model as an encoder, and performs binary classification predictions on the existence or non-existence of relationships for all entity pairs in the document; then fills the document and the entity pairs with existing relationships into a prompt word template, fine-tunes a generative large language model, and generates relationship categories for entity pairs; finally, using a natural language inference model, aligns the generation results of the large language model with preset relationship labels to achieve final document-level relationship extraction. The present invention provides an efficient and accurate document-level relationship extraction method, which overcomes the shortcomings of traditional methods in long text information extraction and logical reasoning, significantly improves the performance and reliability of relationship extraction in practical applications, reduces computational costs, and extracts relationships between specified entities from unstructured documents to generate structured entity relationship triples, which can support downstream applications such as knowledge graph construction, search engines, question-answering systems, and intelligent decision-making systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of the model prediction process of an embodiment of the present invention;

[0046] Figure 2 It is a flowchart of the detailed prediction process of the document-level relationship binary classification model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In this embodiment, BERT-Base is used as the autoencoder pre-trained language model, Llama3-8B is used as the generative large language model, and T5-Base is used as the natural language inference model. All models, word segmenters, and trainers are based on the Transformers library, and additional neural networks are implemented using the Pytorch library.

[0049] The present invention provides a document-level relation extraction method integrating autoencoding and generative language models, such as Figure 1 As shown, the method comprises the following steps:

[0050] Step S1: Using the general-domain document-level relationship extraction dataset DocRED, fine-tune the document-level relationship binary classification model with a small autoencoder pre-trained language model as the encoder, and perform binary classification prediction on all entity pairs in the document, whether there is a relationship or not. The dataset includes input text, entities, and relationships between entities, and BERT-Base is used as the autoencoder pre-trained language model. The specific implementation process is as follows: Figure 2 As shown, including:

[0051] Step S11: Preprocessing the DocRED dataset

[0052] Insert a special marker "*" before each entity mention in the document, and then use the tokenizer to process the document into the input format required by BERT-Base with tokens as the smallest unit. Where l is the input length of document D after being processed by the word segmenter, w i is the token at the i-th position.

[0053] Enumerate all candidate entity pairs in the document <e s , e o >, where s, o∈[1, N], s≠o, and N is the number of entities in the document. <e s , e o >If there is any relationship r, let its binary classification label y be 1, otherwise it is 0.

[0054] Step S12: Obtain the token embedding sequence of document D through the encoder output and attention sequence

[0055] H, A=Encoder([w1,w2,...,w l ]) = [h1, h2, ..., h l ], [a1, a2, ..., a l ]

[0056] Where d=768 is the hidden layer dimension of BERT-Base, which has 12 attention heads, and A is the average of the outputs of all attention heads in the last encoding layer.

[0057] If the document length exceeds the maximum input length of BERT-Base, which is 512, the document is split into multiple text sequences with overlapping parts and a maximum length, which are input into the model separately, and the token embeddings of the overlapping parts are averaged.

[0058] For example, if the original document length is 768, the first 512 tokens and the last 512 tokens of the document are input into BERT-Base respectively, and the corresponding embedding vector sequences of the 256 overlapping tokens in the two outputs are averaged.

[0059] Step S13: insert a special mark “*” before the entity mention as the embedding of the mention, denoted as h m .

[0060] For any entity e i And the corresponding mention in the documentation Use the LogSumExp function to get the embedding of the entity:

[0061]

[0062] in, For entity e i The number of mentions of , which is a smooth approximation of the max function.

[0063] For any entity pair <e s , e o >, calculate its context embedding through the token embedding sequence H and the attention sequence A

[0064]

[0065] c s，o =H T q s，o

[0066] in, is the Hadamard product, is the average of the attention vectors of all mention embeddings of the head and tail entities, is the common context attention vector of the entity pair, is the contextual embedding of the entity pair.

[0067] Step S14: embed the head and tail entities through a fully connected neural network layer and the contextual embedding of the entity pair After fusion, a tanh activation function layer is used to obtain the final fusion representation.

[0068]

[0069] Among them, [] represents the concatenation operation. is a trainable parameter.

[0070] Step S15: Predict entity pairs using a bilinear function <e s , e o > relationship, e s With e o The probability P that there is a relationship between s，o The calculation formula is: s，o =σ(z s W r z o +b r ),in, and b r is a trainable parameter, and σ is the sigmoid activation function. s，o Entity pairs greater than the threshold θ1 are predicted as positive examples, that is, there is a relationship. In this step, θ1 should be set to a larger value, such as 0.8, to ensure a higher recall rate.

[0071] Step S16: According to the binary true label y and the probability P output in step S15 s，o The above document-level relationship binary classification model is trained, where the loss function is calculated as follows:

[0072]

[0073] Among them, α∈(0,1) is a hyperparameter used to adjust the loss weight of positive and negative samples, and γ>0 is a hyperparameter as 1-P s，o The exponent of jointly adjusts the loss weights of difficult-to-classify samples and easy-to-classify samples in positive samples.

[0074] In this step, α should be set to a larger value, such as 0.75, to increase the loss weight of positive samples and reduce the loss weight of negative samples.

[0075] Step S2: Fill the document and the entity pairs with the relationship into the prompt word template, fine-tune a generative large language model, and generate the relationship category of the entity pairs. In this embodiment, the generative large language model adopts the Llama3-8B model. The specific implementation process is as follows:

[0076] Step S21: Fill the input document and the entity pairs predicted as positive in step S15 into the prompt word template, as shown below:

[0077]

[0078] The content corresponding to [] needs to be replaced. For example:

[0079]

[0080] According to the article, the target output of the model is "Employer".

[0081] Step S22: input the filled prompt word template into the Llama3-8B model, and output the corresponding relationship label r′. If there are multiple relationships, each relationship is separated by “,”.

[0082] Step S23: extract the real label r in the dataset according to the document-level relationship and fine-tune the generative large language model.

[0083] Step S3: Perform the document-level relationship extraction task according to the document-level relationship binary classification model and the generative large language model described in steps S1 and S2. A natural language inference model is used to align the generated result of the large language model with the preset relationship label r. In this embodiment, T5-Base is used as the natural language inference model.

[0084] The specific implementation process is as follows:

[0085] Step S31: <e s , e o > and the output r′ in step S22 is in accordance with [e s ],[r′],[e o ] format to form a complete sentence as a premise.

[0086] For example, David Lean, Employer, London Films.

[0087] Step S32: <e s , e o >Fill in the preset hypothesis template as follows:

[0088]

[0089] The content corresponding to [] needs to be replaced. Generate entity pairs through the above template. <e s , e o >Assumptions about arbitrary relations.

[0090] This step requires enumerating entity pairs <e s , e o>Assumptions for all relationships, for example, for the relationship Employer, there is a corresponding assumption: The person or organization for which David Lean works or worked is London Films.

[0091] Step S33: input the premise and hypothesis into T5-Base to obtain the probability P of the hypothesis being true, wherein the relationship corresponding to the hypothesis with a probability greater than the threshold θ2 is the final prediction result.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A document-level relation extraction method integrating autoencoder and generative language model, characterized in that: The method comprises: Step S1: Using a document-level relationship extraction dataset, which includes input text, entities, and relationships between entities, fine-tune a document-level relationship binary classification model with a small autoencoder pre-trained language model as an encoder, and perform binary classification prediction on all entity pairs in the document to determine whether they have a relationship or not; Step S2: Fill the document and the entity pairs with the relationship into the prompt word template, fine-tune the generative large language model, and generate the relationship categories of the entity pairs; Step S3: Based on the fine-tuned document-level relationship binary classification model and the generative large language model, the document-level relationship extraction task is performed, and the generation result of the large language model is aligned with the preset relationship label using the natural language inference model.

2. The method according to claim 1, characterized in that: The step S1 further comprises: Step S11: Preprocess the document-level relation extraction dataset by inserting a special tag before each entity mention in the document, and then use a tokenizer to process the document into the input format required by the autoencoder pre-trained language model with tokens as the smallest unit. Where l is the input length of document D after processing, w i is the token at the i-th position; Step S12: Obtain the token embedding sequence of document D through the encoder output and attention sequence H,A=Encoder([w1,w2,…,w l ])=[h1,h2,…,h l ],[a1,a2,…,a l ] Where d is the hidden layer dimension of the encoder, and A is the average output of all attention heads in the last encoder layer; Step S13: insert a special marker in front of the entity mention as the embedding of the mention, denoted as h m , for any entity e i And the corresponding mention in the documentation Use the LogSumExp function to obtain the embedding of the entity; For any entity pair <e s , e o >, calculate its context embedding through the token embedding sequence H and the attention sequence A Step S14: embedding the head and tail entities through a fully connected neural network layer and the contextual embedding of the entity pair After fusion, a tanh activation function layer is used to obtain the final fusion representation. Step S15: Predict entity pairs through a bilinear function <e s , e o >Probability P of a relationship existing s，o , P s，o Entity pairs greater than the threshold θ1 are predicted as positive examples, that is, there is a relationship; Step S16: According to the binary true label y and the probability P output in step S15 s，o The above document-level relationship binary classification model is trained.

3. The method according to claim 1, characterized in that The step S2 further comprises: Step S21, fill the input document and the entity pairs predicted as positive in step S15 into the prompt word template; Step S22: input the filled prompt word template into the generative large language model, and output the corresponding relationship label r′. If there are multiple relationships, each relationship is separated by ","; Step S23: extract the real label r in the dataset according to the document-level relationship and fine-tune the generative large language model.

4. The method according to claim 1, characterized in that: The step S3 further comprises: Step S31: <e s , e o > and the output r′ in step S22 is in accordance with [e s ],[r′],[e o ] are put together in order to form a complete sentence as a premise; Step S32: <e s , e o >Fill in the preset hypothesis template to generate entity pairs <e s , e o >Assumptions about arbitrary relations; Step S33: input the premise and hypothesis into the natural language reasoning model to obtain the probability P of the hypothesis being true, where the probability is greater than the threshold θ, and the relationship corresponding to the hypothesis is the final prediction result.

5. The method according to claim 2, characterized in that: When training a document-level relationship binary classification model, the loss function is calculated as follows: Among them, α is a hyperparameter used to adjust the loss weight of positive and negative samples, and γ is a hyperparameter, which is 1-P s，o The exponent of jointly adjusts the loss weights of difficult-to-classify samples and easy-to-classify samples in positive samples.

6. The method according to claim 2, characterized in that The LogSumExp function is used to obtain the embedding of the entity, specifically: in, For entity e i The number of mentions.

7. The method according to claim 2, characterized in that The context embedding of any entity pair is calculated by the token embedding sequence H and the attention sequence A Specifically: c s，o =H T q s,o in, is the Hadamard product, is the average of the attention vectors of all mention embeddings of the head and tail entities, is the common context attention vector of the entity pair, is the contextual embedding of the entity pair.

8. The method according to claim 2, characterized in that: The calculation method of step S14 is: Among them, [] represents the concatenation operation. is a trainable parameter.

9. The method according to claim 2, characterized in that: Probability P s，o The calculation formula is: s，o =σ(z s W r z o +b r ),in, and b r is a trainable parameter and σ is the sigmoid activation function.

Images