Document-level relation extraction method based on balanced labels

Abstract

The invention discloses a document-level relation extraction method based on balanced labels, and belongs to the field of relation extraction in the field of natural language processing. The implementation method comprises the following steps of: collecting a relationship between document data of an application field and entities concerned by the field; and based on the entity types and the positions of the entities in the relation semantics, performing permutation and combination to obtain balanced relation tags, adding the balanced relation tags into a relation tag set, and labeling the relation tags existing among the entities of the document data set. A document-level relation extraction model is established, a base coding model is selected, a trilinear pooling module is added to fuse expressions of subject entities, object entities and related contexts, and the weight of the contexts during reasoning is improved. And constructing a loss function for improving the training of the document-level relation extraction model, and improving the confidence score of the relation label obtained by reasoning of the document-level relation extraction model. And the document-level relation extraction model is trained, and the document-level relation extraction model with the best performance is selected for practical application, so that the accuracy of document-level entity relation extraction is improved.

Description

Technical Field

[0001] The present invention relates to a document-level relationship extraction method based on balanced labels, and specifically to a method for adding balanced relationship labels between entities according to entity types to improve the accuracy of the relationship labels between entities extracted by the document-level relationship extraction model, which belongs to the field of relationship extraction in the field of natural language processing. Technical Background

[0002] Text data is a very important part of today's Internet data. Extracting effective structured information from unstructured text data is an important task in the field of natural language processing, namely, information extraction. The basis for clarifying the information in the text is to distinguish the relationship between different entities. The relationship extraction technology between entities aims to give the relationship between different entities in text data, and occupies an important position in the information extraction task. The extracted relationship triples can be used for downstream tasks such as knowledge graph construction, human-computer question and answer, and information retrieval, and are one of the important supporting technologies for machines to understand text data.

[0003] In order to extract the relationship between entities, the current common technical method is to design, train and apply deep neural network models for relationship extraction based on the characteristics of text datasets and relationship tag sets. Traditional relationship extraction models limit the training and application of relationship extraction models to reasoning about the relationship between entities in a single sentence, that is, sentence-level relationship extraction models. Now, thanks to the improvement of computing power, relationship extraction models are used to solve the problem of extracting the relationship between entities expressed in documents composed of multiple sentences, which involves the design, training and application of document-level relationship extraction models.

[0004] The method for constructing application document-level relationship extraction includes the following steps: first, collect document data in the application field to form a document dataset {D}, and collect the relationship label set {R} between entities of interest in the field. Design the architecture of the document-level relationship extraction model, and then train and fine-tune the document-level relationship extraction model. Preprocess the document dataset {D} and the relationship label set {R} as the input and supervision of the document-level relationship extraction model, so that the trained relationship extraction model can understand and distinguish the semantics of each relationship label in the relationship label set {R} and can make inferences about the relationship labels between entities. Finally, use the trained document-level relationship extraction model to infer the relationship labels between entities in the document using the document data to be processed, and give an entity relationship triplet representation of (head entity, relationship label, tail entity).

[0005] The research and application of document-level relation extraction methods can expand the scope of application of relation extraction applications, and it has received increasing attention in the research and application of natural language processing. For entity pairs consisting of two entities, the document-level relation extraction model needs to infer the relation label subset {r} that exists in the entity pair under the document semantics from the relation label set {R}. Such application requirements require that the document-level relation extraction model can distinguish the semantic differences of the relation labels in the relation label set {R}, and require the document-level relation extraction model to infer the most accurate relation labels possible.

[0006] The training goal of the document-level relation extraction model is to obtain a multi-label, binary classification, deep neural network model that can determine whether a certain relation label exists between entities, and distinguish between the relation label set {r} that exists in the document semantics and the relation label set {Rr} that does not exist on the relation label set {R} that the user is interested in. The general method requires setting a threshold label to separate {R} and {Rr}, and using a model training loss function corresponding to a threshold label, so that the model learns the relation extraction capability of using the confidence score of the threshold label to separate the {R} and {Rr} label sets.

[0007] The design of a single threshold label and a training loss function in the general method has led to the following problems in document-level relationship extraction models in applications: 1. The relationship labels in the relationship label set {R} that users pay attention to are semantically similar and lack relevance. 2. The number of labels in the relationship label set {r} between entity pairs is unbalanced, and the model tends to infer and predict that there are no relationship labels between entity pairs. 3. The general loss function causes the document-level relationship extraction model to learn a shortcut to reduce training loss during training: the training loss is reduced by suppressing the confidence scores of relationship labels in the non-existent relationship label set {Rr}, which makes the difference between the confidence scores of relationship labels in the expected inferred relationship label set {r} and the confidence scores of threshold labels not obvious enough, ultimately resulting in suboptimal accuracy performance in document-level relationship extraction applications. Summary of the invention

[0008] In view of the problems and challenges found in the application of the prior art methods, the purpose of the present invention is to provide a document-level relationship extraction method based on balanced labels, which uses balanced relationship labels to expand the relationship label set {R}, increase the correlation between relationship labels, and help the document-level relationship extraction model understand the semantics of relationship labels; supplement the annotation of balanced relationship labels to the documents in the training set and the validation set, and reduce the imbalance in the number of labels in the relationship label set {r} between entity pairs; construct and adopt a loss function suitable for balanced labels to train the document-level relationship extraction model, improve the confidence score of the relationship labels in the relationship label set {r} given by the document-level relationship extraction model, and improve the accuracy of document-level relationship extraction.

[0009] The purpose of the present invention is achieved through the following technical methods.

[0010] The present invention discloses a document-level relationship extraction method based on balanced labels, which collects document data in an application field and the relationship between entities of interest in the field. Based on the entity type and the position of the entity in the relationship semantics, the balanced relationship labels are arranged and combined to add to the relationship label set, and the relationship labels existing between the entity pairs in the document data set are annotated as supervisory signals for document-level relationship extraction model training. A document-level relationship extraction model is built, a base encoding model is selected, and a trilinear pooling module is added to fuse the representations of the subject entity, the object entity, and the related context, thereby increasing the weight of the context during reasoning. An improved loss function for document-level relationship extraction model training is constructed to increase the confidence score of the relationship label obtained by reasoning the document-level relationship extraction model. The hyperparameters of the training document-level relationship extraction model are adjusted on the validation set, and the document-level relationship extraction model with the best performance on the test set is selected for practical application, thereby obtaining a higher accuracy rate for document-level entity relationship extraction.

[0011] The present invention discloses a document-level relationship extraction method based on balanced tags, comprising the following steps:

[0012] Step 1: To actually put the document-level relationship extraction in field A into use, first collect the document data in field A to form a document dataset {D}, and at the same time collect the relationships between entities that field A is concerned about to form a relationship tag set {R}. Divide the total document dataset {D} into a training set, a validation set, and a test set. The training set is used to train the document-level relationship extraction model, the validation set is used to quickly verify the fine-tuning parameters of the document-level relationship extraction model, and the test set is used to measure and compare the relationship extraction effect of the document-level relationship extraction model that will eventually be put into use. Perform named entity recognition on each document d in the total document dataset {D} to obtain the location annotation and type information of the entity in the document. The set of all entity types is recorded as {C}.

[0013] Document d by S d The document d consists of sentences, and the set of entities contained in document d is where ||{e d}|| is the number of entities in document d. Entity e has entity type c∈{C}, [C} is the entity type set. Entity e will appear in multiple "entity mentions" at different locations in document d. The "entity mention" set of entity e is recorded as where ||{m e}|| is the number of “entity mentions” of entity e in document d. h Because it is located in the position of the subject entity in relational semantics, it is called the head entity. t Because it is located in the position of the object entity in relational semantics, it is called the tail entity.

[0014] For the head and tail entity pairs in the document (e h ,e t ), make head and tail entity pairs based on the relational tag set {R} concerned by domain A (e h ,e t ) is labeled with a relation label. For any relation label r * ∈{R}, if (e h ,e t ), label r * Can form relational semantic triples (e h ,r * ,e t ), then add the relationship label r * to the expected relation label set {r} for model reasoning. If (e h ,e t ) Under the document semantics, there does not exist any relation tag r in the relation tag set {R} * , then the expected relation label set {r should only contain “NA” relation labels. h ,e t )Entity pairs are inferred by the model, and the confidence score of the “NA” relation label is used to determine (e h ,e t ) Whether there is a relationship label r * , the relationship label r whose confidence score is greater than the "NA" relationship label * It was judged as (e h ,e t ) relation label, so the “NA” relation label is called the threshold relation label.

[0015] Entity pairs in documents (e h ,e t ) After the preliminary relationship labeling, the expected relationship label set is {r} = r 1 ,r 2 ,...}. Let the head entity e h , tail entity e t The entity types are c h Type and c t Type, you can base it on c h Type and c t Type, c h Type and c t Type of positional relationship, supplement the annotation entity pair (e h ,e t ) of the balanced relation label b, the semantics of label b is c h Type and c tThe relationship labels when the types appear at the head and tail positions respectively, the expected relationship label set with balanced relationship label annotations used for training is {r b}={r 1 ,r 2 ,...,b}, b and some of the relationship tags in the original relationship tag set {R} will have a co-occurrence correlation, which is conducive to the model's understanding and distinguishing the semantics of the relationship tags in the relationship tag set {R}. The other introduced balanced relationship tags are also defined by the entity type and the position of the type appearing at the head and tail. The number of types in the entity type set {C} is ||{C}||, and the entities of the ||{C}|| entity types can appear at the head or tail of the entity pair. The total number of balanced relationship tags that need to be added to the relationship tag set {R} after permutation and combination is ||{C}|| 2 , and obtain the relation label set {R b}.

[0016] Supplement the balanced relationship labels in the training set and validation set documents so that the expected relationship label set {r} of each entity pair contains at least the balanced relationship label, which reduces the entity pair (e h ,e t ) and improve the correlation between relation labels in document datasets.

[0017] Step 2: For the document dataset collected in step 1, including training set, validation set and test set, d Insert the start special token "[CLS]" and the sentence separation special token "[SEP]" at the beginning and end of each sentence. Use a pair of asterisks "*" to highlight each entity mention m of entity e e,j , and then use the word segmenter corresponding to the base encoding model to segment the document, and get the id list of the document's word element in the word segmenter word element table [id 1 ,id 2 ,...]. The start and end positions of the entity mention are determined according to the id of the word with the *, and then a mapping table with the key being entity e and the value being the position table before the * of each entity mention is obtained.

[0018] Create a mapping table with the key being entity e and the value being the position table of the first * sign mentioned for each entity, and get {e:[m e,1 ,m e,2 ,...]}. This mapping table is the pre-processing of entity representation calculation, which reduces the coupling between entity representation calculation module and word segmentation encoding module in document-level relation extraction model. Create a mapping table with the key as entity pair (e h ,e t), for the training set and the validation set, the value of this mapping table is defined as the expected relationship label set {r b}, we get {(e h ,e t ):[r 1 ,r 2 ,...,b]}, for the test set, the value of the mapping table is defined as the expected relationship label set {r}, and we get {(e h ,e t ):[r 1 ,r 2 ,...]}. This mapping table is the pre-processing for entity pair representation calculation, document-level relationship extraction model training loss function calculation, and model evaluation on the test set, which accelerates the calculation of entity pair representation calculation module and loss function module in the document-level relationship extraction model.

[0019] Step 3: List of word ids of the documents obtained in step 2 [id 1 ,id 2 ,...] as input, and encoded using the pre-trained language model according to formula (1):

[0020] H,A=PLM([id 1 ,id 2 ,...]) (1)

[0021] Among them, PLM is the acronym for the pre-trained language encoding model (Pretrained Language Model) selected for building the document-level relationship extraction model, H is the matrix composed of the vector representation of each word element in the word element id list of the document, and t is the abbreviation of token, N t is the number of terms in the document, and h is the dimension of the vector representation. It is a matrix composed of the attention vectors of each word unit output by the last transformer layer of the pre-trained language model PLM. The attention vector of each word unit is obtained by averaging the attention vectors of different attention heads.

[0022] Using the mapping table {e:[m e,1 ,m e,2 ,...]}, get the entity mention position table [m e,1 ,m e,2 ,...]. Take out the entity mention position table [m e,1 ,m e,2,...] The vector representation of the word at each position, and then pooling is performed to fuse the vector representations of entity mentions at different positions in the document to obtain the head entity e h and tail entity e t The vector representation is denoted by H h and H t The vector representation of the head and tail entities is H h and H t After the location information fusion module of the document-level relationship extraction model, the entity vector representation Z with location information is obtained h and Z t .

[0023] Head entity h and tail entity e t The vector representation H h and H t The calculation of is calculated by LogSumExp pooling technology. Let the "entity mention" set of entity e be where ||{m e}|| is the number of “entity mentions” of entity e in document d, then the expression for obtaining the vector representation of entity e is:

[0024]

[0025] in h is the dimension of the word vector output by the pre-trained language encoding model, where is the vector representation of the jth entity mention of entity e, thus obtaining the head entity e h and tail entity e t The vector representation H h and H t .

[0026] Get the entity vector representation Z with position information h and Z t The calculation expression of the position information fusion module is:

[0027] Z e =tanh(WH e +b) (3)

[0028] in Z e is the entity vector representation with position information, h is the dimension of the word vector representation output by the pre-trained language encoding model, the matrix W and the vector b are the mathematical representations of the position information fusion module and are the learnable parameters of the position information fusion module. The position information fusion module makes the vector representation H of entity e obtained by formula (2) e , converted into a solid vector representation Z with position information e .

[0029] Step 4: In order to make the document-level relation extraction model for entity pairs (e h ,e t ) to make inferences about the relationship tags that exist in the document. It is also necessary to add a context vector representation that can support the inference. Using the matrix A composed of the attention vectors of each word obtained in step 3, we can obtain the attention vectors A of the head and tail entities for the document context word. h and A t . The attention vector A of the head and tail entities h and A t First do the Hadamard product, then do the vector normalization operation to get the entity's attention vector A for the document context word. (h,t) . Then A (h,t) Do the Hadamard product with the vector representation matrix H of the document word to get the vector representation C of the context that the head and tail entities pay attention to together. (h,t) .

[0030] First, we use the matrix A composed of the attention vectors of each word to obtain the attention vector A of entity e for the document context. e The calculation expression is:

[0031]

[0032] in h is the dimension of the word vector output by the pre-trained language encoding model, where The attention vector of the j-th entity mention of entity e for the document context word, thus obtaining the head entity e h and tail entity e t The attention vector A for the document context h and A t .

[0033] The entity pair’s attention vector A for the document context token (h,t) The calculation expression is:

[0034]

[0035] in is the operator of the Hadamard product, and Norm represents the vector normalization operation.

[0036] The vector representation C of the context that the head and tail entities pay attention to together (h,t) The calculation expression is:

[0037] C (h,t) =H T ·A (h,t) (6)

[0038] in · is the inner product operator.

[0039] Through the grouped trilinear pooling module, the fusion head entity vector representation Z h , the tail entity vector represents Z t and the context information C associated with the entity pair (h,t) , get the vector representation H of the entity pair (h,t) The vector representation of the entity pair is H (h,t) The confidence score of each relationship label is obtained through the classification module of the document-level relationship extraction model.

[0040] The entity vector with position information obtained according to formula (3) is represented by Z h and Z t , and the vector representation C of the context that the head and tail entities pay attention to together obtained by formula (6) (h,t) , these three vectors Z h , Z t and C (h,t) , and then a dimension conversion module is required before entering the grouped trilinear pooling module. The calculation formula is:

[0041] v * =v.view(g,h / g).unsqueeze((d1,d2)) (7)

[0042] in Represents Z h , Z t and C (h,t) The vector, the “view” method and the “unsqueeze” method respectively operate the vector shape into a matrix and expand the matrix dimension to support matrix multiplication for linear pooling. g is a grouping hyperparameter to reduce the parameters of the linear pooling module. h is the dimension of the word vector output by the pre-trained language encoding model. d1 and d2 are two locations where the dimensions need to be expanded. Z h Expanding the two dimensions (2,3) gives Z t Expanding the two dimensions (1,3) yields C (h,t) Expanding the two dimensions (1,2) yields

[0043] The vector representation H of the entity pair (h,t) The calculation method is First do the Hadamard product, then adjust the matrix to a one-dimensional vector:

[0044]

[0045] The vector representation of the entity pair g is a grouping hyperparameter to reduce the parameters of the linear pooling module, and h is the dimension of the word vector output by the pre-trained language encoding model.

[0046] The number of classification heads of the classification module of the document-level relation extraction model corresponds to the relation label set with balanced labels {R b The classification module of the model obtains the relationship label r * The calculation expression of the confidence score is:

[0047] P(r * |e h ,e t )=W r H (h,t) +b r (9)

[0048] Where P is the classification relationship label r of the neural network model * The confidence score of is the vector representation H of the reduced entity pair (h,t) The learnable parameters of dimension, are the learnable parameters of the classification module.

[0049] Step 5: Use the document d in the training set as the input of the document-level relationship extraction model described in steps 3 and 4 above, and after obtaining the confidence score of the relationship label inferred by the model, calculate the loss function value of the model training. Train the document-level relationship extraction model on the training set documents for multiple rounds until the training loss no longer decreases, adjust the training parameters on the validation set, and save the document-level relationship extraction model parameters with the highest accuracy on the test set for actual document-level relationship extraction.

[0050] For the expected relation label set {r b The calculation of the label inference loss in} adopts the Ranking Loss loss function algorithm. For labels in the expected relationship set {R b -r b}, the pooled Ranking Loss algorithm is used to calculate the training loss to avoid the model learning a shortcut to reduce the training loss: by suppressing entity pairs (e h ,e t ) does not exist in the relation label set {R b -r b} to reduce the training loss. h ,e t )The relationship tag set {r b There is no obvious difference between the confidence scores of the relationship tags in the relationship tag set and the confidence scores of the threshold tags.

[0051] Under the document semantics, entity pairs (e h ,e t )The relationship tag set {r b}={r 1 ,r 2 ,...,b} in the relation label r i Above, let the model predict the relationship label r i The confidence score is recorded as For the entity pair (e h ,e t ) does not exist in the relation label set {r -}={R b -r b} in the relationship tag The confidence score on The confidence score of the threshold relationship label for separating positive and negative relationships is denoted as f 0 .

[0052] The calculation expression of the training loss function of the document-level relation extraction model is as follows:

[0053]

[0054] The above training loss function aims to expand the relationship set {r b}Confidence scores of relation labels in and the confidence score f of the threshold relationship label 0 The gap between the threshold relationship labels and the confidence score f 0 With the entity pair (e h ,e t ) does not exist in the relation label set {r -} in the relationship tag The confidence score on The gap.

[0055] The document-level relationship extraction model is trained on the training set documents for multiple rounds until the training loss no longer decreases. The training parameters are adjusted on the validation set, and the document-level relationship extraction model parameters with the highest accuracy on the test set are saved for actual document-level relationship extraction.

[0056] Step 6: Apply the trained document-level relation extraction model obtained in step 5 to the relation extraction field of natural language processing, perform document-level relation extraction, and obtain document-level relation extraction results.

[0057] Step 7: Solve technical problems related to the corresponding field based on the document-level relationship extraction results obtained in step 6.

[0058] The relevant technical issues include knowledge graph construction and integration, case analysis and decision support in the legal field, intelligent document analysis and academic research, and auxiliary diagnosis and medical research in the medical field.

[0059] The document-level relationship extraction model is applied to the construction and integration of the knowledge graph. The relationship extraction model automatically identifies the relationship between different entities from a large amount of text data. The relationship information is a key element in building a knowledge graph, connecting numerous entities and their relationships to form a huge knowledge network, providing a basis for subsequent knowledge reasoning and query. For the same person, there may be different descriptions in different news reports or documents, but through relationship extraction, it is determined that the description refers to the same entity, and the relevant information is integrated into the knowledge graph.

[0060] The document-level relationship extraction model is applied to the legal field to extract the legal relationships between the parties involved in the case, such as contractual relationships, tort relationships, etc., from legal documents and case texts. This helps lawyers and judges understand the core issues and legal relationships of the case more quickly, and improves the efficiency and accuracy of case analysis.

[0061] The document-level relationship extraction model is applied to literature analysis and academic research. By extracting various entities and relationships in the literature, potential new knowledge and innovations can be discovered. For example, the intersections and potential cooperation opportunities between different research directions can be discovered to provide ideas for interdisciplinary research; problems or contradictions that have not been fully addressed can be extracted from previous research to inspire new research topics.

[0062] Applying document-level relation extraction models to the relationship analysis between diseases and symptoms, extracting the relationship between diseases and symptoms in patient medical records and medical literature, and helping doctors diagnose diseases more accurately. For example, discovering which specific symptoms are usually associated with a certain disease, or which diseases certain symptoms may indicate, provides a basis for early diagnosis and treatment of the disease.

[0063] Beneficial effects:

[0064] 1. The present invention discloses a document-level relationship extraction method based on balanced tags, which uses balanced relationship tags to expand the relationship tag set {R}. During the expansion process, the new balanced relationship tags are integrated into the original tag set through specific relationship semantics. The expansion method of the present invention cleverly increases the correlation between relationship tags, making the semantic connection between each tag closer, so that the document-level relationship extraction model can more accurately grasp the meaning represented by different relationship tags when processing documents, thereby better performing relationship extraction work.

[0065] 2. In actual document data, the relationship label set {r} between entity pairs often has an unbalanced number of labels, which affects the effect and accuracy of model training. The present invention discloses a document-level relationship extraction method based on balanced labels, which performs supplementary annotation of balanced relationship labels for documents in the training set and the validation set. This supplementary annotation method can alleviate this imbalance phenomenon, making the data more balanced, which is conducive to the stable training and accurate prediction of the document-level relationship extraction model.

[0066] 3. A document-level relationship extraction method based on balanced labels disclosed in the present invention adds a trilinear pooling module. The trilinear pooling module is used to fuse the representations of subject entities, object entities and related contexts. For the relevant information of subject entities and object entities, the module will capture the feature information of each dimension, its position characteristics in the text structure, and the potential association between it and other elements. For the relevant context, the module shows a strong integration ability. It is not just a simple collection of word information of the context, but a deep excavation of the semantic logic, emotional tendency and background knowledge contained in the context. Through the trilinear pooling operation, these rich representations of the subject entity, object entity and related context are fused together. The fusion brings significant effects, the most prominent of which is to increase the weight of the context in reasoning. In traditional models, context information is not fully valued in the reasoning process. However, after adding this trilinear pooling module, the status of context information in the reasoning link has been improved. It plays a more critical role when the model performs key reasoning steps such as relationship reasoning and semantic understanding, and helps the document-level relationship extraction model output more accurate and reasonable results.

[0067] 4. The present invention discloses a document-level relationship extraction method based on balanced labels, and constructs a loss function specifically suitable for balanced labels to train the document-level relationship extraction model. The loss function is determined after in-depth research and repeated experiments, and the loss function can have a positive impact on the parameter adjustment of the model during the training process. By using this loss function, the confidence score of the model is significantly improved when predicting the relationship labels in the existing relationship label set {r}. This enables the document-level relationship extraction model to give relationship labels more accurately when facing complex document data, thereby improving the accuracy of document-level relationship extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flowchart of a document-level relationship extraction method based on balanced tags of the present invention;

[0069] Figure 2 It is a schematic diagram of loss function design of a document-level relationship extraction method based on balanced labels of the present invention;

[0070] Figure 3 It is a system block diagram of a document-level relationship extraction method based on balanced tags of the present invention. DETAILED DESCRIPTION

[0071] In order to better illustrate the purpose and advantages of the present invention, the implementation method of the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0072] Embodiment 1:

[0073] This example uses the encyclopedia document dataset. The corpus statistics after sorting are shown in Table 1:

[0074] Table 1 Experimental data of relation extraction from Encyclopedia Document Dataset (DocRED)

[0075]

[0076] The experiment uses F1-measure (F1) and F1-measure (Ign_F1) to evaluate the results, and uses ten-fold cross validation to determine the optimal parameters of the model based on the experimental performance.

[0077] This experiment was conducted on a server with the following configurations: GPU 309024GB×2, memory 256GB, hard disk 4TB, and operating system Centos GNU / Linux 7.9.200964-bit.

[0078] The specific configuration of the embodiment model is as shown in the attached Figure 3 As shown, the embodiment adopts BERT and RoBERTa as base coding models.

[0079] like Figure 1 As shown, the document-level relationship extraction method based on balanced tags disclosed in this embodiment has the following specific implementation steps:

[0080] Step 1: As shown in Table 1, the encyclopedia document dataset consists of 5053 documents, including 3053 training set documents, 1000 validation set documents and 1000 test set documents. The encyclopedia document dataset contains 96 types of entity relationships to be extracted, and there are 6 types of entities: person type entity, organization type entity, place type entity, time type entity, number type entity and other type entity.

[0081] Entity pairs in documents (e h ,e t ) The expected relationship tag set given by the encyclopedia document dataset is {r} = {r 1 ,r 2,...}. Let the head entity e h , tail entity e t The entity types are c h Type and c t Type, you can base it on c h Type and c t Type, c h Type and c t Type of positional relationship, supplement the annotation entity pair (e h ,e t ) of the balanced relation label b, the semantics of label b is c h Type and c t The relationship labels when the types appear at the head and tail positions respectively, the expected relationship label set with balanced relationship label annotations used for training is {r b}={r 1 ,r 2 ,...,b}, b and some of the relationship tags in the original relationship tag set {R} will have a co-occurrence correlation, which is conducive to the model's understanding and distinguishing the semantics of the relationship tags in the relationship tag set {R}. The other introduced balanced relationship tags are also defined by the entity type and the position of the type appearing at the head and tail. The number of types in the entity type set {C} ||{C}||=6. The entities of these ||{C}|| entity types can appear at the head or tail of the entity pair. The total number of balanced relationship tags that need to be added to the relationship tag set {R} after permutation and combination is ||{C}|| 2 =36, and we get a relation label set with balanced labels {R b}.

[0082] Supplement the balanced relationship labels in the training set and validation set documents so that the expected relationship label set {r} of each entity pair contains at least the balanced relationship label, which reduces the entity pair (e h ,e t ) and improves the correlation between relation labels in document datasets.

[0083] Step 2: For the encyclopedia document dataset, including training set, validation set and test set, d Insert the start special token "[CLS]" and the sentence separation special token "[SEP]" at the beginning and end of each sentence. Use a pair of asterisks "*" to highlight each entity mention of entity e. e,j , and then use the word segmenter corresponding to the base encoding model to segment the document, and get the id list of the document's word element in the word segmenter word element table [id 1 ,id 2 ,...],like Figure 3"Document d" is shown. The start and end positions of the entity mention are determined based on the ID of the "*" word, and then a mapping table is obtained in which the "key is entity e and the value is the position table of each entity mention before the "*"."

[0084] Create a mapping table with the key being entity e and the value being the position table of the first * sign mentioned for each entity, and get {e:[m e,1 ,m e,2 ,...]}. This mapping table is the pre-processing of entity representation calculation, which reduces the coupling between entity representation calculation module and word segmentation encoding module in document-level relation extraction model. Create a mapping table with the key as entity pair (e h ,e t ), for the training set and the validation set, the value of this mapping table is defined as the expected relationship label set {r b}, we get {(e h ,e t ):[r 1 ,r 2 ,...,b]}, for the test set, the value of the mapping table is defined as the expected relationship label set {r}, and we get {(e h ,e t ):[r 1 ,r 2 ,...]}. This mapping table is the pre-processing for entity pair representation calculation, document-level relationship extraction model training loss function calculation, and model evaluation on the test set, which accelerates the calculation of entity pair representation calculation module and loss function module in the document-level relationship extraction model.

[0085] Step 3: List of word ids of the documents obtained in step 2 [id 1 ,id 2 ,...] as input, and encoded using the pre-trained language model according to formula (1):

[0086] H,A=PLM([id 1 ,id 2 ,...]) (1)

[0087] Among them, PLM is the acronym for the pre-trained language encoding model (Pretrained Language Model) selected for building the document-level relationship extraction model, H is the matrix composed of the vector representation of each word element in the word element id list of the document, and like Figure 3 The “word element vector representation H” in the figure is shown in Figure 1. t is the abbreviation of the word element (token), N t is the number of terms in the document, and h is the dimension of the vector representation. It is a matrix composed of the attention vectors of each word unit output by the last transformer layer of the pre-trained language model PLM. The attention vector of each word unit is obtained by averaging the attention vectors of different attention heads. Figure 3 The “attention matrix A” in .

[0088] Using the mapping table {e:[m e,1 ,m e,2 ,...]}, get the entity mention position table [m e,1 ,m e,2 ,...]. Take out the entity mention position table [m e,1 ,m e,2 ,...] The vector representation of the word at each position, and then pooling is performed to fuse the vector representations of entity mentions at different positions in the document to obtain the head entity e h and tail entity e t The vector representation is denoted by H h and H t The vector representation of the head and tail entities is H h and H t After the location information fusion module of the document-level relationship extraction model, the entity vector representation Z with location information is obtained h and Z t ,like Figure 3 As shown in the “head entity representation” and “tail entity representation” in .

[0089] Head entity h and tail entity e t The vector representation H h and H t The calculation of is calculated by LogSumExp pooling technology. Let the "entity mention" set of entity e be where ||{m e}|| is the number of “entity mentions” of entity e in document d, then the expression for obtaining the vector representation of entity e is:

[0090]

[0091] in h is the dimension of the word vector output by the pre-trained language encoding model, where is the vector representation of the jth entity mention of entity e, thus obtaining the head entity e h and tail entity e t The vector representation H h and H t .

[0092] Get the entity vector representation Z with position information h and Z t The calculation expression of the position information fusion module is:

[0093] Z e =tanh(WH e +b) (3)

[0094] in Z e is the entity vector representation with position information, h is the dimension of the word vector representation output by the pre-trained language encoding model, the matrix W and the vector b are the mathematical representations of the position information fusion module and are the learnable parameters of the position information fusion module. The position information fusion module makes the vector representation H of entity e obtained by formula (2) e , converted into a solid vector representation Z with position information e ,like Figure 3 As shown in the “head entity representation” and “tail entity representation” in .

[0095] Step 4: In order to make the document-level relation extraction model for entity pairs (e h ,e t ) to make inferences about the relationship tags that exist in the document. It is also necessary to add a context vector representation that can support the inference. Using the matrix A composed of the attention vectors of each word obtained in step 3, we can obtain the attention vectors A of the head and tail entities for the document context word. h and A t . The attention vector A of the head and tail entities h and A t First do the Hadamard product, then do the vector normalization operation to get the entity's attention vector A for the document context word. (h,t) . Then A (h,t) Do the Hadamard product with the vector representation matrix H of the document word to get the vector representation C of the context that the head and tail entities pay attention to together. (h,t) ,like Figure 3 As shown in the "Context Representation" in .

[0096] The matrix A composed of the attention vectors of each word unit is used to obtain the attention vector A of entity e for the document context. e The calculation expression is:

[0097]

[0098] in h is the dimension of the word vector output by the pre-trained language encoding model, where The attention vector of the j-th entity mention of entity e for the document context word, thus obtaining the head entity e hand tail entity e t The attention vector A for the document context h and A t .

[0099] The entity pair’s attention vector A for the document context token (h,t) The calculation expression is:

[0100]

[0101] in is the operator of the Hadamard product, and Norm represents the vector normalization operation.

[0102] The vector representation C of the context that the head and tail entities pay attention to together (h,t) The calculation expression is:

[0103] C (h,t) =H T ·A (h,t) (6)

[0104] in · is the inner product operator.

[0105] Through the grouped trilinear pooling module, the fusion head entity vector representation Z h , the tail entity vector represents Z t and the context information C associated with the entity pair (h,t) , get the vector representation H of the entity pair (h,t) The vector representation of the entity pair is H (h,t) The confidence score of each relationship label is obtained through the classification module of the document-level relationship extraction model.

[0106] The entity vector with position information obtained according to formula (3) is represented by Z h and Z t , and the vector representation C of the context that the head and tail entities pay attention to together obtained by formula (6) (h,t) , these three vectors Z h , Z t and C (h,t) , and then a dimension conversion module is required before entering the grouped trilinear pooling module. The calculation formula is:

[0107] v * =v.view(g,h / g).unsqueeze((d1,d2)) (7)

[0108] in Represents Z h , Z t and C (h,t)The vector, the “view” method and the “unsqueeze” method respectively operate the vector shape into a matrix and expand the matrix dimension to support matrix multiplication for linear pooling. g is a grouping hyperparameter to reduce the parameters of the linear pooling module. h is the dimension of the word vector output by the pre-trained language encoding model. d1 and d2 are two locations where the dimensions need to be expanded. Z h Expanding the two dimensions (2,3) gives Z t Expanding the two dimensions (1,3) yields C (h,t) Expanding the two dimensions (1,2) yields

[0109] The vector representation H of the entity pair (h,t) The calculation method is First do the Hadamard product, then adjust the matrix to a one-dimensional vector:

[0110]

[0111] The vector representation of the entity pair g is a grouping hyperparameter to reduce the parameters of the linear pooling module, and h is the dimension of the word vector output by the pre-trained language encoding model.

[0112] The number of classification heads of the classification module of the document-level relation extraction model corresponds to the relation label set with balanced labels {R b The classification module of the model obtains the relationship label r * The calculation expression of the confidence score is:

[0113] P(r * |e h ,e t )=W r H (h,t) +b r (9)

[0114] Where P is the classification relationship label r of the neural network model * The confidence score of is the vector representation H of the reduced entity pair (h,t) The learnable parameters of dimension, are the learnable parameters of the classification module.

[0115] Step 5: Use the document d in the training set as the input of the document-level relationship extraction model described in steps 3 and 4 above, and after obtaining the confidence score of the relationship label inferred by the model, calculate the loss function value of the model training. Train the document-level relationship extraction model on the training set documents for multiple rounds until the training loss no longer decreases, adjust the training parameters on the validation set, and save the document-level relationship extraction model parameters with the highest accuracy on the test set for actual document-level relationship extraction.

[0116] For the expected relation label set {r b The calculation of the label inference loss in} adopts the Ranking Loss loss function algorithm. For labels in the expected relationship set {R b -r b The calculation of label inference loss in} uses the pooled Ranking Loss algorithm to calculate the training loss, such as Figure 2 As shown. Avoid the model learning shortcuts to reduce training loss: by suppressing entity pairs (e h ,e t ) does not exist in the relation label set {R b -r b} to reduce the training loss. h ,e t )The relationship tag set {r b There is no obvious difference between the confidence scores of the relationship tags in the relationship tag set and the confidence scores of the threshold tags.

[0117] Under the document semantics, entity pairs (e h ,e t )The relationship tag set {r b}={r 1 ,r 2 ,...,b} in the relation label r i Above, let the model predict the relationship label r i The confidence score is recorded as For the entity pair (e h ,e t ) does not exist in the relation label set {r -}={R b -r b} in the relationship tag The confidence score on The confidence score of the threshold relationship label for separating positive and negative relationships is denoted as f 0 .

[0118] The calculation expression of the training loss function of the document-level relation extraction model is as follows:

[0119]

[0120] The document-level relationship extraction model is trained on the training set documents for multiple rounds until the training loss no longer decreases. The training parameters are adjusted on the validation set, and the document-level relationship extraction model parameters with the highest accuracy on the test set are saved for actual document-level relationship extraction.

[0121] Step 6: Apply the document-level relation extraction model obtained in step 5 to the encyclopedia document dataset for relation extraction.

[0122] This embodiment shows the performance effect on an encyclopedia document dataset, using a document-level relationship extraction method based on balanced labels of the present invention. The following Table 2 compares the effects of the present invention and other methods on the dataset. It can be seen that the present invention is superior to other methods and can more accurately extract relationship triples, achieving better knowledge graph construction and integration effects.

[0123] Table 2 Comparison of the effects of various methods

[0124]

[0125] In summary, the document-level relationship extraction method based on balanced labels disclosed in this embodiment is applied to an encyclopedia document dataset, and the beneficial effects are as follows:

[0126] Balanced relationship tags are used to expand the relationship tag set. During the expansion process, new balanced relationship tags are integrated into the original tag set through specific relationship semantics. The expansion method of the present invention cleverly increases the correlation between relationship tags, making the semantic connection between each tag closer, so that the document-level relationship extraction model can more accurately grasp the meaning represented by different relationship tags when processing documents, thereby better performing relationship extraction work.

[0127] Supplementary annotation of balanced relationship labels is performed on the documents in the training set and the validation set. This supplementary annotation method can alleviate the imbalance phenomenon, make the data more balanced, and contribute to the stable training and accurate prediction of the document-level relationship extraction model.

[0128] The added trilinear pooling module increases the weight of context in reasoning. In traditional models, context information is not given enough attention in the reasoning process. However, after adding this trilinear pooling module, the status of context information in the reasoning link has been improved. It plays a more critical role when the model performs key reasoning steps such as relational reasoning and semantic understanding, and helps the document-level relation extraction model output more accurate and reasonable results.

[0129] A loss function specifically suitable for balanced labels is constructed to train the document-level relationship extraction model. The loss function is determined after in-depth research and repeated experiments, and the loss function can have a positive impact on the parameter adjustment of the model during the training process. By using this loss function, the confidence score of the model is significantly improved when predicting the relationship labels in the existing relationship label set {r}. This enables the document-level relationship extraction model to give relationship labels more accurately when facing complex document data, thereby improving the accuracy of document-level relationship extraction.

[0130] Finally, a good effect was achieved in the construction and integration of knowledge graphs. The extracted relationship information is the key element in building knowledge graphs, connecting numerous entities and the relationships between them to form a huge knowledge network, providing a basis for subsequent knowledge reasoning and query.

[0131] The specific description above further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A document-level relationship extraction method based on balanced labels, characterized by: The following steps are included: Step 1: For document-level relation extraction that will be actually put into use in field A, first collect document data in field A to form a document dataset {D}, and at the same time collect the relationships between entities that field A is concerned about to form a relationship label set {R}; divide the total document dataset {D} into a training set, a validation set, and a test set; the training set is used for training the document-level relation extraction model, the validation set is used for rapid verification of the fine-tuning parameters of the document-level relation extraction model, and the test set is used to measure and compare the relation extraction effect of the document-level relation extraction model that will be finally put into use; perform named entity recognition on each document d in the total document dataset {D} to obtain the location annotation and type information of the entity in the document, and the set of all entity types is recorded as {C}; Step 2: For the document dataset collected in step 1, including training set, validation set and test set, d Insert the start special token "[CLS]" and the sentence separation special token "[SEP]" at the beginning and end of each sentence; use a pair of asterisks "*" to highlight each entity mention of entity e e,j , and then use the word segmenter corresponding to the base encoding model to segment the document, and obtain the id list [id1, id2, ...] of the document's word in the word segmenter word table; determine the start and end positions of the entity mention based on the id of the "*" word, and then obtain a mapping table where "the key is entity e, and the value is the table of positions before the *" of each entity mention"; Step 3: Take the document word id list [id1, id2, ...] obtained in step 2 as input and encode it using the pre-trained language model according to formula (1): H,A=PLM([id1,id2,...]) (1) Among them, PLM is the acronym for the pre-trained language encoding model (PretrainedLanguage Model) selected for building the document-level relationship extraction model, H is the matrix composed of the vector representation of each word element in the word element id list of the document, and t is the abbreviation of token, N t is the number of word units in the document, and h is the dimension of the vector representation; It is a matrix composed of the attention vectors of each word unit output by the last transformer layer of the pre-trained language model PLM. The attention vector of each word unit is obtained by averaging the attention vectors of different attention heads. Using the mapping table obtained in step 2, where the key is entity e and the value is the position table of the first * number mentioned by each entity, {e: [ m e,1 ,m e,2 ,...]}, get the entity mention position table [m e,1 ,m e,2 ,...]; extract the entity mention position table [m e,1 ,m e,2 ,...] The vector representation of the word at each position, and then pooling is performed to fuse the vector representations of entity mentions at different positions in the document to obtain the head entity e h and tail entity e t The vector representation is denoted by H h and H t ; The vector representation of the head and tail entities H h and H t After the location information fusion module of the document-level relationship extraction model, the entity vector representation Z with location information is obtained h and Z t ; Head entity h and tail entity e t The vector representation H h and H t The calculation is done by LogSumExp pooling technology. Let the "entity mention" set of entity e be where ||{m e }|| is the number of "entity mentions" of entity e in document d, then the expression for obtaining the vector representation of entity e is: in h is the dimension of the word vector output by the pre-trained language encoding model, where is the vector representation of the j-th entity mention of entity e, thus obtaining the head entity e h and tail entity e t The vector representation H h and H t ; Get the entity vector representation Z with position information h and Z t The calculation expression of the position information fusion module is: Z e =tanh(WH e +b) (3) in Z e is the entity vector representation with position information, h is the dimension of the word vector representation output by the pre-trained language encoding model, the matrix W and the vector b are the mathematical representations of the position information fusion module, and are the learnable parameters of the position information fusion module; the position information fusion module makes the vector representation H of the entity e obtained by formula (2) e , converted into a solid vector representation Z with position information e ; Step 4: In order to make the document-level relation extraction model for entity pairs (e h ,e t ) to make inferences about the relationship tags that exist, and it is also necessary to add a context vector representation that can support the inference; using the matrix A composed of the attention vectors of each word obtained in step 3, we can get the attention vectors A of the head and tail entities for the document context word respectively h and A t ; The attention vector A of the head and tail entities h and A t First do the Hadamard product, then do the vector normalization operation to get the entity's attention vector A for the document context word. (h,t) ; Then A (h,t) Do the Hadamard product with the vector representation matrix H of the document word to get the vector representation C of the context that the head and tail entities pay attention to together. (h,t) ; Step 5: Use the document d in the training set as the input of the document-level relationship extraction model described in steps 3 and 4 above, and after obtaining the confidence score of the relationship label inferred by the model, calculate the loss function value of the model training; train the document-level relationship extraction model on the training set documents for multiple rounds until the training loss no longer decreases, adjust the training parameters on the validation set, and save the document-level relationship extraction model parameters with the highest accuracy on the test set for actual document-level relationship extraction; Step 6: Apply the trained document-level relation extraction model obtained in step 5 to the relation extraction field of natural language processing, perform document-level relation extraction, and obtain document-level relation extraction results.

2. The document-level relationship extraction method based on balanced tags as claimed in claim 1, characterized in that: Step 1 is implemented as follows: Document d by S d The document d consists of sentences, and the set of entities contained in document d is where ||{e d }|| is the number of entities in document d; entity e has entity type c∈{C}, {C} is the entity type set; entity e will appear in multiple "entity mentions" at different locations in document d, and the "entity mention" set of entity e is recorded as where ||{m e }|| is the number of "entity mentions" of entity e in document d; e h Because it is located in the position of the subject entity in relational semantics, it is called the head entity. t It is called the tail entity because of its position after the object entity in relational semantics; For the head and tail entity pairs in the document (e h ,e t ), make head and tail entity pairs based on the relational tag set {R} concerned by domain A (e h ,e t ) relation label annotation; for any relation label r * ∈{R}, if (e h ,e t ), label r * Can form relational semantic triples (e h ,r * ,e t ), then add the relationship label r * to the expected relation label set {r} for model reasoning; if (e h ,e t ) Under the document semantics, there does not exist any relation tag r in the relation tag set {R} * , then the expected relation label set {r} inferred by the model should only contain "NA" relation labels; h ,e t )Entity pairs are inferred by the model, and the confidence score of the "NA" relation label is used to determine (e h ,e t ) Whether there is a relationship label r * , the relationship label r whose confidence score is greater than the "NA" relationship label * It was judged as (e h ,e t ) so the "NA" relationship label is called the threshold relationship label; Entity pairs in documents (e h ,e t ) After the preliminary relationship labeling, the expected relationship label set is {r} = {r1, r2, ...}; let the head entity e h , tail entity e t The entity types are c h Type and c t Type, you can base it on c h Type and c t Type, c h Type and c t Type of positional relationship, supplement the annotation entity pair (e h ,e t ) of the balanced relation label b, the semantics of label b is c h Type and c t The relationship labels when the types appear at the head and tail positions respectively, the expected relationship label set with balanced relationship label annotations used for training is {r b }={r1,r2,...,b}, b and some of the relationship tags in the original relationship tag set {R} will have a co-occurrence correlation, which is conducive to the model understanding and distinguishing the semantics of the relationship tags in the relationship tag set {R}; the other introduced balanced relationship tags are also defined by the entity type and the position of the type appearing at the head and tail; the number of types in the entity type set {C} is ||{C}||, and the entities of the ||{C}|| entity types can appear at the head or tail position of the entity pair. The total number of balanced relationship tags that need to be added to the relationship tag set {R} after permutation and combination is ||{C}|| 2 , and obtain the relation label set {R b }; Supplement the balanced relationship labels in the training set and validation set documents so that the expected relationship label set {r} of each entity pair contains at least the balanced relationship label, which reduces the entity pair (e h ,e t ) and improve the correlation between relation labels in document datasets.

3. The document-level relationship extraction method based on balanced tags as claimed in claim 2, characterized in that: Step 2 is implemented as follows: Create a mapping table with the key being entity e and the value being the position table of the first * sign mentioned for each entity, and get {e:[m e,1 ,m e,2 ,...]}; This mapping table is the pre-processing of entity representation calculation, which reduces the coupling degree between entity representation calculation module and word segmentation encoding module in document-level relation extraction model; establish a mapping table with key as entity pair (e h ,e t ), for the training set and the validation set, the value of this mapping table is defined as the expected relationship label set {r b }, we get {(e h ,e t ):[r1,r2,...,b]}, for the test set, the value of the mapping table is defined as the expected relationship label set {r}, and we get {(e h ,e t ):[r1,r2,...]}; This mapping table is the pre-processing for entity pair representation calculation, document-level relationship extraction model training loss function calculation, and model evaluation on the test set, which accelerates the calculation of entity pair representation calculation module and loss function module in the document-level relationship extraction model.

4. The document-level relationship extraction method based on balanced tags as claimed in claim 3, characterized in that: Step 4 is implemented as follows: First, we use the matrix A composed of the attention vectors of each word to obtain the attention vector A of entity e for the document context. e The calculation expression is: in h is the dimension of the word vector output by the pre-trained language encoding model, where The attention vector of the j-th entity mention of entity e for the document context word, thus obtaining the head entity e h and tail entity e t The attention vector A for the document context h and A t ; The entity pair’s attention vector A for the document context token (h,t) The calculation expression is: A (h,t) =Norm ( A h οA t ) (5) in ° is the operator of Hadamard product, and Norm represents the vector normalization operation; The vector representation C of the context that the head and tail entities pay attention to together (h,t) The calculation expression is: C (h,t) =H T ·A (h,t) (6) in · is the inner product operator; Through the grouped trilinear pooling module, the fusion head entity vector representation Z h , the tail entity vector represents Z t and the context information C associated with the entity pair (h,t) , get the vector representation H of the entity pair (h,t) ; The vector representation of the entity pair H (h,t) The confidence score of each relationship label is obtained through the classification module of the document-level relationship extraction model; The entity vector with position information obtained according to formula (3) is represented by Z h and Z t , and the vector representation C of the context that the head and tail entities pay attention to together obtained by formula (6) (h,t) , these three vectors Z h , Z t and C (h,t) , and then a dimension conversion module is required before entering the grouped trilinear pooling module. The calculation formula is: v * =v.view(g,h / g).unsqueeze((d1,d2)) (7) in Represents Z h , Z t and C (h,t) The "view" method and the "unsqueeze" method respectively operate the vector shape into a matrix and expand the matrix dimension to support matrix multiplication for linear pooling. g is a grouping hyperparameter to reduce the parameters of the linear pooling module. h is the dimension of the word vector output by the pre-trained language encoding model. d1 and d2 are two locations where the dimensions need to be expanded. Z h Expanding the two dimensions (2,3) gives Z t Expanding the two dimensions (1,3) yields C (h,t) Expanding the two dimensions (1,2) yields The vector representation H of the entity pair (h,t) The calculation method is First do the Hadamard product, then adjust the matrix to a one-dimensional vector: The vector representation of the entity pair g is a grouping hyperparameter to reduce the parameters of the linear pooling module, and h is the dimension of the word vector output by the pre-trained language encoding model; The number of classification heads of the classification module of the document-level relation extraction model corresponds to the relation label set with balanced labels {R b }; the classification module of the model obtains the relationship label r * The calculation expression of the confidence score is: P(r * |e h ,e t )=W r H (h,t) +b r (9) Where P is the classification relationship label r of the neural network model * The confidence score of is the vector representation H of the reduced entity pair (h,t) The learnable parameters of dimension, are the learnable parameters of the classification module.

5. The document-level relationship extraction method based on balanced tags as claimed in claim 4, characterized in that: Step 5 is implemented as follows: For the expected relation label set {r b The calculation of the label inference loss in} adopts the Ranking Loss loss function algorithm. For labels in the expected relationship set {R b -r b }, the pooled RankingLoss algorithm is used to calculate the training loss to avoid the model learning a shortcut to reduce the training loss: by suppressing the entity pair (e h ,e t ) does not exist in the relation label set {R b -r b } to reduce the training loss; so that the entity pair (e h ,e t )The relationship tag set {r b The confidence score of the relationship tag in the relationship tag set is not significantly different from the confidence score of the threshold tag; Under the document semantics, entity pairs (e h ,e t )The relationship tag set {r b }=relation label r in {r1,r2,...,b} i Above, let the model predict the relationship label r i The confidence score is recorded as For the entity pair (e h ,e t ) does not exist in the relation label set {r - }={R b -r b } in the relationship tag The confidence score on The confidence score of the threshold relationship label for separating positive and negative relationships is recorded as f0; The calculation expression of the training loss function of the document-level relation extraction model is as follows: The above training loss function aims to expand the relationship set {r b }Confidence scores of relation labels in The gap between the confidence score f0 of the threshold relationship label and the confidence score f0 of the entity pair {e h ,e t ) does not exist in the relation label set {r - } in the relationship tag The confidence score on the gap; The document-level relationship extraction model is trained on the training set documents for multiple rounds until the training loss no longer decreases. The training parameters are adjusted on the validation set, and the document-level relationship extraction model parameters with the highest accuracy on the test set are saved for actual document-level relationship extraction.

6. A document-level relationship extraction method based on balanced labels as described in claim 1, 2, 3, 4 or 5, characterized in that: It also includes step 7: solving relevant technical problems in the corresponding field according to the document-level relationship extraction results obtained in step 6; The relevant technical issues include knowledge graph construction and integration, case analysis and decision support in the legal field, intelligent document analysis and academic research, and auxiliary diagnosis and medical research in the medical field; Apply document-level relation extraction models to knowledge graph construction and integration. Relation extraction models automatically identify the relationships between different entities from large amounts of text data. The relationship information is a key element in constructing a knowledge graph, which connects entities and their relationships to form a knowledge network, providing a basis for subsequent knowledge reasoning and query. For the same person, there may be different expressions in different news reports or documents, but through relationship extraction, it is determined that the expressions refer to the same entity, and the relevant information is integrated into the knowledge graph. Apply the document-level relationship extraction model to the legal field, extract the legal relationship between the parties involved in the case from legal documents and case texts, and improve the efficiency and accuracy of case analysis; Apply document-level relationship extraction models to literature analysis and academic research, and mine potential new knowledge and innovations by extracting various entities and relationships in the literature; The document-level relationship extraction model is applied to the relationship analysis between diseases and symptoms, and the relationship between diseases and symptoms in patient medical records and medical literature is extracted to help doctors diagnose diseases more accurately.

Images