Entity pair characterization method and few-sample document-level relation extraction method adopting entity pair characterization method

Abstract

The invention relates to the field of natural language processing, and provides an entity pair characterization method and a few-sample document-level relation extraction method adopting the entity pair characterization method, and the entity pair characterization method comprises the steps that the attention score of an entity pair is calculated through the attention score of the entity; weighting context embedding of the document by taking the attention score of the entity pair as a weight to obtain context representation which is more emphasized on the current entity pair; afterwards, the context representation and entity embedding of the corresponding entity are fused, entity representation retaining fine-grained entity semantics and global dependency information is obtained, and entity pair representation is constructed based on the entity representation; according to the few-sample document-level relation extraction method, the entity pair constructed by adopting the method is characterized, the diversity of instance-level distribution is reserved through the relation prototype in a vector group form, the maximum value in the vector group is used as the final matching degree, and the influence of noise on calculation is reduced, so that the problem of classification confusion caused by insufficient semantics of the relation prototype is solved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to an entity pair representation method and a few-sample document-level relationship extraction method using the same. Background Art

[0002] The task of relation extraction aims to identify entities in unstructured text sources and explore the semantic relationships between entities, thereby mapping unstructured text corpora to structured knowledge. Relation extraction is a core task in the automated construction of large-scale knowledge graphs, and also provides knowledge support for applications such as question-answering systems and search engines. In recent years, sentence-level relation extraction methods based on deep learning have achieved very high scores on benchmark datasets. However, in actual application scenarios, the vast majority of entity relations in an article require cross-sentence reasoning to be correctly identified, which has prompted researchers to shift their focus to document-level relation extraction. Compared with sentence-level relation extraction, document-level relation extraction faces the problems of a significant increase in the number of candidate pairs, severe category imbalance, and difficulty in cross-sentence semantic reasoning.

[0003] These issues have prompted researchers to explore document-level relation extraction, aiming to rapidly identify novel relation types with only a few labeled samples. Current mainstream few-shot document-level relation extraction methods are mostly based on a metric meta-learning framework. By constructing a transferable metric space, the model can classify based on the similarity between query samples and relation prototypes. For example, in "Few-Shot Document-Level Relation Extraction" published by Popovic et al., entity representations are obtained by mean pooling of all mentions, and two prototype construction methods are proposed: DL-MNAV defines the prototype representation of a relation type as the mean of all instance embeddings; DL-MNAV-SIE directly constructs the relation prototype from a single instance. In "Multi-relation Identification for Few-Shot Document-Level Relation Extraction" published by Wang et al., entity representations are obtained by mean pooling of mentions, and instance embeddings are calculated using entity-guided attention and relation-guided attention mechanisms. The mean of all instance embeddings is then used as the prototype representation of the relation type.

[0004] However, existing methods lack sufficient representation of instances, or entity pairs, and the semantics of the prototype representations of relationship types constructed from them are also insufficient, which can easily lead to classification confusion. Therefore, how to better extract category features to achieve more representative class prototypes and obtain relationship descriptions that are highly resistant to noise and highly adaptable to domains remains a major challenge that needs to be overcome. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to propose an entity pair representation method and a few-sample document-level relationship extraction method using the same, so as to solve the classification confusion caused by insufficient semantics of relationship prototypes during the prototype construction process.

[0006] The technical solution adopted by the present invention to solve the above technical problems is:

[0007] In one aspect, the present invention provides an entity pair representation method, which constructs an entity pair representation of an input entity pair according to the following steps:

[0008] For each entity in an entity pair, its attention vector is constructed using the self-attention vectors of each entity mention corresponding to it in its own document; its entity embedding is constructed using the embeddings of each entity mention corresponding to it in its own document; the self-attention vectors and embeddings of entity mentions in the own document are calculated by the pre-trained language model of the own document;

[0009] The attention vectors of the two entities contained in the entity pair are fused to obtain an entity pair attention vector of dimension L; the entity pair attention vector of dimension L is used to weight the L dimension of the context embedding matrix of the document to obtain the context representation of the entity pair; L is the number of tokens in the document; the context embedding matrix is ​​composed of the embedding concatenation of the L tokens of the document;

[0010] For each entity contained in the entity pair, use the fusion network to fuse its entity embedding with the context representation of the entity pair to which it belongs to obtain its entity representation;

[0011] The entity representations of the two entities contained in the entity pair are fused to obtain an entity pair representation of the entity pair.

[0012] Furthermore, based on the relationship between the entity pairs and the documents, the entity pairs are grouped by document, including:

[0013] B1. Input the document corresponding to the group into the pre-trained language model to obtain the embedding of L tokens of the document with a dimension of H. The self-attention layer of the pre-trained language model obtains the self-attention vector of L tokens of the document. Using the embedding of L tokens of the document, a context embedding matrix of dimension L×H is constructed for the document.

[0014] B2. For each entity pair in the group, use the self-attention vector and embedding obtained in step B1 to construct its entity pair representation;

[0015] In step B1, the documents corresponding to the group are input into the pre-trained language model, including:

[0016] B11. Segment the input document to obtain the initial word sequence;

[0017] B12. In the initial word sequence, insert the entity tag of each entity at the entity mention corresponding to the first occurrence of the entity, and the entity tag is inserted at a position adjacent to the starting position of the corresponding entity mention;

[0018] In step B2, for each entity contained in the entity pair, the self-attention vectors of its various entity mentions in the corresponding document are used for average pooling to construct its attention vector, where the entity mentions adjacent to the entity tag use the self-attention vector of the entity tag as its self-attention vector; the embeddings of its various entity mentions in the corresponding document are used for average pooling to construct its entity embedding, where the entity mentions adjacent to the entity tag use the entity embedding of the entity tag as its entity embedding.

[0019] Furthermore, the fusion network is a linear network, and the two entities contained in the entity pair correspond to independent fusion networks respectively, and are fused according to the following formulas:

[0020]

[0021] in, represents the entity representation of the head entity in the entity pair, represents the entity representation of the tail entity in the entity pair, h represents the entity embedding of the head entity in the entity pair, t represents the entity embedding of the tail entity in the entity pair, and C (s,o) is the context representation, tanh represents the tanh activation function, [·] represents concatenation, and W h and W t is the learnable weight, b h and b t is the learnable bias.

[0022] Furthermore, according to the following formula, the attention vectors of the two entities contained in the entity pair are fused to obtain the entity pair attention vector with a dimension of L:

[0023]

[0024] in, and Represents the attention vectors of the two entities contained in the entity pair, A (s,o) Represents the entity pair attention vector, T represents matrix transpose, and ⊙ represents element-by-element multiplication;

[0025] According to the following formula, the entity pair attention vector with dimension L of the entity pair is used to weight the L dimension of the context embedding matrix of the document to obtain the context representation of the entity pair:

[0026] C (s,o) =A (s,o) ·H T

[0027] Among them, H is the context embedding matrix, C (s,o) is the context representation, T represents the matrix transpose;

[0028] According to the following formula, the entity representations of the two entities contained in the entity pair are fused to obtain the entity pair representation of the entity pair:

[0029]

[0030] Among them, s (h,r,t) Represents entity pair representation, represents the entity representation of the head entity in the entity pair, It represents the entity representation of the tail entity in an entity pair, and [·] represents concatenation.

[0031] On the other hand, the present invention also provides a method for extracting document-level relations at a small number of samples. According to the entity pair representation method described above, an entity pair feature extraction module is constructed. The training of the relationship extraction model includes:

[0032] A1. Input samples for this round of training; the samples include support set, query set and candidate relation set;

[0033] A2. For each entity pair included in the supporting entity pair set of the input sample, using the entity pair feature extraction module to obtain its entity pair representation; the supporting entity pair set includes related entity pairs, and each related entity pair included in the supporting entity pair set is composed of the head entity and the tail entity of each triple included in the triple set of the sample's support set;

[0034] A3. For each relationship type included in the candidate relationship set of the input sample, construct a relationship prototype in the form of a vector group using the entity pair representations of each triple contained in its prototype set; the triples contained in the prototype set of the relationship type are obtained by sampling a set number of triples from the support set of the sample according to the relationship type;

[0035] A4. Based on the entity pair representations of each entity pair contained in the query entity pair set and the relationship prototypes of each relationship type in the candidate relationship set, calculate the matching score between each entity pair and each relationship type combination according to the following formula:

[0036] logit=sim(q n ,P r )=max 1≤k≤K [sim(q n ,s k )]

[0037] Among them, sim(q n ,s k ) represents the entity pair representation q of the nth entity pair in the query entity pair set n and the constituent relationship prototype P r The similarity between the kth vector in the k vectors; max 1≤k≤K [sim(q n ,s k )] represents the entity pair representation q n The maximum value of the similarity between K vectors; logit represents the matching score; each entity pair contained in the query entity pair set is composed of two combinations of each entity contained in the sample query set, and its entity pair representation is calculated using the entity pair feature extraction module;

[0038] A5. Based on the matching score, construct the prediction loss for this round of training and reversely update the entity pair feature extraction module;

[0039] A6. Determine whether the training is completed. If so, end the training; otherwise, return to step A1.

[0040] Furthermore, step A2 further includes: for each relationship type included in the candidate relationship set of the input sample, first, extracting its name and description from the relationship dictionary, and concatenating them to form its relationship description text; then, inputting the relationship description text into the pre-trained language model to obtain its relationship representation; and then, constructing its relationship description prototype based on its relationship representation;

[0041] In step A3, for each relation type contained in the candidate relation set of the input sample, the relation prototype P in the form of a vector group is constructed using the entity pair representations of each triple contained in its prototype set and the relation description prototype of the relation type according to the following formula: r :

[0042] P r =[s1⊕s2⊕…,⊕s i …,⊕s I ⊕P r-rel ]

[0043] Among them, s i is the entity pair representation of the relationship entity pair corresponding to the i-th triple in the prototype set, I represents the number of triples contained in the prototype set; P r-rel It describes the prototype of the relationship, and ⊕ represents the vector concatenation operation.

[0044] Furthermore, for each relationship type included in the candidate relationship set of the input sample, first, extract its name and description from the relationship dictionary respectively, and concatenate them to form its relationship description text in the following way:

[0045] r=[CLS:r name :r dec :SEP]

[0046] Among them, CLS and SEP are the start and end markers respectively; name is the word sequence of the name of the relation type, r dec A sequence of words describing the type of relationship;

[0047] Then, the relationship description text r is input into the pre-trained language model; the embedding of the [CLS] position output by the pre-trained language model is used as the global representation of the relationship description text; the embedding of each word in the relationship description text output by the pre-trained language model is average pooled to obtain the local representation of the relationship description text;

[0048] Then, according to the following formula, the global representation r of the fusion relation description text is cls and local representation r Avg , obtain the relationship representation r of this relationship type final :

[0049] r final =λ·r cls +(1-λ)·r Avg

[0050] Among them, λ is the fusion weight.

[0051] Furthermore, based on the relationship representation of the relationship type, a relationship description prototype is constructed, which includes the following steps:

[0052] First, for each triple contained in the prototype set of the input relation type, the entity pair representation s of the corresponding relation entity pair is calculated. i , and the relationship representation r of this relationship type final The semantic alignment between sim(s i ,r final );

[0053] Then, according to the following formula, the semantic alignment of each triple contained in the prototype set of the relationship type is used to represent the relationship type r final Make corrections to obtain the relationship description prototype P of this relationship type r-rel :

[0054] P r-rel =α·r final

[0055]

[0056] Where K is the number of triplets contained in the prototype set; sim(·) represents the similarity measurement function.

[0057] Furthermore, the supporting entity pair set of the sample includes relational entity pairs and unrelational entity pairs; the unrelational entity pairs are entity pairs consisting of entities included in the triple set of the sample and not belonging to relational entity pairs, and the two entities included in the unrelational entity pairs belong to the same document;

[0058] According to the following formula, sample the set number of unrelated entity pairs, use the entity pair representations of each unrelated entity pair obtained by sampling, and construct a relationship prototype P with a relationship type of unrelated NOTA :

[0059] P NOTA =[s1⊕s2⊕…,⊕s j …,⊕s J ]

[0060] Among them, s j is the entity pair representation of the jth unrelated entity pair obtained by sampling, J represents the number of samples of unrelated entity pairs; ⊕ represents the vector splicing operation;

[0061] In step A4, it also includes: based on the entity pair representations of each entity pair included in the query entity pair set and the relationship prototype of no relationship, calculating the matching score between each entity pair included in the query entity pair set and the relationship type of no relationship.

[0062] Furthermore, in step A5, first, the prediction loss L is calculated based on the matching score between each entity pair and the relationship type combination according to the following formula: ATLoss Then, the prediction loss of each entity pair and each relationship type is summed up to obtain the prediction loss of this round of training:

[0063] L aTLoss =L T-NOTA +L NOTA-N

[0064]

[0065] Among them, P r Represents the set of relationship types to which the entity pair belongs in the triple set of the sample; N T Indicates that the candidate relation set is excluding P r A collection of other relationship types other than the included relationship types; NOTA represents a relationship type with no relationship; {P r ∪NOTA} represents P rThe set of contained relationship types and NOTA relationship types; {N T ∪NOTA} represents N T The set of included relationship types and NOTA relationship types; logit n Represents the entity pair with P r The matching score of the relation prototype of the nth relation type in ; Represents entity pairs with {P r The matching score of the relation prototype of the m1-th relation type in ∪NOTA}; Represents entity pairs with {N T The matching score of the relation prototype of the m2-th relation type in ∪NOTA}.

[0066] The beneficial effects of the present invention are:

[0067] The entity pair representation method of the present invention calculates entity representations through an entity position-aware attention mechanism in order to generate richer and more accurate entity representations and better identify and distinguish the boundaries and types of different entities. It calculates the entity representations by the attention score of the entity corresponding to the entity mention; calculates the entity embedding of the entity by the embedding of the entity corresponding to the entity mention; uses the attention score of the entity pair as the weight to weight the context embedding matrix of the document to obtain a context representation that emphasizes the current entity pair; finally, the context representation is fused with the entity embedding of the corresponding entity to obtain the final entity representation of the corresponding entity. Among them, the entity embedding of the entity is derived from the word embedding of the entity mention, which captures the semantics of the word itself; the context representation dynamically filters out the core context associated with the target entity pair through the entity position-aware attention mechanism. Therefore, by fusing the two, the basic semantics of the word are retained, and its specific meaning in the context is taken into account, and an entity representation and an entity pair representation that retain fine-grained entity semantics and global dependency information are obtained.

[0068] The few-sample document-level relationship extraction method of the present invention adopts the entity pair representation method of the present invention to construct entity pair representation, constructs a prototype set according to a set number through sampling, and constructs a relationship prototype in the form of a vector group using the entity pair representation of each triple contained in the prototype set, retaining the diversity of instance-level distribution; when calculating the matching degree, the calculation is performed in the form of a vector group, and the maximum value is used as the final matching degree to reduce the impact of noise on the calculation. As long as there is a high-matching match in the vector group, the overall matching degree evaluation can be effectively improved, avoiding the classification confusion problem caused by insufficient semantics of the relationship prototype. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Schematic diagram of the sample composition;

[0070] Figure 2 A prototype space distribution diagram of the training sample set obtained when the embodiment model is initially implemented;

[0071] Figure 3 A prototype space distribution diagram of a test sample set obtained after model training in an embodiment;

[0072] Figure 4 is the prototype spatial distribution map of the MNAV model;

[0073] Figure 5 This is the prototype spatial distribution diagram of the MNAV-SIE-SBN model. DETAILED DESCRIPTION

[0074] The present invention aims to propose an entity pair representation method and a few-sample document-level relationship extraction method using the same, in order to solve the classification confusion caused by insufficient semantics of relationship prototypes during prototype construction.

[0075] The entity pair representation method constructs the entity pair representation of the input entity pair according to the following steps:

[0076] For each entity in an entity pair, its attention vector is constructed using the self-attention vectors of each entity mention corresponding to it in its own document; its entity embedding is constructed using the embeddings of each entity mention corresponding to it in its own document; the self-attention vectors and embeddings of entity mentions in the own document are calculated by the pre-trained language model of the own document;

[0077] The attention vectors of the two entities contained in the entity pair are fused to obtain an entity pair attention vector of dimension L; the entity pair attention vector of dimension L is used to weight the L dimension of the context embedding matrix of the document to obtain the context representation of the entity pair; L is the number of tokens in the document; the context embedding matrix is ​​composed of the embedding concatenation of the L tokens of the document;

[0078] For each entity contained in the entity pair, use the fusion network to fuse its entity embedding with the context representation of the entity pair to which it belongs to obtain its entity representation;

[0079] The entity representations of the two entities contained in the entity pair are fused to obtain an entity pair representation of the entity pair.

[0080] It introduces an entity position-aware attention mechanism to calculate entity representations, which can better identify and distinguish the boundaries and types of different entities and avoid ambiguity caused by homonyms.

[0081] The few-shot document-level relationship extraction method uses the above method to construct entity pair representations and calculates matching scores in the form of vector groups. The maximum value is used as the final matching score. The vector group preserves the diversity of instance-level distribution and avoids using the mean method to mask the true distribution of the data. As long as there is a high-matching match in the vector group, the overall matching score evaluation can be effectively improved.

[0082] Therefore, the method of the present invention can avoid the classification confusion problem caused by insufficient semantics of relation prototypes.

[0083] Secondly, we introduce relationship description prototypes into the vector group of the few-shot document-level relationship extraction method to strengthen the prototype construction. When extracting relationship representations, we use dual-view fusion to obtain more discriminative relationship representations. Specifically, for each relationship type included in the candidate relationship set of the input sample, we first extract its name and description from the relationship dictionary and concatenate them to form its relationship description text as follows:

[0084] r=[CLS:r name :r dedc :SEP]

[0085] Among them, CLS and SEP are the start and end markers respectively; name is the word sequence of the name of the relation type, r dec A sequence of words describing the type of relationship;

[0086] Then, the relationship description text r is input into the pre-trained language model; the embedding of the [CLS] position output by the pre-trained language model is used as the global representation of the relationship description text; the embedding of each word in the relationship description text output by the pre-trained language model is average pooled to obtain the local representation of the relationship description text;

[0087] Then, according to the following formula, the global representation r of the fusion relation description text is cls and local representation r Avg , obtain the relationship representation r of this relationship type final :

[0088] r final =λ·r cls +(1-λ)·r Avg

[0089] Among them, λ is the fusion weight.

[0090] To avoid directly injecting relation representations, which may introduce irrelevant information, generate noise, or lead to information redundancy, we also generate relation-aware weights based on the similarity between each supporting instance and the relation representation. This is used to weight the confidence of the relation representation and form a semantically enhanced composite prototype representation. Specifically, based on the relation representation of the relation type, we construct its relation description prototype, which includes the following steps:

[0091] First, for each triple contained in the prototype set of the input relation type, the entity pair representation s of the corresponding relation entity pair is calculated. i , and the relationship representation r of this relationship type final The semantic alignment between sim(s u ,r final );

[0092] Then, according to the following formula, the semantic alignment of each triple contained in the prototype set of the relationship type is used to represent the relationship type r final Make corrections to obtain the relationship description prototype P of this relationship type r-rel :

[0093] P r-rel =α·r final

[0094]

[0095] Where K is the number of triplets contained in the prototype set; sim(·) represents the similarity measurement function.

[0096] In addition, in order to help the model better identify and process entity pairs that do not contain candidate relations, the few-sample document-level relation extraction method of the present invention also introduces the NOTA prototype, which is a special mechanism for processing "no-relationship" relationship types. Specifically, the supporting entity pair set of the sample includes entity pairs with relations and entity pairs without relations; the entity pairs without relations are composed of the entities contained in the triple set of the sample and do not belong to entity pairs with relations, and the two entities contained in the entity pairs without relations belong to the same document;

[0097] According to the following formula, sample the set number of unrelated entity pairs, use the entity pair representations of each unrelated entity pair obtained by sampling, and construct a relationship prototype P with a relationship type of unrelated NOTA :

[0098] P NOTA =[s1⊕s2⊕…,⊕s j …,⊕s J ]

[0099] Among them, s jis the entity pair representation of the jth unrelated entity pair obtained by sampling, J represents the number of samples of unrelated entity pairs; ⊕ represents the vector splicing operation;

[0100] In step A4, it also includes: based on the entity pair representations of each entity pair included in the query entity pair set and the relationship prototype of no relationship, calculating the matching score between each entity pair included in the query entity pair set and the relationship type of no relationship.

[0101] The following is further described with reference to the embodiments.

[0102] Example:

[0103] The training of the relationship extraction model of the few-sample document-level relationship extraction method of this embodiment includes:

[0104] A1. Data Preparation

[0105] In this step, samples for this round of training are input, which include a support set, a query set, and a candidate relation set.

[0106] The candidate relation set R contains candidate D that matches the relation extraction task r relation types, the support set includes D S supporting documents, the query set includes D Q query documents.

[0107] like Figure 1 As shown, both the support documents and the query document are documents that have completed entity and entity mention annotations. The support documents provide instances, that is, triples representing entities and relationships between entities; while the query document does not provide instances. Therefore, the triple set of the sample's support set is the set of triple subsets of each support document contained in the support set. The triple subset of the support document is the set of triples representing the relationships between entities in the support document.

[0108] Each entity pair contained in the query entity pair set of the sample is composed of two combinations of each entity contained in the query set of the sample. The support entity pair set of the sample includes relational entity pairs and unrelational entity pairs. Each relational entity pair contained in the support entity pair set is composed of the head entity and tail entity of each triple contained in the triple set of the support set of the sample; each unrelational entity pair contained in the support entity pair set is composed of each entity contained in the triple set of the sample and is not a relational entity pair, and the two entities contained in the unrelational entity pair belong to the same document.

[0109] The triples contained in the prototype set of each relationship type included in the candidate relationship set are obtained by sampling a set number of triples from the support set of the sample according to the relationship type. The prototype set of the unrelated relationship type is constructed by sampling a set number of unrelated entity pairs. In this embodiment, the sampling method is random sampling, and the set number is set to 10 based on experimental results.

[0110] A2. Extract entity pair representations of entity pairs supporting entity pair sets and relationship description prototypes of relationship types

[0111] In this step, the entity pair feature extraction module is used to obtain the entity pair representation for each entity pair included in the supporting entity pair set of the input sample. Specifically, in this embodiment, to improve computational efficiency, the entity pair feature extraction module first groups the entity pairs by document based on the relationship between the entity pairs and the documents, including:

[0112] B1. Input the document corresponding to the group into the pre-trained language model to obtain the embedding of the L tokens of the document with a dimension of H; obtain the self-attention vector of the L tokens of the document with a dimension of L from the self-attention layer of the pre-trained language model; use the embedding of the L tokens of the document to construct the context embedding matrix of the document with a dimension of L×H.

[0113] B2. For each entity pair in the group, use the self-attention vector and embedding obtained in step B1 to construct its entity pair representation.

[0114] In step B1, the documents corresponding to the group are input into the pre-trained language model, including:

[0115] B11. Segment the input document to obtain the initial word sequence;

[0116] B12. In the initial word sequence, insert the entity tag of each entity at the entity mention corresponding to the first occurrence of the entity, and insert the entity tag at a position adjacent to the start position of the corresponding entity mention. In this embodiment, the pre-trained language model is the BERT model, and the entity tag uses "*".

[0117] In step B2, for each entity in the entity pair, the self-attention vectors of each entity mention in the corresponding document are average-pooled to construct its attention vector. Entity mentions adjacent to the entity tag use the entity tag's self-attention vector as their self-attention vector. The entity embeddings of each entity mention in the corresponding document are average-pooled to construct their entity embeddings. Entity mentions adjacent to the entity tag use the entity embedding of the entity tag as their entity embedding. The entity marker "*" marks the first mention of the entity immediately adjacent to the entity. Its corresponding embedding combines both the entity mention itself and its surrounding context information. This better reflects the core semantics of the entity in the current context than simply averaging the embeddings of all entity mentions of the entire entity.

[0118] In step B2, after obtaining the attention vector of the entity, the attention vectors of the two entities contained in the entity pair are fused according to the following formula to obtain the entity pair attention vector with a dimension of L, where L is the number of tokens in the document:

[0119]

[0120] in, and Represents the attention vectors of the two entities contained in the entity pair, A (s,o) Represents the entity pair attention vector, T represents matrix transpose, and ⊙ represents element-wise multiplication.

[0121] In step B2, the entity pair attention vector with dimension L of the entity pair is weighted on the L dimension of the context embedding matrix of the document to obtain the context representation of the entity pair according to the following formula:

[0122] C (s,o) =A (s,o) ·H T

[0123] Among them, H is the context embedding matrix, C (s,o) is the context representation, and T represents the matrix transpose.

[0124] In step B2, for each entity included in the entity pair, a fusion network is used to fuse its entity embedding with the contextual representation of the entity pair to obtain its entity representation. The fusion network is a linear transformation network. The two entities included in the entity pair correspond to independent fusion networks, and the fusion is performed according to the following formula:

[0125]

[0126] in, represents the entity representation of the head entity in the entity pair, represents the entity representation of the tail entity in the entity pair, h represents the entity embedding of the head entity in the entity pair, t represents the entity embedding of the tail entity in the entity pair, and C (s,o) is the context representation, tanh represents the tanh activation function, [·] represents concatenation, and W h and W t is the learnable weight, b h and b t is the learnable bias.

[0127] In step B2, the entity representations of the two entities contained in the entity pair are fused according to the following formula to obtain the entity pair representation of the entity pair:

[0128]

[0129] Among them, s (h,r,t) Represents entity pair representation, represents the entity representation of the head entity in the entity pair, It represents the entity representation of the tail entity in an entity pair, and [·] represents concatenation.

[0130] In this step, for each relationship type contained in the candidate relationship set of the input sample, a relationship description prototype is constructed, including:

[0131] C1. Extract the name and description from the relationship dictionary respectively, and concatenate them to form the relationship description text as follows:

[0132] r=[CLS:r name :r dec :SEP]

[0133] Among them, CLS and SEP are the start and end markers respectively; name is the word sequence of the name of the relation type, r dec A sequence of words describing the type of relationship.

[0134] C2. Input the relationship description text into a pre-trained language model and use the embedding at the [CLS] position output by the pre-trained language model as the global representation of the relationship description text. Average pooling is performed on the embeddings of each word in the relationship description text output by the pre-trained language model to obtain local representations of the relationship description text. In this embodiment, the pre-trained language model is the BERT model.

[0135] C3, according to the following formula, the global representation r of the fusion relation description text cls and local representation r Avg , obtain the relationship representation r of this relationship type final :

[0136] r final =λ·rcls +(1-λ)·r Avg

[0137] Among them, λ is the fusion weight;

[0138] C4. For each triple contained in the prototype set of the input relation type, calculate the entity pair representation s of the corresponding relation entity pair according to the following formula: i , and the relationship representation r of this relationship type ftnal The semantic alignment between sim(s i ,r final ):

[0139]

[0140] Among them, sim(·) is the similarity measurement function, s i is the entity pair representation of the relationship entity pair corresponding to the i-th triple in the prototype set, r final is the relation representation, and ||·|| represents the norm.

[0141] C5. According to the following formula, the semantic alignment of each triple contained in the prototype set of this relationship type is used to represent the relationship type r final Make corrections to obtain the relationship description prototype P of this relationship type r-rel :

[0142] P r-rel =α·r final

[0143]

[0144] Here, K is the number of triplets contained in the prototype set.

[0145] In this embodiment, the relationship dictionary is constructed based on Wikipedia data and includes 755 relationship name-relationship description pairs. For example, the relationship name "date of birth" corresponds to the relationship description "date on which the subject was born" and the reference code is "P569".

[0146] A3. Build relationship prototypes

[0147] In this step, for each relationship type contained in the candidate relationship set of the input sample, the entity pair representations of each triple contained in its prototype set are used to construct a relationship prototype in the form of a vector group.

[0148] Specifically, for each relation type contained in the candidate relation set of the input sample, the following formula is used to construct the relation prototype P in the form of a vector group using the entity pair representation of each triple contained in its prototype set and the relation description prototype of the relation type: r :

[0149] P r =[s1⊕s2⊕…,⊕s i …,⊕s I ⊕P r-rel ]

[0150] Among them, s i is the entity pair representation of the relationship entity pair corresponding to the i-th triple in the prototype set, I represents the number of triples contained in the prototype set; P r-rel It describes the prototype of the relationship, and ⊕ represents the vector concatenation operation.

[0151] According to the following formula, construct the relationship prototype P with no relationship type NOTA :

[0152] P NOTA =[s1⊕s2⊕…,⊕s j …,⊕s J ]

[0153] Among them, s j is the entity pair representation of the j-th unrelated entity pair obtained by sampling, J represents the number of samples of unrelated entity pairs; ⊕ represents the vector splicing operation.

[0154] A4. Matching Calculation

[0155] In this step, based on the entity pair representations of each entity pair contained in the query entity pair set, as well as the relationship prototypes of each relationship type and the relationship prototypes without relationships in the candidate relationship set, the matching scores between each entity pair and each relationship type combination are calculated according to the following formula:

[0156] logit=sim(q n ,P r )=max 1≤k≤K [sim(q n ,s k )]

[0157] Among them, sim(q n ,s k ) represents the entity pair representation q of the nth entity pair in the query entity pair set n and the constituent relationship prototype P r The similarity between the kth vector in the K vectors; max 1≤k≤K [sim(q n ,sk )] represents the entity pair representation q n The maximum similarity between the K vectors; logit represents the matching score. The entity pair representations of each entity pair contained in the query entity pair set are similarly obtained by using the entity pair feature extraction module and the method in step A3. In this embodiment, sim(q n ,s k ) is measured using cosine similarity.

[0158] A5. Parameter Update

[0159] In this step, the prediction loss of this round of training is constructed based on the matching score, and the entity pair feature extraction module is updated in reverse. In this embodiment, specifically, first, the prediction loss L is calculated based on the matching score between each entity pair and the relationship type combination according to the following formula: ATLoss ; Then, the prediction loss of this round of training is obtained by summing up the prediction losses of each entity pair and each relationship type.

[0160] L aTLoss =L T-NOTA +L NOTA-N

[0161]

[0162]

[0163] Among them, P r Represents the set of relationship types to which the entity pair belongs in the triple set of the sample; N T Indicates that the candidate relation set is excluding P r A collection of other relationship types other than the included relationship types; NOTA represents a relationship type with no relationship; {P r ∪NOTA} represents P r The set of contained relationship types and NOTA relationship types; {N T ∪NOTA} represents N T The set of included relationship types and NOTA relationship types; logit n Represents the entity pair with P r The matching score of the relation prototype of the nth relation type in ; Represents entity pairs with {P r The matching score of the relation prototype of the m1-th relation type in ∪NOTA}; Represents entity pairs with {N T The matching score of the relation prototype of the m2-th relation type in ∪NOTA}.

[0164] A6. End judgment

[0165] In this step, it is determined whether the training is completed. If so, the training is ended; otherwise, the process returns to step A1.

[0166] After completing the training and obtaining the model, the actual relationship extraction task can be carried out. In the usage stage, first, the sample used for the query is input into the relationship extraction model obtained through training. The sample has the same composition as the sample in the training stage; then, according to the method of steps A2 to A4, the matching scores between each entity pair contained in the query entity pair set of the input sample and each relationship type in the candidate relationship set are obtained; finally, based on the matching scores, the relationship type of each entity pair contained in the query entity pair set of the input sample is determined.

[0167] Experimental verification:

[0168] In order to verify the effect of the present invention, a comparative experiment was conducted based on the model obtained by training in the embodiment. In order to prove that the model has learned classification ability, the candidate relationship set involved in the document in the test phase is disjoint with that in the training phase.

[0169] The comparison models include 10 existing models, including: DL-Base, DL-MNAV, DL-MNAV-SIE, DL-MNAV-SIE-SBN, ATLOP, HCRP, CHAN, KDDocRE, RAPL, and TPN.

[0170] Existing few-shot document-level benchmarks typically use two tasks: 1-Doc and 3-Doc. 1-Doc is a task where the support set consists of one support document and the query set consists of three query documents; 3-Doc is a task where the support set consists of three support documents and the query set consists of one query document. Training data can be sampled using two different strategies: single-shot and hard-shot. Single-shot is a simple sampling strategy where the training task consists of one actual relation class and a NOTA class. Hard-shot is a difficult sampling strategy where the training task consists of multiple actual relation classes and a NOTA class.

[0171] In the experiment, each comparison model was set up with 1-Doc and 3-Doc tasks, but the sampling strategies for the training set data varied. Each model used a sampling strategy that was more suitable for its own model. The sampling strategy for the training set of the first eight models was single, due to unsatisfactory test results under the hard sampling strategy. In addition to the first eight models, the sampling strategy of the RAPL model was: using the data of the single sampling strategy as the training set and the data of the single-hard sampling strategy as the development set. The sampling strategy of the TPN model was single. The model of the present invention was experimented on both single and hard tasks, denoted as Ours-single and Ours-hard, respectively.

[0172] The experimental results are shown in Table 1. FREDo (Few-Shot Document-Level Relation Extraction) is a few-shot document-level relation extraction benchmark built on the DocRE and SciERC datasets; ReFREDo is a revised version of FREDo designed to address the annotation errors present in FREDo. The values ​​are the Macro-F1 scores obtained for each model test. Macro-F1 score: When dealing with multi-classification problems, the Macro-F1 score calculates the F1 score for each category and then averages these F1 scores. The F1 score is the harmonic mean of precision and recall.

[0173] Experimental results show that compared to methods adapted from sentence-level models such as ATLOP and HCRP, our model achieves significantly higher Macro-F1 scores on document-level tasks. This suggests that directly migrating sentence-level models to document-level tasks presents certain difficulties and may fail to fully utilize document-level contextual information. Our model, however, incorporates a unique instance encoding method during design that fully considers document-level information, resulting in superior semantic understanding and information integration.

[0174] In document-level models, DL-Base, DL-MNAV, CHAN, and RAPL all employ instance-mean-based approaches to represent prototypes. CHAN and RAPL further improve on this by integrating relational information with attention mechanisms. However, experiments demonstrate that our method still offers certain advantages. This is particularly true for hard tasks involving multiple relational categories, enabling more discriminative relation embedding and instance representation.

[0175] Regarding the method of using instance groups to construct prototype representations, the DL-MNAV-SIE, DL-MNAV-SIE-SBN, and TPN models involved in the experiment all adopted this strategy. As the basic models, DL-MNAV-SIE and DL-MNAV-SIE-SBN have already demonstrated certain advantages in constructing instance group representations; TPN further strengthens the instance encoding process, achieving even better performance. However, the method of the present invention improves both the enhanced relationship description and prototype construction, achieving even better results.

[0176] Table 1. Test results

[0177]

[0178] In order to visualize the enhanced effect of the vector group prototype proposed in this paper on the prototype network, the latent feature space learned during training is visualized on a two-dimensional plane using the t-SNE method, mainly based on the data of the first 100 batches of the 3-Doc task in the ReFREDo dataset.

[0179] in, Figure 2 The training set data is visualized in the training set visualization, which divides the data into four relation types: NOTA, P131, P527, and P800. As can be seen from the figure, the NOTA category occupies the vast majority of the space, while samples of other relation types are mixed together, with blurred boundaries and difficulty in distinguishing them. This indicates that in the initial stage, the model's ability to distinguish different relation types is not yet fully utilized, and there is a significant overlap between valid relation types.

[0180] Figure 3 After 25 epochs of training, the visualization results on the test set show that the data is divided into four relation types: NOTA, P17, P361, and P463. A clear boundary has been established between the NOTA category and other valid relation types, and each valid relation type is also tightly clustered, forming a clear distinction overall. This demonstrates that after sufficient training, the model is able to better capture relational semantics and significantly improve its ability to discern subtle differences between different categories.

[0181] In order to further verify the advantages of the model in prototyping space construction, the effects of two comparative models were compared. Figure 4 The prototype space distribution results of the MNAV model are shown. The model uses a learnable vector group to represent NOTA categories, resulting in a very wide and scattered distribution of NOTA categories in the two-dimensional space. Although other relationship types are mainly concentrated in the middle area, the boundaries are still not clear and there is a lot of overlap. Figure 5The prototype spatial distribution results of the MNAV-SIE-SBN model are presented. The results show that NOTA categories and valid relationship types are completely mixed together, with no clear distinction in spatial distribution, and no effective clustering can be formed.

[0182] Although the present invention is described herein with reference to the embodiments of the present invention, the above embodiments are merely preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. Entity pair representation method, characterized in that: Construct an entity pair representation of the input entity pair as follows: For each entity in the entity pair, its attention vector is constructed using the self-attention vectors of each entity mention corresponding to it in the document to which it belongs; Build the entity embedding using the embedding of each entity mention in the document to which it belongs; The self-attention vectors and embeddings of entity mentions in the document are calculated using a pre-trained language model. The attention vectors of the two entities contained in the entity pair are fused to obtain an entity pair attention vector of dimension L; the entity pair attention vector of dimension L is used to weight the L dimension of the context embedding matrix of the document to obtain the context representation of the entity pair; where L is the number of tokens in the document; The context embedding matrix is ​​composed of the embedding concatenation of L tokens of the document; For each entity contained in the entity pair, use the fusion network to fuse its entity embedding with the context representation of the entity pair to which it belongs to obtain its entity representation; The entity representations of the two entities contained in the entity pair are fused to obtain an entity pair representation of the entity pair.

2. The entity pair representation method according to claim 1, wherein: According to the relationship between entity pairs and documents, entity pairs are grouped by document, including: B1. Input the document corresponding to the group into the pre-trained language model to obtain the embedding of L tokens of the document with a dimension of H. The self-attention layer of the pre-trained language model obtains the self-attention vector of L tokens of the document. Using the embedding of L tokens of the document, a context embedding matrix of dimension L×H is constructed for the document. B2. For each entity pair in the group, use the self-attention vector and embedding obtained in step B1 to construct its entity pair representation; In step B1, the documents corresponding to the group are input into the pre-trained language model, including: B11. Segment the input document to obtain the initial word sequence; B12. In the initial word sequence, insert the entity tag of each entity at the entity mention corresponding to the first occurrence of the entity, and the entity tag is inserted at a position adjacent to the starting position of the corresponding entity mention; In step B2, for each entity contained in the entity pair, the self-attention vectors of its various entity mentions in the corresponding document are used for average pooling to construct its attention vector, where the entity mentions adjacent to the entity tag use the self-attention vector of the entity tag as its self-attention vector; the embeddings of its various entity mentions in the corresponding document are used for average pooling to construct its entity embedding, where the entity mentions adjacent to the entity tag use the entity embedding of the entity tag as its entity embedding.

3. The entity pair representation method according to claim 1, wherein: The fusion network is a linear network. The two entities contained in the entity pair correspond to independent fusion networks respectively, and are fused according to the following formula: in, represents the entity representation of the head entity in the entity pair, represents the entity representation of the tail entity in the entity pair, h represents the entity embedding of the head entity in the entity pair, t represents the entity embedding of the tail entity in the entity pair, and C (s,o) is the context representation, tanh represents the tanh activation function, [·] represents concatenation, and W h and W t is the learnable weight, b h and b t is the learnable bias.

4. The entity pair representation method according to any one of claims 1 to 3, characterized in that: According to the following formula, the attention vectors of the two entities contained in the entity pair are fused to obtain the entity pair attention vector with a dimension of L: in, and Represents the attention vectors of the two entities contained in the entity pair, A (s,o) Represents the entity pair attention vector, T represents matrix transpose, and ⊙ represents element-by-element multiplication; According to the following formula, the entity pair attention vector with dimension L of the entity pair is used to weight the L dimension of the context embedding matrix of the document to obtain the context representation of the entity pair: C (s,o) =A (s,o) ·H T Among them, H is the context embedding matrix, C (s,o) is the context representation, T represents the matrix transpose; According to the following formula, the entity representations of the two entities contained in the entity pair are fused to obtain the entity pair representation of the entity pair: Among them, s (h,r,t) Represents entity pair representation, represents the entity representation of the head entity in the entity pair, It represents the entity representation of the tail entity in an entity pair, and [·] indicates concatenation.

5. A few-sample document-level relation extraction method, characterized by: According to the entity pair representation method according to any one of claims 1 to 4, constructing its entity pair feature extraction module; The training of its relationship extraction model includes: A1. Input samples for this round of training; the samples include support set, query set and candidate relation set; A2. For each entity pair included in the supporting entity pair set of the input sample, using the entity pair feature extraction module to obtain its entity pair representation; the supporting entity pair set includes related entity pairs, and each related entity pair included in the supporting entity pair set is composed of the head entity and the tail entity of each triple included in the triple set of the sample's support set; A3. For each relationship type included in the candidate relationship set of the input sample, construct a relationship prototype in the form of a vector group using the entity pair representations of each triple contained in its prototype set; the triples contained in the prototype set of the relationship type are obtained by sampling a set number of triples from the support set of the sample according to the relationship type; A4. Based on the entity pair representations of each entity pair contained in the query entity pair set and the relationship prototypes of each relationship type in the candidate relationship set, calculate the matching score between each entity pair and each relationship type combination according to the following formula: logit=sim(q n ,P r )=max 1≤k≤K [sim(q n ,s k )] Among them, sim(q n ,s k ) represents the entity pair representation q of the nth entity pair in the query entity pair set n and the constituent relationship prototype P r The similarity between the kth vector in the K vectors; max 1≤k≤K [sim(q n ,s k )] represents the entity pair representation q n The maximum value of the similarity between K vectors; logit represents the matching score; each entity pair contained in the query entity pair set is composed of two combinations of each entity contained in the sample query set, and its entity pair representation is calculated using the entity pair feature extraction module; A5. Based on the matching score, construct the prediction loss for this round of training and reversely update the entity pair feature extraction module; A6. Determine whether the training is completed. If so, end the training; otherwise, return to step A1.

6. The method for extracting document-level relations at a small number of samples according to claim 5, wherein: Step A2 further includes: for each relationship type included in the candidate relationship set of the input sample, first, extracting its name and description from the relationship dictionary, and concatenating them to form its relationship description text; then, inputting the relationship description text into the pre-trained language model to obtain its relationship representation; and then, constructing its relationship description prototype based on its relationship representation; In step A3, for each relation type contained in the candidate relation set of the input sample, the relation prototype P in the form of a vector group is constructed using the entity pair representations of each triple contained in its prototype set and the relation description prototype of the relation type according to the following formula: r : Among them, s i is the entity pair representation of the relationship entity pair corresponding to the i-th triple in the prototype set, I represents the number of triples contained in the prototype set; P r-rel Describe the prototype for the relationship, Represents a vector concatenation operation.

7. The method for extracting document-level relations at a small sample level according to claim 6, wherein: For each relationship type contained in the candidate relationship set of the input sample, first, extract its name and description from the relationship dictionary respectively, and concatenate them to form its relationship description text as follows: r=[CLS:r name :r dec :SEP] Among them, CLS and SEP are the start and end markers respectively; name is the word sequence of the name of the relation type, r dec A sequence of words describing the relationship type; Then, the relationship description text r is input into the pre-trained language model; the embedding of the [CLS] position output by the pre-trained language model is used as the global representation of the relationship description text; the embedding of each word in the relationship description text output by the pre-trained language model is average pooled to obtain the local representation of the relationship description text; Then, according to the following formula, the global representation r of the fusion relation description text is cls and local representation r Avg , obtain the relationship representation r of this relationship type final : r final =λ·r cls +(1-λ)·r Avg Among them, λ is the fusion weight.

8. The method for extracting document-level relations at a small number of samples according to claim 6, wherein: Based on the relationship representation of the relationship type, construct its relationship description prototype, including the following steps: First, for each triple contained in the prototype set of the input relation type, the entity pair representation s of the corresponding relation entity pair is calculated. i , and the relationship representation r of this relationship type final The semantic alignment between sim(s i ,r final ); Then, according to the following formula, the semantic alignment of each triple contained in the prototype set of the relationship type is used to represent the relationship type r final Make corrections to obtain the relationship description prototype P of this relationship type r-rel : P r-rel ＝α·r final Where K is the number of triplets contained in the prototype set; sim(·) represents the similarity measurement function.

9. The method for extracting document-level relationships at a small sample level according to any one of claims 6 to 8, wherein: The supporting entity pair set of the sample includes relational entity pairs and unrelational entity pairs; the unrelational entity pairs are entity pairs consisting of entities included in the triple set of the sample and do not belong to relational entity pairs, and the two entities included in the unrelational entity pairs belong to the same document; According to the following formula, sample the set number of unrelated entity pairs, use the entity pair representations of each unrelated entity pair obtained by sampling, and construct a relationship prototype P with a relationship type of unrelated NOTA : Among them, s j is the entity pair representation of the jth unrelated entity pair obtained by sampling, and J represents the number of samples of unrelated entity pairs; Represents vector concatenation operation; In step A4, it also includes: based on the entity pair representations of each entity pair included in the query entity pair set and the relationship prototype of no relationship, calculating the matching score between each entity pair included in the query entity pair set and the relationship type of no relationship.

10. The method for extracting document-level relations at a small number of samples according to claim 9, wherein: In step A5, first, the prediction loss L is calculated based on the matching score between each entity pair and relationship type combination according to the following formula: ATLoss Then, the prediction loss of each entity pair and each relationship type is summed up to obtain the prediction loss of this round of training: L ATLoss =L T-NOTA +L NOTA-N Among them, P r Represents the set of relationship types to which the entity pair belongs in the triple set of the sample; N T Indicates that the candidate relation set is excluding P r A collection of other relationship types other than the included relationship types; NOTA represents a relationship type with no relationship; {P r ∪NOTA} represents P r The set of contained relationship types and NOTA relationship types; {N T ∪NOTA} represents N T The set of included relationship types and NOTA relationship types; logit n Represents entity pairs with P r The matching score of the relation prototype of the nth relation type in ; Represents entity pairs with {P r The matching score of the relation prototype of the m1-th relation type in ∪NOTA}; Represents entity pairs with {N T The matching score of the relation prototype of the m2-th relation type in ∪NOTA}.

Images