Document-level relation extraction method and system based on reasoning path and related equipment

By constructing document graphs and generating inference paths, combining self-attention mechanism and loss function optimization, the accuracy of long-distance dependency modeling in document-level relationship extraction is solved, and the relationship extraction performance and robustness of the model are improved.

CN119047561APending Publication Date: 2024-11-29GUANGXI POWER GRID CORP

Patent Information

Application Number
CN202411027794.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-29

Smart Images

  • Figure CN119047561A_ABST
    Figure CN119047561A_ABST
Patent Text Reader

Abstract

The invention is suitable for the technical field of document relationship extraction, and provides a document-level relationship extraction method and system based on a reasoning path and related equipment, and the method comprises the following steps: marking entities in an obtained document text, inputting the marked entities into an encoder for vectorization, and obtaining an initial word vector; according to the initial word vector, constructing a document graph comprising a syntactic dependency edge, a co-referring mention edge, a pronoun mention edge and a cross-sentence connecting edge; generating a target reasoning path of any two entities according to a syntactic dependency edge, a co-reference mention edge, a pronoun mention edge or a cross-sentence connecting edge in the document graph; performing path perception enhancement on the target reasoning path by using an improved self-attention mechanism to obtain a target variable; the target variable is used as the input of the model for training, and the document-level relationship type of any two entities is generated based on the model optimized by the self-adaptive threshold loss function and the self-adaptive regular loss function, so that the accuracy of the model in the relationship reasoning process is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of document relationship extraction, and in particular to a document level relationship extraction method, system and related equipment based on reasoning path. Background Art

[0002] The core of the document-level relationship extraction task is to solve the problem of relationship extraction between entities across sentences. The key lies in how to effectively capture the key information between long-distance entities. At present, some research works construct document graph structures and use graph neural networks to learn the representation of nodes to fully capture the relationship and context information between nodes, and classify them based on the learned node representations. Some other research works use pre-trained language models to implicitly model the interactive relationship between entity pairs to improve the accuracy of relationship extraction. However, due to the complexity of the context in the document, the above methods are difficult to effectively model the dependency relationship between long-distance entities.

[0003] In the prior art, for the dependency relationship between long-distance entities, some researchers have proposed different strategies from the perspective of document pruning to extract key evidence sentences that support the facts of entity-to-entity relationships, and infer the relationship between entities based on the evidence sentences. However, in the process of extracting evidence sentences, the above methods ignore the problem that there may be irrelevant fragments or clauses in the sentences, causing the model's attention to be distracted to irrelevant information, which in turn affects the accuracy of the model in the process of relational reasoning. Summary of the invention

[0004] The embodiments of the present application provide a document-level relationship extraction method, system and related equipment based on reasoning paths, which are used to solve the problem that affects the accuracy of the model in the relationship reasoning process.

[0005] The first aspect of the embodiment of the present application provides a document-level relationship extraction method based on an inference path, comprising:

[0006] Label the entities in the acquired document text, input the labeled entities into the encoder for vectorization, and obtain the initial word vector;

[0007] Constructing a word-level document graph based on the initial word vector, wherein the document graph includes syntactic dependency edges, coreference mention edges, pronoun mention edges, and cross-sentence edges;

[0008] Generate a target reasoning path between any two entities according to a syntactic dependency edge, a coreference mention edge, a pronoun mention edge, or a cross-sentence connection edge in the document graph, wherein the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path;

[0009] Using the improved self-attention mechanism to perform path perception enhancement on the target reasoning path to obtain the target variable;

[0010] The target variable is used as an input of the model for training, and the document-level relationship type of the arbitrary two entities is generated based on the model optimized by the adaptive threshold loss function and the adaptive regularization loss function.

[0011] Furthermore, the target reasoning path of any two entities is generated according to the syntactic dependency edge, the coreference mention edge, the pronoun mention edge or the cross-sentence connection edge in the document graph, wherein the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path, including:

[0012] Determine whether there is a syntactic dependency edge between any two entities in the document graph, and if so, generate an intra-sentence reasoning path for the two entities;

[0013] Determine whether any two entities have a co-reference mention edge, a pronoun mention edge, or a cross-sentence edge in the document graph. If so, generate an inter-sentence reasoning path between the two entities.

[0014] Furthermore, the generating a target reasoning path of any two entities according to the syntactic dependency edge, the coreference mention edge, the pronoun mention edge or the cross-sentence connection edge in the document graph includes:

[0015] The target reasoning path is determined according to the preset reasoning path priority, wherein the reasoning path priority includes:

[0016] When there are both intra-sentence reasoning paths and inter-sentence reasoning paths, the intra-sentence reasoning path is preferred as the target reasoning path;

[0017] When only inter-sentence reasoning paths exist, the inter-sentence reasoning path with the least bridge nodes is preferentially selected as the target reasoning path.

[0018] Furthermore, the improved self-attention mechanism is used to enhance the path perception of the target reasoning path to obtain the target variable, including:

[0019] The improved self-attention mechanism is provided with a mask strategy of intra-sentence reasoning path perception and an inter-sentence reasoning path perception. The expression of the self-attention mechanism is as follows:

[0020]

[0021] Where: Concat(·) represents the concatenation operation, h represents the number of heads of the multi-head attention mechanism, represents the hidden layer representation of the i-th head in the attention mechanism, is a trainable learning parameter, T represents the transposition operation, and M e represents the mask matrix, e∈{local, path} represents different mask types.

[0022] Furthermore, the intra-sentence reasoning path-aware masking strategy is provided with an intra-sentence path-aware masking matrix M local , the expression is as follows:

[0023]

[0024] Where: P i Get node t for the dependency syntax tree structure i The set of neighbor nodes;

[0025] The inter-sentence reasoning path-aware mask strategy is provided with an inter-sentence path-aware mask matrix M path , the expression is as follows:

[0026]

[0027] Where: Q i is the number of nodes mentioned from a specific node t i A collection of all other paths mentioned in the document.

[0028] Furthermore, the use of the improved self-attention mechanism to enhance the path perception of the target reasoning path includes:

[0029] The formula for the feature fusion process of the target reasoning path using the improved self-attention mechanism is as follows:

[0030] I l,p =[H l ;H p ;H l -H p ;H l ⊙H p ];

[0031] G l,p =σ(WI l,p + b);

[0032] H′=G l,p ⊙H l +(1-G l,p )⊙H p ;

[0033] Where: H l and H p are the representations of intra-sentence reasoning path perception nodes and inter-sentence reasoning path perception nodes, respectively. l,p H l and H p The difference and consistency between the two node representations, G l,p is the weight distribution, W is the learnable network parameter, and σ(·) is the activation function, which maps the output of the linear layer to a probability value from 0 to 1.

[0034] Furthermore, the expression of the adaptive threshold loss function is as follows:

[0035]

[0036] Where: For an entity pair (e i , e j ), For an entity pair (e i , e j ), and TH is the negative class set of the relations that do not exist in .

[0037] Furthermore, the expression of the adaptive regularized loss function is as follows:

[0038]

[0039] Where: δ is a small positive number, L i,j,r is the model's predicted score for relation r, Q i,j,r is the soft label score of the model for relation r.

[0040] A second aspect of an embodiment of the present application provides a document-level relationship extraction system based on an inference path, comprising:

[0041] An initial word vector determination unit is used to annotate entities in the acquired document text, and input the annotated entities into an encoder for vectorization to obtain an initial word vector;

[0042] A document graph construction unit, configured to construct a word-level document graph according to the initial word vector, wherein the document graph includes syntactic dependency edges, coreference mention edges, pronoun mention edges, and cross-sentence connection edges;

[0043] a target reasoning path generating unit, configured to generate a target reasoning path between any two entities according to a syntactic dependency edge, a coreference mention edge, a pronoun mention edge, or a cross-sentence connection edge in the document graph, wherein the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path;

[0044] A target variable determination unit, used for performing path perception enhancement on the target reasoning path by using an improved self-attention mechanism to obtain a target variable;

[0045] The document-level relationship type generating unit is used to train the target variable as the input of the model, and generate the document-level relationship type of the arbitrary two entities based on the model optimized by the adaptive threshold loss function and the adaptive regularization loss function.

[0046] A third aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the steps of the document-level relationship extraction method based on the reasoning path as described in any one of the above are implemented.

[0047] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0048] This application first annotates the entities of the acquired document text, and then inputs the annotated entities into the encoder for vectorization to obtain word vector representation. Based on the word vector, a document graph containing four types of words is constructed at the word level, and then the shortest dependency path between entities is explicitly modeled to capture the semantic dependency information supporting the entity relationship; the improved self-attention mechanism is then used to enhance the path perception of the reasoning path to obtain the target variable, and finally the model is trained. At the same time, the model is optimized based on the adaptive threshold loss function and the adaptive regularized loss function to generate document-level relationship types. This application combines the attention mechanism based on intra-sentence and inter-sentence path perception to constrain the information attention scope of the model in the process of relational reasoning, thereby enhancing the model's learning ability for the semantic support of entity relationships, and thus improving the model's relational extraction performance. In addition, in response to the problem of missing data labels, an adaptive regularized loss function is introduced to constrain the output distribution of the model, thereby improving the robustness of the model and improving the accuracy of the model in the process of relational reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of an embodiment of the document-level relationship extraction method based on reasoning path in the present invention;

[0050] Figure 2 It is a structural diagram of the document-level relationship extraction model in the present invention;

[0051] Figure 3 It is a schematic diagram of the document graph structure in the present invention;

[0052] Figure 4 It is a structural block diagram of an embodiment of a document-level relationship extraction system based on reasoning paths in the present invention;

[0053] Figure 5 The figure is a structural block diagram of a computer device in the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] In this embodiment, the document-level relationship extraction method based on path perception is used to improve the accuracy of the model in the relationship reasoning process. In this embodiment, the implementation method can be implemented in the system, can be implemented in the server, and can also be implemented in the terminal, without specific limitation.

[0056] Embodiment 1

[0057] See also Figure 1 , a document-level relationship extraction method based on path perception in the present invention comprises the following steps:

[0058] S11. Annotate the entities in the acquired document text, and input the annotated entities into the encoder for vectorization to obtain the initial word vector;

[0059] In this embodiment, a document-level relation extraction model (PARN) based on path-aware enhancement is constructed. Figure 2 As shown in the figure, PARN constructs a document graph at the word level, and introduces four edge connection methods on this basis, aiming to more comprehensively reflect the relationship between nodes in the document. Two key modules are proposed to enhance the relational reasoning ability of the model: first, the shortest reasoning path between any two entity mentions is constructed through a heuristic strategy to ensure that the model can capture the semantic dependency information supporting the entity relationship in the document; second, the intra-sentence and inter-sentence path-aware attention mechanism is introduced to constrain the information attention scope of the model in the relational reasoning process, thereby guiding the model to achieve more accurate information positioning.

[0060] First, the document encoding module aims to encode the input document sequence to obtain the word embedding representation of each word in the document and the embedding representation of entity mentions. Specifically, for a given document D = {t1, t2, ..., t N}, first encode it to obtain the representation of the entity mention. In order to optimize the representation learning of entity mention, a special marker "*" is inserted at the beginning and end of the entity mention. Subsequently, the preprocessed document is input into the pre-trained language model BERT to obtain the corresponding context representation H. The specific calculation process is shown in the following formula 1:

[0061] H={h1,h2,...,h N′}=BERT([t1,t2,...,t N ]); (1)

[0062] Where N represents the number of words in the original document; N′ represents the length of the sequence after segmentation by the BERT segmenter; represents the corresponding context representation; d h Represents the dimension of the hidden layer.

[0063] Given that the BERT tokenizer may i Therefore, in the above document encoding process, the hidden layer representation of the first subword of the word is selected as the overall word embedding representation. For each mention of an entity in the document, the representation vector corresponding to the "*" mark inserted at its starting position is used as the initial representation of the entity mention.

[0064] S12. Construct a word-level document graph based on the initial word vector, where the document graph includes syntactic dependency edges, coreference mention edges, pronoun mention edges, and cross-sentence edges;

[0065] In this embodiment, the word-level document graph is constructed to construct the dependency relationship between any two nodes in the document. In the document graph, each word in the document is used as a node, and four types of edges are defined, namely, syntactic dependency edge, coreference mention edge, pronoun mention edge, and cross-sentence edge, such as Figure 3 As shown, each node in the graph corresponds to a word in the document.

[0066] A syntactic dependency edge is an edge used to represent the grammatical relationship between words in syntactic analysis, which indicates the dependency relationship between vocabulary units. Specifically, the spacy toolkit is first used to parse each sentence in the document to obtain the dependency tree structure of each sentence. Then, based on the obtained dependency tree structure, an edge is constructed between each word node and its adjacent node on the dependency tree structure.

[0067] A co-referenced mention edge is an edge that connects any two mention nodes belonging to the same entity in the document-level relationship extraction task. Although these mention nodes have different expressions, they semantically point to the same entity. By constructing co-referenced mention edges, the model can better understand and utilize the different mentions of entities in the document, thereby more accurately capturing the relationship between entities. Figure 3 As shown, the edge between the two "MaryLand" represents a co-referential mention relationship, indicating that they belong to the same entity.

[0068] Pronoun mention edges are used to connect demonstrative pronouns in a document with the entity mentions they point to. Demonstrative pronouns, such as "He", "She", "It", etc., play a referential role in a document, pointing to a previously mentioned entity. However, the referential relationship of pronouns is usually not explicitly annotated in the dataset. By using a referential annotation tool such as AllenNLP, construct edges between pronouns and the mentions they point to. Figure 3 As shown, the edge between "He" and "EliasBrown" represents a pronoun mention relationship, indicating that "He" refers to "EliasBrown".

[0069] Cross-sentence edges are used to establish the dependency relationship between any two sentences in a document. When the above three edges cannot establish a reachable path between two mentions, the root nodes of the dependency tree structure of any two sentences are connected to ensure that the dependency relationship between any two sentences is established. For example, in Figure 3 In the figure, the root nodes of the dependency tree structure corresponding to the first sentence and the second sentence are "was" and "served" respectively. The edge between them represents the cross-sentence edge, which establishes the dependency relationship between the two sentences.

[0070] S13. Generate a target reasoning path between any two entities according to the syntactic dependency edge, coreference mention edge, pronoun mention edge or cross-sentence connection edge in the document graph, where the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path;

[0071] Specifically, the intra-sentence reasoning path refers to the shortest dependency path connecting two entity mentions in the same sentence, which can effectively retain the key part of the sentence that contains the semantic information of the relational facts, while ignoring irrelevant semantic structure information. In the sentence-level relation extraction scenario, the shortest dependency path has been proven to be able to retain the key semantic information for inferring relational facts to the greatest extent. Therefore, for two entity mentions in the same sentence, their reasoning path can be represented by the shortest dependency path, where the shortest dependency path is connected by syntactic dependency edges. For example, in Figure 3 In the first sentence of , the intra-sentence reasoning path between “Elias” and “MaryLand” is “Elias→Brown→was→Representative→from→Maryland”.

[0072] The inter-sentence reasoning path refers to the dependency path connecting cross-sentence entity mentions in a document, which helps the model capture the semantic relationship and relevance across sentences. For two entity mentions in different sentences, since the syntactic dependency edge cannot directly establish the connection between cross-sentence entities, it is necessary to use a bridge node to build the relationship dependency between them. The selection of the bridge node depends on the type of relationship instance. If the relationship instance involves logical reasoning, the bridge node is the middle entity node; if the relationship instance involves reference reasoning, the bridge node is the pronoun of the entity reference. Therefore, the semantic dependency association between cross-sentence entity mentions is established through pronoun nodes or entity nodes as bridge nodes.

[0073] In this embodiment, generating a target reasoning path between any two entities includes the following steps:

[0074] 1. Determine whether there is a syntactic dependency edge between any two entities in the document graph. If so, generate the intra-sentence reasoning path of the two entities;

[0075] 2. Determine whether there are co-reference mention edges, pronoun mention edges, or cross-sentence edges between any two entities in the document graph. If so, generate an inter-sentence reasoning path between the two entities.

[0076] The designed coreference mention edges and pronoun mention edges are used to establish cross-sentence dependency associations between mentions. Then, between nodes in the same sentence, syntactic dependency edges are used to construct intra-sentence dependency relationships, thereby generating corresponding paths, namely, intra-sentence reasoning paths. Figure 3 As shown in the figure, "He" serves as a bridge node between "Elias" and "Monroe", connecting the two mentions in different sentences, and the final generated path is "Elias→He→served→on→ticket→of→Monroe". In addition, if the two sentences cannot establish a cross-sentence dependency relationship through coreference mention edges and pronoun mention edges, cross-sentence edges are used to establish a connection.

[0077] In addition, this embodiment also includes the following steps:

[0078] The target reasoning path is determined according to the priority of the preset reasoning path. The priority of the reasoning path includes:

[0079] When there are both intra-sentence reasoning paths and inter-sentence reasoning paths, the intra-sentence reasoning path is preferred as the target reasoning path;

[0080] When only inter-sentence reasoning paths exist, the inter-sentence reasoning path with the least bridge nodes is preferentially selected as the target reasoning path.

[0081] Considering that there may be multiple reachable paths between two entity mentions, the priority of the reasoning path is predefined. For a given mention pair (m k , m t ), first check whether there is an intra-sentence reasoning path; if not, select the inter-sentence reasoning path with the least bridge nodes. Based on the heuristic rules generated by the above defined paths, the distance between any two entity mentions (m k , m t ) can find a shortest dependency path p k,t , as shown in the following formula 2:

[0082] p k,t = {u1, u2, ..., u r ); (2)

[0083] Among them, u i Indicates the corresponding intermediate node on the reasoning path.

[0084] S14. Use the improved self-attention mechanism to enhance the path perception of the target reasoning path and obtain the target variable;

[0085] According to the description of formula 1, the initial node representation H takes all words in the document into equal consideration during the self-attention calculation process. However, for entity mentions, not all document information is equally important. Therefore, in response to the above problems, two self-attention mechanisms with mask strategies are proposed to more accurately guide the model to focus on local and global information related to entity mentions. Different from the traditional self-attention mechanism, this attention mechanism explicitly limits the attention range of each node. The expression of the self-attention mechanism is shown in the following formulas 3 and 4:

[0086]

[0087] Where: Concat(·) represents the concatenation operation, h represents the number of heads of the multi-head attention mechanism, represents the hidden layer representation of the i-th head in the attention mechanism, is a trainable learning parameter, T represents the transposition operation, and M e represents the mask matrix, e∈{local, path) represents different mask types.

[0088] The intra-sentence reasoning path-aware mask strategy is set with the intra-sentence path-aware mask matrix M local , the inter-sentence reasoning path-aware mask strategy is set with the inter-sentence path-aware mask matrix M path .

[0089] Specifically, the intra-sentence path-aware mask matrix aims to guide the model to focus on the local information around the entity mention. In the relation extraction task, the neighbor node information connected to the entity node in the dependency grammar tree can directly reflect the syntactic relationship and semantic connection between the entity and the surrounding words. In the specific implementation, firstly, the node t is obtained according to the dependency grammar tree structure obtained by parsing. i The neighbor node set P i Then, using the neighbor node set P i , construct the intra-sentence path-aware mask matrix, the expression is as follows:

[0090]

[0091] The inter-sentence path-aware mask matrix aims to enhance the model's attention to the dependency path information between entity mentions. In the document-level relationship extraction task, in addition to considering the neighbor node information of entity mentions, the shortest path connecting two entity mentions is also of great significance because they can best characterize the semantic relationship and relevance between entities. In the specific implementation, first calculate the specific mention node t according to Formula 2. i The set Q of paths to all other mentions in the document i , where Q i =∑k=1 p k,i , k represents any one in the document except t i Then, the path set information is used to construct the inter-sentence path-aware mask matrix, which is expressed as follows:

[0092]

[0093] Through the above construction process, we get the intra-sentence path-aware mask matrix M local and the inter-sentence path-aware mask matrix M path Then, they are input into Formula 4 to obtain two complementary node representations H l and H p .

[0094] In order to effectively fuse the above two node representations, a fusion mechanism based on a gating mechanism is adopted. First, H is calculated by element concatenation, subtraction, and dot multiplication operations. l and H p The difference and consistency between the two node representations. Next, calculate H l and H p The two types of nodes represent the proportion in the fusion process, and the weight distribution of the proportion is expressed by G l,p Finally, based on the obtained weight distribution G l,p , H l and H p The two node representations are fused to obtain the path-aware enhanced node representation H′, which is expressed as follows:

[0095] I l,p =[H l ;H p ;H l -H p ;H l ⊙H p ]; (7)

[0096] G l,p =σ(WI l,p +b); (8)

[0097] H′=G l,p ⊙H l +(1-G l,p )⊙H p ; (9)

[0098] Where W represents the learnable network parameters and σ(·) represents the activation function, which maps the output of the linear layer to a probability value from 0 to 1.

[0099] S15. The target variable is used as the input of the model for training, and the document-level relationship type between any two entities is generated based on the model optimized by the adaptive threshold loss function and the adaptive regularization loss function.

[0100] In the document-level relation extraction scenario, only entity pair-level labels can be obtained. However, multiple relations of entity pairs are usually implicit in specific mention pair instances. Therefore, the idea of ​​multi-instance learning is adopted to treat all mention pair representations contained in the entity pair as a latent variable, and then all mention pair representations are fused to predict the corresponding entity pair relations. Specifically, for any entity pair (e i , e j ),in First, we calculate each mention pair (m k , m t )∈e i ×e j Specifically, for each mention pair (m k , m t ), the path representation information between two mentions is obtained through the maximum pooling function, as shown in the following formula 10. Then the mention representation and the path representation p k,t and distance embedding d k,t The concatenation is performed and fed into the tanh activation function as shown in Formula 11 and Formula 12. Then, as shown in Formula 13, the corresponding mention pair relationship representation is calculated using a bilinear function:

[0101]

[0102]

[0103] r k,t =z k W7 t +b; (13)

[0104] Where: d k,t Represents the relative distance representation between two entity mentions, u1, u2, u r It means that the value defined in formula 2 is k to m t The intermediate nodes, W1 to W7 represent the learnable network parameters, and represents the path-aware enhanced mention representation obtained in Formula 9.

[0105] Next, a maximum pooling operation is used to select the target entity pairs (e i , e j) are aggregated, and a linear layer is used to obtain the model's predicted score for the relation category r, as shown in Formulas 14 and 15 below:

[0106]

[0107] y r =W r r i,j +b r ; (15)

[0108] Among them, max{·} represents the maximum pooling operation, N i Represents entity e i The number of mentions included, N j Represents entity e j The number of mentions included, W r represents learnable parameters.

[0109] In the training phase of the model, unlike previous work, an adaptive regularized loss function is proposed to constrain the output distribution of the model during the training process, thereby improving the robustness of the model in the face of noisy samples. The loss function used in the training process is shown in the following formula 16:

[0110]

[0111] Among them, γ represents a hyperparameter, represents the classification loss function used, Represents the proposed adaptive regularized loss function.

[0112] In this embodiment, the model is optimized by using an adaptive threshold loss function and an adaptive regularization loss function, as follows:

[0113] 1. For entity pairs (e i , e j ), the relationship between them is defined as the positive class set A non-existent relationship is defined as a negative class set The threshold class is defined as TH, and the adaptive threshold loss function aims to push the output score of the positive class higher than the output score of the threshold class and make the output score of the negative class lower than the output score of the threshold class, so that the model learns a dynamic threshold. The calculation process of the adaptive threshold loss function is shown in Formula 17, Formula 18 and Formula 19:

[0114]

[0115] 2. For each entity pair (e i , e j ), let y rrepresents the output score of the model for relation r, y TH represents the output score of the model for the threshold class TH, then the difference between the two can be used to represent the model's predicted score value S for the relationship r. i,j,r , as shown in the following formula 20:

[0116] S i,j,r =y r -y TH ; (20)

[0117] In order to map the prediction score to a non-negative interval, the prediction score value S i,j,r Input into the sigmoid function to get the mapped value L i,j,r , as shown in Formula 21. Through this mapping process, we can see that when the mapping score L i,j,r When it is greater than 0.5, it means that there is a relationship r between the entity pairs, and vice versa.

[0118] L i,j,r =σ(S i,j,r ); (twenty one)

[0119] Next, the prediction results of the two models with the same structure but different initialization parameters are averaged as the soft label score value of the entity pair relationship, as shown in Formula 22 below.

[0120]

[0121] Finally, the KL divergence loss function is used to measure the difference between the model's prediction and the soft label score value, as shown in Formula 23 and Formula 24 below:

[0122]

[0123] Among them, δ represents a small positive number, L i,j,r represents the model's prediction score for relation r, Q i,j,r Represents the soft label score of the model for relation r.

[0124] In the classification process, the above adaptive threshold loss function outputs a threshold class score in addition to the scores of all relationship categories, so as to more accurately capture the model's threshold for predicting different entity relationships. The adaptive regularization loss function fully considers the impact of the threshold class score on the model during the training process. Specifically, during the model training process, the threshold class is regularized together with the predefined relationship category to obtain a more reliable threshold class score. The reason for using this loss function to train the model is that when the model encounters noisy labels during the training process, inconsistent prediction results are often generated in different training steps, which leads to higher However, by minimizing The value can prompt the model's classifier to produce prediction results similar to another model during the training process, rather than overfitting the labels of noise samples, thereby effectively reducing the impact of noise samples on model performance and improving the robustness and accuracy of the model.

[0125] The following experiments are combined to verify:

[0126] The experiments are evaluated on three benchmark document-level relation extraction datasets, namely DocRED, CDR, and GDA datasets. The detailed information of the datasets is shown in Table 1.

[0127] Table 1 Statistical information of the dataset used in the experiment

[0128]

[0129] The experimental part aims to evaluate the effectiveness of the bundled recommendation model proposed in this invention on different datasets. Specifically, we use the Ign F1 and F1 results of the model on three datasets as the evaluation indicators of the model, and the experimental comparison results are given in Tables 2 and 3.

[0130] Table 2 Experimental results of PAEN and the baseline models compared on the DocRED dataset

[0131]

[0132] Table 3 Experimental results of PAEN and the baseline models compared on CDR and GDA datasets

[0133]

[0134] From the experimental results in Tables 2 and 3, it can be seen that PARN achieved the best performance. The experimental results show that compared with the DRN model, the performance of the proposed model in the Intra F1 and Inter F1 indicators is improved by 1.19% and 3.27% respectively; and compared with the EIDER model, the performance improvement is 0.32% and 0.77% respectively. It is worth noting that the DRN and EIDER models respectively propose different algorithms to extract the evidence sentence information supporting the entity-to-relationship facts in the document. However, compared with these two models, the proposed model PAEN shows the best performance in both intra-sentence and inter-sentence relationship instances, and the improvement in InterF1 value is more significant. The experimental results fully verify the effectiveness of the method in the above embodiment, which can effectively capture the semantic dependency information at the word-level granularity in the document, reduce the interference of irrelevant fragments in the sentence, and thus improve the accuracy of the model in the relational reasoning process.

[0135] The above embodiment constructs a document graph at the word level and designs heuristic rules based on it, by explicitly modeling the shortest dependency path between entities to capture the semantic dependency information that supports entity relationships. At the same time, combined with the attention mechanism based on intra-sentence and inter-sentence path perception, the information attention scope of the model in the process of relational reasoning is constrained, thereby enhancing the model's learning ability for the semantic support of entity relationships, and thus improving the model's relation extraction performance. In addition, in response to the problem of missing data labels, an adaptive regularized loss function is introduced to constrain the output distribution of the model, thereby improving the robustness of the model.

[0136] Embodiment 2

[0137] See also Figure 4 , an embodiment of the document-level relationship extraction system based on reasoning path in the present invention comprises the following steps:

[0138] The initial word vector determination unit 101 is used to mark the entities in the acquired document text, and input the marked entities into the encoder for vectorization to obtain the initial word vector;

[0139] A document graph construction unit 102 is used to construct a word-level document graph based on the initial word vector, wherein the document graph includes syntactic dependency edges, coreference mention edges, pronoun mention edges, and cross-sentence edges;

[0140] A target reasoning path generating unit 103 is used to generate a target reasoning path between any two entities according to the syntactic dependency edge, the coreference mention edge, the pronoun mention edge or the cross-sentence connection edge in the document graph, wherein the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path;

[0141] A target variable determination unit 104 is used to perform path perception enhancement on the target reasoning path using the improved self-attention mechanism to obtain a target variable;

[0142] The document-level relationship type generating unit 105 is used to train the target variable as the input of the model, and generate the document-level relationship type of any two entities based on the model optimized by the adaptive threshold loss function and the adaptive regularization loss function.

[0143] For the specific definition of the system, please refer to the definition of the method above, which will not be repeated here. Each module in the above system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0144] Embodiment 3

[0145] See also Figure 5The present invention provides a computer device, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the above method are implemented.

[0146] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored. In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0149] It can be understood that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A document-level relationship extraction method based on reasoning path, It is characterized in that include: Label the entities in the acquired document text, input the labeled entities into the encoder for vectorization, and obtain the initial word vector; Constructing a word-level document graph based on the initial word vector, wherein the document graph includes syntactic dependency edges, coreference mention edges, pronoun mention edges, and cross-sentence edges; Generate a target reasoning path between any two entities according to a syntactic dependency edge, a coreference mention edge, a pronoun mention edge, or a cross-sentence connection edge in the document graph, wherein the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path; Using the improved self-attention mechanism to perform path perception enhancement on the target reasoning path to obtain the target variable; The target variable is used as an input of the model for training, and the document-level relationship type of the arbitrary two entities is generated based on the model optimized by the adaptive threshold loss function and the adaptive regularization loss function.

2. According to the document-level relationship extraction method based on reasoning path according to claim 1, It is characterized in that The generating a target reasoning path of any two entities according to the syntactic dependency edge, the coreference mention edge, the pronoun mention edge or the cross-sentence connection edge in the document graph, wherein the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path, comprises: Determine whether there is a syntactic dependency edge between any two entities in the document graph, and if so, generate an intra-sentence reasoning path for the two entities; Determine whether any two entities have a co-reference mention edge, a pronoun mention edge, or a cross-sentence edge in the document graph. If so, generate an inter-sentence reasoning path between the two entities.

3. According to the document-level relationship extraction method based on reasoning path according to claim 2, It is characterized in that The generating a target reasoning path of any two entities according to the syntactic dependency edge, the coreference mention edge, the pronoun mention edge or the cross-sentence connection edge in the document graph includes: The target reasoning path is determined according to the preset reasoning path priority, wherein the reasoning path priority includes: When there are both intra-sentence reasoning paths and inter-sentence reasoning paths, the intra-sentence reasoning path is preferred as the target reasoning path; When only inter-sentence reasoning paths exist, the inter-sentence reasoning path with the least bridge nodes is preferentially selected as the target reasoning path.

4. According to the document-level relationship extraction method based on reasoning path according to claim 1, It is characterized in that The improved self-attention mechanism is used to enhance the path perception of the target reasoning path to obtain the target variable, including: The improved self-attention mechanism is provided with a mask strategy of intra-sentence reasoning path perception and an inter-sentence reasoning path perception. The expression of the self-attention mechanism is as follows: Where: Concat(·) represents the concatenation operation, h represents the number of heads of the multi-head attention mechanism, represents the hidden layer representation of the i-th head in the attention mechanism, is a trainable learning parameter, T represents the transposition operation, and M e represents the mask matrix, e∈{local,path} represents different mask types.

5. According to the document-level relationship extraction method based on reasoning path according to claim 4, It is characterized in that The intra-sentence reasoning path-aware masking strategy is provided with an intra-sentence path-aware masking matrix M local , the expression is as follows: Where: P i Get node t for the dependency syntax tree structure i The set of neighbor nodes; The inter-sentence reasoning path-aware mask strategy is provided with an inter-sentence path-aware mask matrix M path , the expression is as follows: Where: Q i is the number of nodes mentioned from a specific node t i A collection of all other paths mentioned in the document.

6. According to the document-level relationship extraction method based on reasoning path according to claim 4, It is characterized in that The step of using the improved self-attention mechanism to enhance the path perception of the target reasoning path includes: The formula for the feature fusion process of the target reasoning path using the improved self-attention mechanism is as follows: I l,p =[H l ;H p ;H l ―H p ;H l ⊙H p ]; G l,p =σ(WI l,p +b); H′=G l,p ⊙H l +)1―G l,p )⊙H p ; Where: H l and H p are the representations of the intra-sentence reasoning path perception nodes and the inter-sentence reasoning path perception nodes, respectively. l,p H l and H p The difference and consistency between the two node representations, G l,p is the weight distribution, W is the learnable network parameter, and σ(·) is the activation function, which maps the output of the linear layer to a probability value from 0 to 1.

7. The document-level relationship extraction method based on reasoning path according to claim 1, It is characterized in that The expression of the adaptive threshold loss function is as follows: Where: For an entity pair (e i ,e j ), For an entity pair (e i ,e j ), and TH is the threshold class.

8. The document-level relationship extraction method based on reasoning path according to claim 1, It is characterized in that The expression of the adaptive regularized loss function is as follows: Where: δ is a small positive number, L i,j,r is the model's predicted score for relation r, Q i,j,r is the soft label score of the model for relation r.

9. A document-level relation extraction system based on reasoning paths, It is characterized in that include: An initial word vector determination unit is used to annotate entities in the acquired document text, and input the annotated entities into an encoder for vectorization to obtain an initial word vector; A document graph construction unit, configured to construct a word-level document graph according to the initial word vector, wherein the document graph includes syntactic dependency edges, coreference mention edges, pronoun mention edges, and cross-sentence connection edges; a target reasoning path generating unit, configured to generate a target reasoning path between any two entities according to a syntactic dependency edge, a coreference mention edge, a pronoun mention edge, or a cross-sentence connection edge in the document graph, wherein the target reasoning path includes an intra-sentence reasoning path or an inter-sentence reasoning path; A target variable determination unit, used for performing path perception enhancement on the target reasoning path by using an improved self-attention mechanism to obtain a target variable; The document-level relationship type generating unit is used to train the target variable as the input of the model, and generate the document-level relationship type of the arbitrary two entities based on the model optimized by the adaptive threshold loss function and the adaptive regularization loss function.

10. A computer device comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer-readable instructions, the steps of the document-level relationship extraction method based on reasoning paths as described in any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Document-level relation extraction method based on heterogeneous graph attention network

    CN114648017A

  • Knowledge graph construction method and device based on document relationship extraction

    CN115269857A

  • Document-level relation extraction method for fusing subgraph and displaying and constructing reasoning path

    CN122021625A

Cited By

  • Method for extracting energy power document level event argument based on dependency perception and role guidance

    CN120493937A

  • An energy power document-level event argument extraction method based on dependency perception and role guidance

    CN120493937B

  • Handwritten Chinese text recognition method

    CN120853192A

  • A method for recognizing handwritten Chinese text

    CN120853192B