Network document relationship extraction method and system

Abstract

The application provides a network document relationship extraction method and system, the network document relationship extraction method comprises the following steps: replacing all mentions in each entity with the first occurrence of the mention; training a pre-training model using document-level contrast pre-training and mask language model pre-training respectively; using the trained pre-training model to encode the context of a document sample to obtain a vector representation of each token, converting the named entity recognition information and part-of-speech tagging information of each token into a vector representation respectively and splicing the vector representation of each token to obtain a word embedding representation and a document embedding representation; performing representation enhancement on the entity to obtain a final embedding representation; using a binary cross-entropy loss function to calculate the possibility of the existence of a relationship between entities, and selecting the relationship with the highest possibility as the result of relationship classification. The application can effectively extract the relationship of network documents and provide effective support for network text processing.

Description

TECHNICAL FIELD

[0001] The application relates to a network document relationship extraction method and system, and belongs to the technical field of natural language processing and artificial intelligence. BACKGROUND

[0002] Relationship extraction is a key subtask in the field of natural language processing, aiming to extract relationship information between entities from unstructured text, and has been widely applied to automatic construction of large-scale knowledge graphs, knowledge question and answer systems and intelligent dialogue systems. Early relationship extraction solves the problem of predicting the relationship between a given entity pair in a single sentence, i.e. sentence-level relationship extraction. However, in real scenarios, there are still a large number of entity relationships that must be obtained through multiple sentences, which requires advancing the relationship extraction task to the document level. Compared with sentence-level relationship extraction, document-level relationship extraction has the difficulties of multiple mentions of entities, long distance between entities and mentions of entities, and a number of entities much larger than the number of entities labeled in the sentence-level relationship extraction dataset.

[0003] At present, most document-level relationship extraction methods are based on graphs. Graphs have a unique advantage in mining the structural features of text. Existing graph-based methods often construct different graph structures or use different graph neural network models to construct document graphs and predict the relationship between entities. However, graph-based methods usually require manual design of graph node and edge types, which is not simple and often introduces artificial noise. In addition, some pre-trained model-based methods can directly learn the sequential representation of text without introducing graph structures. However, these methods often design pre-training tasks to solve the multi-entity and multi-mention problem in the document-level relationship extraction task, and the ability of the model to resolve references is limited. Many sentence-level methods are also difficult to generalize to the document level.

[0004] Therefore, it is necessary to provide a network document relationship extraction method to solve the above problems. SUMMARY

[0005] The application aims to provide a network document relationship extraction method, which can effectively process network text.

[0006] To achieve the above-mentioned purpose, the application provides a network document relationship extraction method, mainly comprising the following steps:

[0007] Step 1, replacing all mentions in the entity with the first occurrence of the mention in the entity;

[0008] Step 2: Build a pre-trained model. Train the pre-trained model using document-level contrastive pre-training and masked language model pre-training respectively to obtain the loss function of document-level contrastive pre-training and the loss function of masked language model pre-training. After weighting, obtain the overall loss function.

[0009] Step 3: Insert start and end symbols before and after all mentions of each entity. Use the pre-trained model to perform context encoding on the document samples to obtain the vector representation of each tag. Convert the named entity recognition information and part-of-speech tagging information of each tag into vector representations and concatenate them with the vector representation of each tag to obtain word embedding representation and document embedding representation.

[0010] Step 4: Perform representation enhancement on the head entity or tail entity to obtain the final embedded representation of the head entity and tail entity;

[0011] Step 5: Map the head entity and tail entity to the hidden state respectively, use the binary cross-entropy loss function to calculate the probability that there is a relationship between the head entity and the tail entity, and select the relationship with the highest probability as the result of relationship classification.

[0012] As a further improvement of the present invention, in step 1, for entity M, there are n references {M1, M2, ..., M}. n}, where n≥1, when n>1, replace {M2, M3, ..., M} with M1. n},when n If the value is 1, skip step 1.

[0013] As a further improvement of the present invention, in step 2, the loss function for document-level comparison pre-training is as follows:

[0014]

[0015] Where DA represents the document sample containing the target relation to be predicted. It is the transpose of DA, where DB represents a document sample containing the same target relationship. This represents a randomly selected document sample containing different relationships, where N represents the number of randomly selected negative samples, and the logarithm is base e.

[0016] The loss function for pre-training the masked language model is as follows:

[0017]

[0018] in, Let S represent the masking markers, S represent the input word sequence, m(S) represent the set of masking markers in S, S\m(s) represent the unmasked markers in S, and P represent selecting a marker from the vocabulary to restore the masking markers. The probability of.

[0019] As a further improvement to the present invention, the overall loss function is as follows:

[0020]

[0021] Wherein, γ1 and γ2 are the weight parameters of the loss function for balancing the document-level contrastive pre-training loss function and the loss function for masked language model pre-training.

[0022] As a further improvement of the present invention, in step 3, for each word w in the document sample i w i Word embedding representation x i The calculation formula is as follows:

[0023] [x1, x2, ..., x l′ ] = BERT([w1, w2, ..., w l′ ]),

[0024] Where l′ is the document length, BERT represents the pre-trained model after document-level contrastive pre-training and masked language model pre-training; the named entity recognition information and part-of-speech tagging information of each tag are converted into vector representations to obtain vectors. and vector and The word embedding representation of the word wi is x i By concatenating the elements, we obtain the word embedding representation x′. i :

[0025]

[0026] Where, ; represents a concatenation operation, and document embedding is represented by H as:

[0027]

[0028] As a further improvement of the present invention, the embedding representations of the head entity and tail entity obtained in step 3 are enhanced to obtain the final embedding representations εh and εt of the head entity and tail entity:

[0029]

[0030]

[0031] Where → represents h m and t m It is a reference to the head entity h and the tail entity t, εh m and εt m A mention h represents the head entity h and the tail entity t.m and t m , dist() denotes the relationship distance between mentions.

[0032] As a further improvement of the present application, after obtaining the final embedding representation of the head entity and tail entity, the head entity and tail entity are mapped to hidden states z h and z t , respectively:

[0033] z h = σ(W h εh),

[0034] z t = σ(W t εt),

[0035] where {W h , W t} ∈ R d×d denote weight matrices, and σ denotes a nonlinear activation function.

[0036] As a further improvement of the present application, the relationship prediction is performed on the head entity and tail entity, and the possibility that the relationship r exists between the head entity and tail entity:

[0037]

[0038] where W r ∈ R d×d denotes a weight matrix, and b r ∈ R d denotes a bias term.

[0039] As a further improvement of the present application, the possibility that the relationship r exists between the head entity and tail entity is calculated using a binary cross-entropy loss function, and the calculation formula of the binary cross-entropy loss function is as follows: r

[0040]

[0041] where c is the preset number of relationship categories, r i is a binary label 0 or 1, r i is 1 when the relationship r exists, and 0 otherwise.

[0042] To achieve the above purpose, the present application provides a network document relationship extraction system, which applies the network document relationship extraction method as described above.

[0043] ​The beneficial effects of the present application are: the present application effectively solves the reference resolution problem caused by multiple entities and multiple mentions in the document-level relation extraction task, reduces the pressure of the model on different mentions in reference resolution while ensuring the completeness of expression and semantics, can make the model focus on the recognition and extraction of relations, and provides effective support for network text processing. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a structural schematic diagram of the network document relation extraction system of the present application.

[0045] Figure 2 is a schematic diagram of the present application in which the number of different mentions in the Chinese text under the document-level contrast pre-training of the network document relation extraction method. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in detail below in combination with the drawings and specific embodiments.

[0047] Here, it should be noted that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the scheme of the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0048] In addition, it should also be noted that the terms "include", "contain" or any other variant are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or equipment.

[0049] As shown in Figure 1 and Figure 2 , the present application proposes a network document relation extraction method based on relation distance and contrast pre-training and a network document relation extraction system applying the method, the network document relation extraction method introduces the concept of relation distance, uses document-level contrast pre-training to improve the ability of the system to extract relation facts from context, and then performs weighted embedding on the relation information of distant mentions, fully utilizes the relation information in the context of the mention, and mainly includes the following steps:

[0050] Step 1, using the first occurrence of each entity to replace all remaining mentions in the entity;

[0051] Step 2, establishing a pre-training model, respectively using document-level contrast pre-training and mask language model pre-training to train the pre-training model, obtaining the loss function of document-level contrast pre-training and the loss function of mask language model pre-training, and obtaining the overall loss function after weighting;

[0052] Step 3: Insert start and end symbols before and after all mentions of each entity. Use the pre-trained model to perform context encoding on the document samples to obtain the vector representation of each tag. Convert the named entity recognition information and part-of-speech tagging information of each tag into vector representations and concatenate them with the vector representation of each tag to obtain word embedding representation and document embedding representation.

[0053] Step 4: Perform representation enhancement on the head entity or tail entity to obtain the final embedded representation of the head entity and tail entity;

[0054] Step 5: Map the head entity and tail entity to the hidden state respectively, and use the binary cross-entropy loss function to calculate the probability that there is a relationship between the head entity and the tail entity to obtain the final prediction result.

[0055] The following will provide a detailed explanation of steps 1 through 9.

[0056] In step 1, preprocessing is performed first, replacing all remaining mentions of each entity with its first mention. In document relation extraction tasks, a token refers to all words and all additional concatenated symbols or vectors in a sentence. An entity is a concept in natural language. The two entities whose relations need to be predicted are called the head entity and the tail entity, respectively. Each entity has one or more different representations in the text, and each representation is called a mention of that entity. Each mention is also a token. That is, for entity M, there are n mentions {M1, M2, ..., M}. n}, where n≥1, when n>1, replace {M2, M3, ..., M} with M1. n When n=1, skip step 1.

[0057] In step 2, for document-level comparative pre-training, the concept of relational distance is introduced. The relational distance between mention pairs within the same sentence is 0, the relational distance between mention pairs in adjacent sentences is 1, and so on, to find one or more mention pairs most likely expressing a relational fact. A mention pair includes a head entity and a tail entity. That is, when the number of mentions for both the head and tail entities is 1, there is only one mention pair, and this mention pair is selected. When the number of mentions is one-to-many or many-to-many, there may be multiple mention pairs with the closest relational distance, which also means that there may be multiple relationships between these two entities. One or more mention pairs with the closest relational distance are selected, and these mention pairs are processed as multiple samples. The pre-training corpus used for comparison comes from large-scale remote supervision data. Remote supervision can use external knowledge bases to provide labels for the data, thus eliminating the need for manual annotation. Reference for mention pair selection under different mention ratios. Figure 2 As shown, the loss function for document-level contrastive pre-training is as follows:

[0058]

[0059] Where DA represents the document sample containing the target relation to be predicted. It is the transpose of DA, where DB represents a document sample containing the same target relationship. This represents a randomly selected document sample containing different relationships. N represents the number of randomly selected negative samples. The logarithm is based on the natural constant e. Each document sample selects the required mention pairs based on the relationship distance.

[0060] For masked language model pre-training, tags in a document are randomly masked, and the training system predicts missing tags using the remaining tags. The loss function for masked language model pre-training is as follows:

[0061]

[0062] in, Let S represent the masking markers, S represent the input word sequence, m(S) represent the set of masking markers in S, S\m(s) represent the unmasked markers in S, and P represent selecting a marker from the vocabulary to restore the masking markers. Mean probability.

[0063] The overall loss function of the system is obtained by weighting the loss function of the document-level contrastive pre-training and the loss function of the masked language model pre-training. The overall loss function is as follows:

[0064]

[0065] Wherein, γ1 and γ2 are the weight parameters of the loss function for balancing the document-level contrastive pre-training loss function and the loss function for masked language model pre-training.

[0066] In step 3, a start symbol [E] and an end symbol [ / E] are inserted before and after all references to each entity, respectively. For entity E i The start and end symbols are [E] and [E], respectively. i ] and [ / E i All mentions within the same entity use the same tokens. The network document is context-encoded using a pre-trained model to obtain a vector representation of each token.

[0067] For each word w in document sample D i w i Word embedding representation x i The calculation formula is as follows:

[0068] [x1, x2, ..., x l′ ]=BERT([w1, W2,..., w1′]),

[0069] where l' is the length of the document, and BERT represents the pre-trained model after document-level contrastive pre-training and mask language model pre-training. The named entity recognition information and the part-of-speech tagging information of each token are respectively converted into vector representations, obtaining vectors and The vectors and are concatenated with the word embedding representation x i of the word wi, to obtain the word embedding representation x' i :

[0070]

[0071] where represents the concatenation operation, and the document embedding representation H is:

[0072]

[0073] In step 4, the embedding representation of the entity is enhanced using the relationship distance, and the relationship information contained in the mention with a larger relationship distance is converted into a weight and integrated into the embedding representation of the entity.

[0074] In the word sequence, the closer the relationship distance between the mentions is, the more likely there is a relationship. The size of the relationship distance is converted into a weight, that is, for a mention h m or t m of the head entity h or the tail entity t, the closer the mention t m or h m of the other entity is to it, the higher the weight is. The embedding representations of the head entity and the tail entity obtained in step 3 are enhanced to obtain the final embedding representations εh and εt of the head entity and the tail entity:

[0075]

[0076]

[0077] where represents h m and t m is a mention of the head entity h and the tail entity t, and εh m and εt m represent the embedding representations of the mention h m and t m of the head entity h and the tail entity t, and dist() represents the relationship distance between the mentions. Here, the distance refers to the absolute distance, that is, the number of tokens between the two mentions.

[0078] In step 5, after obtaining the final embedding representations εh and εt of the head entity and the tail entity, the head entity and the tail entity are respectively mapped to the hidden state zh and z t :

[0079] z h = σ(W h εh),

[0080] z t = σ(W t εt),

[0081] where {W h , W t} ∈ R d×d denote the weight matrix, and σ denotes a nonlinear activation function such as tanh and ReLU, to obtain the hidden state z h and z t After that, the head entity and the tail entity are subjected to relationship prediction, the possibility of the existence of a relationship is calculated, and the relationship with the highest possibility is selected as the result of relationship classification, and the possibility of the existence of a relationship r between the head entity and the tail entity is:

[0082]

[0083] where W r ∈ R d×d denotes the weight matrix, and b r ∈ R d denotes the bias term, the document relationship extraction task is a multi-label classification task, and a binary cross-entropy loss function is used to compensate for the defects that the derivative form of the nonlinear activation function is easy to saturate and the gradient update is slow, and the calculation formula of the binary cross-entropy loss function is as follows:

[0084]

[0085] where c is a preset number of relationship categories, r i is a binary label 0 or 1, r i is 1 when the relationship r exists, and 0 otherwise.

[0086] In summary, the present application first effectively solves the problem of reference resolution caused by multiple entities and multiple mentions in the document-level relation extraction task through simple mention replacement, reduces the pressure of the model on different mentions in reference resolution while ensuring the completeness of expression and semantics, and enables the model to focus on relation recognition and extraction. Secondly, the present application introduces the concept of relation distance, which extends the contrast pre-training task suitable for sentence-level relation extraction to document-level, helping the system to find the mention pair between entities that is most likely to have a relationship. Thirdly, in order to make up for the relationship information contained in the long-distance mention that may be ignored by the document-level contrast pre-training, the present application integrates all mentions into the embedding representation of the entity with weight, so that the same entity presents different embedding representations in different entity pairs, making the relation prediction more accurate. Finally, the present application designs a complete document-level relation extraction system, which first performs mention replacement preprocessing on the initial text, then encodes the context through the existing pre-training model, uses document-level contrast pre-training and mask language model pre-training to enhance the context perception ability of the existing model, and finally uses the binary cross-entropy loss function for relation prediction.

[0087] The above examples are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for extracting relationships from network documents, characterized in that, The main steps include: Step 1: Replace all remaining mentions in each entity with the first occurrence of that mention; Step 2: Build a pre-trained model. Train the pre-trained model using both document-level contrastive pre-training and masked language model pre-training to obtain the loss functions for document-level contrastive pre-training and masked language model pre-training. Weight the results to obtain the overall loss function. The loss function for document-level contrastive pre-training in Step 2 is as follows: Among them, D A This represents a sample of documents containing the target relation that we want to predict. It is D A The transpose of D B This indicates a document sample that shares the same target relationship. This represents a randomly selected document sample containing different relationships, where N represents the number of randomly selected negative samples, and the logarithm is base e. Step 3: Insert start and end symbols before and after all mentions of each entity. Use the pre-trained model to perform context encoding on the document samples to obtain the vector representation of each tag. Convert the named entity recognition information and part-of-speech tagging information of each tag into vector representations and concatenate them with the vector representation of each tag to obtain word embedding representation and document embedding representation. Step 4: Perform representation enhancement on the head entity or tail entity to obtain the final embedding representations of the head entity and tail entity; perform representation enhancement on the embedding representations of the head entity and tail entity obtained in Step 3 to obtain the final embedding representations εh and εt of the head entity and tail entity: Where → represents h m and t m It is a reference to the head entity h and the tail entity t, εh m and εt m A mention h represents the head entity h and the tail entity t. m and t m The embedding representation, dist() represents the distance between references; Step 5: Map the head entity and tail entity to the hidden state respectively, use the binary cross-entropy loss function to calculate the probability that there is a relationship between the head entity and the tail entity, and select the relationship with the highest probability as the result of relationship classification.

2. The method for extracting network document relationships according to claim 1, characterized in that: In step 1, for entity M, there are n mentions {M1, M2, ..., M}. n }, where n≥1, when n>1, replace {M2,M3,…,M} with M1. n When n=1, skip step 1.

3. The method for extracting network document relationships according to claim 1, characterized in that: In step 2, the loss function for pre-training the masked language model is as follows: in, Let S represent the input word sequence, and m(S) represent the set of masking tags in S. \m(S) S represents the unmasked marker in S, and P represents selecting a marker from the vocabulary to restore the masked marker. The probability of.

4. The method for extracting network document relationships according to claim 3, characterized in that: The overall loss function is as follows: Wherein, γ1 and γ2 are the weight parameters of the loss function for balancing the document-level contrastive pre-training loss function and the loss function for masked language model pre-training.

5. The method for extracting network document relationships according to claim 3, characterized in that: In step 3, for each word w in the document sample i w i Word embedding representation x i The calculation formula is as follows: [x1,x2,…,x l′ ] = BERT([w1,w2,...,w l′ ]), where l′ is the document length, and BERT represents the pre-trained model after document-level contrastive pre-training and masked language model pre-training; the named entity recognition information and part-of-speech tagging information of each tag are converted into vector representations to obtain vectors. and vector and With the word w i Word embedding representation x i By concatenating the elements, we obtain the word embedding representation x′. i : Where, ; represents a concatenation operation, and document embedding is represented by H as:

6. The method for extracting network document relationships according to claim 5, characterized in that: After obtaining the final embedding representations εh and εt of the head and tail entities, the head and tail entities are mapped to the hidden state z, respectively. h and z t : With h =σ(W h εh), z t <σ(W t εt), Among them, {W h W t }∈R d×d Let σ represent the weight matrix and σ represent the nonlinear activation function.

7. The method for extracting network document relationships according to claim 6, characterized in that: Predict the relationship between the head and tail entities, and the probability that a relationship r exists between them: Among them, W r ∈R d×d Let b represent the weight matrix. r ∈R d This indicates the bias term.

8. The method for extracting network document relationships according to claim 7, characterized in that: The probability r of a relationship between the head entity and the tail entity is calculated using the binary cross-entropy loss function. The formula for the binary cross-entropy loss function is as follows: Where c is the preset number of relation types, r i The label is a binary label of 0 or 1, and r is used when relation r exists. i It is 1 if it is not 1, otherwise it is 0.

9. A network document relationship extraction system, characterized in that: The network document relationship extraction method as described in any one of claims 1-8 is applied.

Images