Overlapping relation triple extraction method based on global detection and bidirectional marking

Through the joint extraction model of global detection and bidirectional marking, the problem of inaccurate identification of unidirectional extraction mode and entity pairs in overlapping relationship triple extraction is solved, the completeness and accuracy of entity pairs are achieved, and the generalization of the model is improved. Performance and semantic relationship recognition capabilities.

CN119089908APending Publication Date: 2024-12-06SHENYANG AEROSPACE UNIVERSITY
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411141681.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art has the risk of a one-way extraction pattern leading to task failure in overlapping relationship triple extraction, and the entity's incomplete and inaccurate identification, and insufficient utilization of sentence context information.

Method used

Using the overlapping relationship triple extraction method based on global detection and bidirectional marking, the joint extraction model is constructed, including the encoder module, the entity pair recognition module based on bidirectional marking, and the relationship extraction module based on entity attention, to ensure the completeness of the entity pair. and accuracy, and fully integrate entity and context information.

Benefits of technology

The completeness and accuracy of entity pairs are achieved, the generalization performance of the model is improved, and the overlapping problem of relationship triplets can be efficiently handled, and the semantic relationships between entities are accurately identified.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089908A_ABST
    Figure CN119089908A_ABST
Patent Text Reader

Abstract

The invention discloses an overlapping relation triple extraction method based on global detection and bidirectional marking, and belongs to the technical field of natural language processing. Firstly, a data set is preprocessed, a model uses a pre-training model BERT as an encoder, initial semantic feature representation of sentences is obtained, and four linear layers are used for abstracting the encoded semantic feature representation so as to be suitable for different tasks; an entity pair identification module based on bidirectional marking extracts all entity pairs with potential relations from two directions; the global detection module filters the extracted entity pairs and screens out the entity pairs with low confidence; finally, a relation extraction module based on entity attention extracts a semantic relation between the entity pairs, and extraction of relation triples is completed; in the training stage, an entity pair negative sampling strategy is used, the generalization of the model is improved, the model is jointly extracted by training the entity relationship, the model is tested, and then the model is updated according to a test result. According to the invention, the knowledge of the unstructured text can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer natural language processing, and in particular to a method for extracting overlapping relationship triples based on global detection and bidirectional labeling. Background Art

[0002] With the advent of the digital age, fields such as big data, artificial intelligence, and natural language processing have developed rapidly, and the amount of information flow around the world has shown an unprecedented explosive growth. This trend not only comes from the Internet, but also includes social media, IoT devices, scientific research, medical records, and other fields. Numerous data sources generate a large amount of unstructured data, including text, images, audio, video, etc. These data are usually diverse, complex, and highly dynamic. How to effectively organize and efficiently use these data is a huge challenge. In response to the above challenges, Google proposed the knowledge graph technology, which expresses data as a large-scale semantic network of knowledge, which is currently one of the most effective means for multi-source heterogeneous data fusion.

[0003] Entity relationship extraction is an important task in the field of natural language processing and is also the key to building knowledge graphs. This task aims to identify entities and their semantic relationships from unstructured text and express them in a structured form, for example: usually in the form of entity relationship triples (head entity, relationship, tail entity).

[0004] At present, overlapping relation triple extraction has attracted more and more researchers' attention and has become a hot topic in current research. It aims to extract multiple relation triples that share entities from a given sentence. According to the degree of entity overlap, it can be divided into four types: normal, single entity overlap (SEO), entity pair overlap (EPO), and subject object overlap (SOO). Figure 1 As shown in the figure, the overlapping of relation triples is very common in large-scale natural language texts, and the extraction of overlapping relation triples is more challenging than the extraction of classic relation triples.

[0005] Although a lot of work has been invested in overlapping entity relationship triple extraction, the current overlapping entity relationship triple extraction task still has the following problems:

[0006] (1) In the extraction model based on annotation strategy, the mainstream extraction methods include head entity priority extraction, relationship priority extraction and entity pair priority extraction, but most of these methods adopt a one-way extraction mode, that is, the extraction work is carried out from one direction, which will lead to a problem. If the first step of the extraction task fails, the entire task will fail.

[0007] (2) For the model that prioritizes entity pairs for extraction, its performance is seriously affected by incomplete and inaccurate entity pair recognition, which leads to a decline in the overall performance of the model, and it fails to fully utilize the contextual information in the sentence in the relation extraction stage.

[0008] Therefore, how to better extract valuable information from massive unstructured texts and efficiently solve the overlapping problem in entity relationship extraction has attracted the attention of many researchers. With the development of deep learning technology, this field has further developed rapidly. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a method for extracting overlapping relationship triples based on global detection and bidirectional labeling. The method completes the extraction work by constructing a joint extraction model. The model mainly includes four modules: an encoder module, an entity pair recognition module based on bidirectional labeling, a global detection module and a relationship extraction module based on entity attention. First, the encoder module encodes the input sentence and obtains special representations for different tasks; secondly, the entity pair recognition module based on bidirectional labeling will identify all entity pairs with potential relationships; then the global detection module will filter entity pairs with low confidence; finally, the relationship extraction module based on entity attention will extract the relationship between entity pairs, realize the joint extraction of entity relationships, and efficiently solve the overlapping problem.

[0010] To achieve the above purpose, the technical solution adopted is:

[0011] A method for extracting overlapping relationship triples based on global detection and bidirectional labeling, the steps are as follows:

[0012] Encode the input sentence and obtain special representations for different tasks;

[0013] Construct an entity pair recognition module based on bidirectional marking and a global detection module. The entity pair recognition module based on bidirectional marking will extract all entity pairs with potential relationships from the direction of head entity-tail entity and tail entity-head entity to ensure the completeness of entity pair extraction; the global detection model will judge the extracted entity pairs and filter out entity pairs with low confidence to ensure the accuracy of entity pair extraction;

[0014] Construct a relationship extraction module based on entity attention. The relationship extraction module fully integrates entity and context information through the entity attention network to complete the extraction of the relationship between the head and tail entities;

[0015] The loss functions of the entity pair recognition module based on bidirectional labeling, the global detection module and the entity attention-based relationship extraction module are set, and the loss functions of the three modules are combined. At the same time, in order to enhance the generalization ability of the model, the entity pair negative sampling strategy is adopted in the training process to complete the training and testing of the entity relationship joint extraction model, and the entity relationship joint extraction model is updated according to the test results.

[0016] The encoding of the input sentence and obtaining special representations for different tasks specifically includes the following steps:

[0017] S1.1: Use the BERT tokenizer to tokenize the sentences in the dataset and get the initial encoding representation X = {x 1 ,x 2 ,…,x L}.

[0018] S1.2: Initialization encoding representation of the sentence X = {x 1 ,x 2 ,…,x L}, use the pre-trained model BERT to obtain the words x in the sentence i The corresponding semantic feature representation As shown in formula (1):

[0019] {h 1 ,h 2 ,…,h L}=BERT{x 1 ,x 2 ,…,x L} (1)

[0020] S1.3: After using the pre-trained model BERT to obtain the semantic feature representation of the sentence, the encoded semantic feature representation is further abstracted through four linear layers to suit different task requirements. Specifically, the semantic representation of the i-th word of the four context features is recorded as and The calculation process is shown in formulas (2) to (5):

[0021]

[0022] in They represent the weight matrices of the specific task representation of the global detection module, the specific representation of the head entity recognition in the entity pair recognition module, the specific representation of the tail entity recognition in the entity pair recognition module, and the specific task representation of the relation extraction module. They represent the corresponding bias vectors respectively.

[0023] The construction of the entity pair recognition module and the global detection module based on bidirectional marking specifically includes the following steps:

[0024] S2.1: The entity pair recognition module based on bidirectional labeling first identifies the head entity and the tail entity from the two directions of head entity-tail entity and tail entity-head entity respectively, then labels the corresponding tail entity and head entity, and finally forms an entity pair. This working mode can naturally solve the overlapping problem of shared entities, namely: SEO and SOO, and ensure the completeness of entity pair recognition. Since the entity pair recognition in the two directions is the same in internal structure and working principle, the specific introduction is based on first identifying the head entity and then labeling the corresponding tail entity;

[0025] S2.2: The global detection module uses the global detection matrix to filter out entity pairs with low confidence based on the recognition of entity pairs in S2.1 to ensure the accuracy of entity pair recognition.

[0026] The S2.1 specifically includes the following steps:

[0027] S2.1.1: For the head entity tagger in the sentence, a binary tagging strategy is adopted, marking the start and end positions of the head entity with "1" and other tags with "0";

[0028] The purpose of the head entity tagger is to identify all head entities from the input sentence. The tag prediction probability of each word in the sentence is calculated as shown in formulas (6) to (7):

[0029]

[0030] in, and They represent the probability that the i-th word is the starting position or the ending position of the head entity. The weight matrix representing the head entity start position marker or the head entity end position marker, is the corresponding bias, and σ represents the activation function.

[0031] S2.1.2: After obtaining the tagging result of the head entity tagger, the tail entity tagger tags the corresponding tail entity based on the head entity. Specifically, for a given head entity, the average pooling operation is first used to extract the features of the semantic representations corresponding to each word contained in the head entity to obtain the vector representation of the head entity. As shown in formula (8);

[0032]

[0033] in Represents the semantic representation of each word contained in the k-th head entity, and Avgpool represents the average pooling operation.

[0034] S2.1.3: The head entity vector representation is fused with the sentence through the Hadamard product. Based on the fusion process, the binary tagging strategy is also used for tail entity recognition, as shown in formulas (9) and (10):

[0035]

[0036] in and They respectively represent the predicted probability of the i-th word under the current head entity as the starting position and ending position of the corresponding tail entity. Hadamard product The weight matrix representing the head entity start position marker or the head entity end position marker, represents the corresponding bias, and σ represents the activation function.

[0037] S2.1.4: Combine all extracted head entities with their corresponding tail entities to form entity pairs with potential relationships.

[0038] The S2.2 specifically includes the following steps:

[0039] S2.2.1: After obtaining all entity pairs with potential relationships in the sentence, in order to ensure the accuracy of entity pair recognition, the global detection matrix is ​​used to eliminate incorrect entity pairs. The detailed process is as follows: for all entity pairs, the score value corresponding to each entity pair in the global detection matrix is ​​checked. If the score value exceeds the set threshold, the entity pair is retained, otherwise it is filtered out;

[0040] S2.2.2: If Figure 3 As shown in the green matrix in , given a sentence containing n words, the shape of the global detection matrix is ​​R n×n , each element of the matrix corresponds to the starting position of the head entity and the tail entity in each entity pair. The value of the element represents the confidence of the entity pair. The larger the value, the higher the confidence of the corresponding entity pair. The calculation formula of each element value in the global detection matrix is ​​shown in (11):

[0041]

[0042] in, are the encoded representations of the i-th and j-th tokens of the head and tail entity pairs in the input sentence. represents the weight matrix of the global detection task, b d ∈R represents the corresponding bias, and σ represents the activation function.

[0043] S2.2.3: Filter low-confidence entity pairs through the global detection matrix to complete the screening of entity pairs.

[0044] The construction of the entity attention-based relationship extraction module specifically includes the following steps:

[0045] S3.1: According to the entity pairs retained in S2.2, average pooling operations are performed on the semantic representations of the words contained in the head entity and the tail entity to obtain the vector representations of the head entity and the tail entity. and The formula description is shown in (12)(13):

[0046]

[0047] in, It is the semantic representation of each word contained in the k-th head entity or tail entity, and Avgpool represents the average pooling operation.

[0048] S3.2: After obtaining the vector representation of the head entity and the tail entity, the sentence is used as global information to enhance the head entity and the tail entity respectively, and the attention mechanism is used to calculate the relevance between the i-th word in the sentence and the entity respectively. As shown in formula (14):

[0049]

[0050] in, represents the weight matrix, v T express The transpose of .

[0051] S3.3: The importance of each word in the sentence is normalized by softmax, and the entity pair attention weight α corresponding to the sentence is obtained. 1 ,α 2 ,…,α n}, where the attention weight of the i-th word The calculation is shown in formula (15):

[0052]

[0053] S3.4: Use the output h of the encoding layer in S1.3 r Multiply it with the attention weight to calculate the sentence vector representation of the head entity and the tail entity respectively. and As shown in formula (16):

[0054]

[0055] S3.5: Concatenate the vector representations of the head entity and the tail entity with their corresponding sentence representations to obtain the enhanced entity representation, as shown in formulas (17) and (18):

[0056]

[0057] Among them, concat represents a concatenation operation.

[0058] S3.6: Use the Add&Norm layer to normalize each vector and effectively prevent the gradient from disappearing, as shown in formula (19):

[0059]

[0060] S3.7: Get the semantic representation m of the entity pair in S3.6 k Finally, the semantic relationship between entity pairs is predicted by multi-label classification. This classification method can handle the case where entity pairs have multiple semantic relationships (EPO). The prediction probability of the i-th relationship It can be calculated by formula (20);

[0061]

[0062] in, is the weight matrix of the tagger specific to relation i, represents the corresponding bias, and σ represents the activation function.

[0063] The updating of the entity relationship joint extraction model specifically includes the following steps:

[0064] S4.1: The loss of the entity pair recognition module based on bidirectional labeling consists of two parts, namely the loss from the head entity to the tail entity extraction direction and the loss from the tail entity to the head entity extraction direction. The loss from the head entity to the tail entity direction is defined as follows:

[0065]

[0066] Where L is the number of words in the input sentence, Respectively represent the predicted probabilities of the start and end positions of the head entity and the start and end positions of the tail entity, are the true value labels indicating the start and end positions of the head entity and the tail entity, respectively, and BCE(·) represents the binary cross entropy loss.

[0067] S4.2: The loss definition is the same as that in S4.1. There are also two losses in the direction from the tail entity to the head entity, denoted as L s2 and L o2 , the calculation formula is similar to (21)(22);

[0068] S4.3: For the global detection module, its loss function is as follows:

[0069]

[0070] Where L is the number of words in the input sentence, p i,j represents the predicted probability of the i-th word and the j-th word in the input sentence as the start position of the head entity and the start position of the tail entity in the global detection matrix, y i,j is the corresponding true value label, and BCE(·) represents the binary cross entropy loss.

[0071] S4.4: For the entity attention-based relation extraction module, the loss function is as follows:

[0072]

[0073] Where R is the set of predefined relations, |R| is the total number of relation types, represents the predicted probability of the i-th relationship, is its corresponding true value label, and BCE(·) represents the binary cross entropy loss.

[0074] S4.5: Calculate the total loss value L = L using the joint loss function s1 +L o1 +L s2 +L o2 +L g +L r ;

[0075] S4.6: To improve the model's ability to discriminate negative entity pairs, an entity pair negative sampling strategy was designed and adopted. During the training process, the negative sampling strategy is used to generate incorrect entity pairs in each sentence. The correct head entity or tail entity in each true triple is replaced by other incorrect head entities or tail entities. A simple example is shown in Table 1:

[0076] Table 1 Negative sampling strategy

[0077]

[0078] S4.7: Use the test set to test the overlapping relationship triple extraction model based on global detection and bidirectional labeling, record the model's accuracy, recall rate and F1 value as evaluation indicators of the detection model effect, and save them;

[0079] S4.8: Repeat the test on the overlapping relationship triple extraction model based on global detection and bidirectional labeling by modifying the hyperparameters, and save the hyperparameter with the highest F1 value in the test result as the final hyperparameter of the model; the hyperparameters include learning rate and number of iterations;

[0080] S4.9: Analyze the results of the overlapping relationship triplet extraction model based on global detection and bidirectional labeling, and verify the effectiveness of the model through the set F1 value.

[0081] Beneficial effects of the present invention:

[0082] 1. The present invention proposes a method for extracting overlapping relationship triples based on global detection and bidirectional labeling, which ensures the completeness and accuracy of entity pair recognition through a bidirectional labeling strategy and a global detection module.

[0083] 2. The present invention proposes an overlapping relationship triple extraction method based on global detection and bidirectional labeling. Through the entity attention-based relationship extraction module, it fully utilizes the entity representation information and the context information of the sentence, so it can accurately identify the semantic relationship between entities.

[0084] 3. The present invention proposes a method for extracting overlapping relationship triples based on global detection and bidirectional labeling. The entity pair negative sampling strategy designed in the training phase can improve the generalization performance of the model. At the same time, the model can efficiently handle the problem of overlapping relationship triples. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 An example diagram of the problem of overlapping relation triples provided by an embodiment of the present invention.

[0086] Figure 2 A flow chart of a method for extracting overlapping relationship triples based on global detection and bidirectional labeling provided in an embodiment of the present invention.

[0087] Figure 3 A specific example diagram of the overlapping relationship triple extraction method based on global detection and bidirectional labeling provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. This embodiment provides a method for extracting overlapping relationship triples based on global detection and bidirectional marking, and by constructing an overlapping relationship triple extraction model based on global detection and bidirectional marking, entities and inter-entity relationships can be effectively extracted from unstructured texts. Figure 2 As shown, the following steps are included:

[0089] S1: preprocess the dataset;

[0090] S1.1: In this embodiment, the datasets used are the NYT dataset and the WebNLG dataset, which are classic public datasets for joint extraction of entity relations; the NYT dataset is generated by remote supervision and has certain noise; the WebNLG dataset was originally created for natural language processing tasks. These two datasets contain two different versions, version 1 annotates the entire entity span, and version 2 only annotates the last word of the entity; in this embodiment, the two datasets are represented as NYT and WebNLG in version 1, and as NYT* and WebNLG* in version 2. By segmenting the datasets, the datasets NYT, WebNLG, NYT*, and WebNLG* are obtained;

[0091] The statistical information of the data set is shown in Table 2:

[0092] Table 2 Information statistics of the data set

[0093] Dataset Relationship Type Number of sentences in the training set Number of sentences in the validation set Number of sentences in the test set NYT 24 56195 5000 5000 WebNLG 246 5019 500 703 NYT* 24 56195 4999 5000 WebNLG* 171 5019 500 703

[0094] S1.2: To evaluate the performance of the model in handling the problem of overlapping relation triples, the test sets of NYT* and WebNLG* are split according to the triple overlapping patterns in the sentences.

[0095] The statistical information of the two is shown in Table 3:

[0096] Table 3 Overlap information statistics of the test set

[0097] Dataset Normal SEO EPO SOO NYT* 3222 1273 969 117 WebNLG* 239 448 6 85

[0098] S2: Construct an overlapping relation triple extraction model based on global detection and bidirectional labeling, such as Figure 2 As shown, the relation triple extraction model includes: an encoder module, an entity pair recognition module based on bidirectional labeling, a global detection module and a relation extraction module based on entity attention, including the following steps:

[0099] S2.1: Encode the input sentence and obtain special representations for different tasks;

[0100] In this embodiment, the encoding layer uses BERT to encode the sentence and obtains special representations for different tasks through the linear layer, including the following steps:

[0101] S2.1.1: Use the BERT tokenizer to tokenize the sentences in the dataset and get the initial encoding representation X = {x 1 ,x 2 ,…,x L}.

[0102] S2.1.2: For the initialization encoding representation of the sentence X = {x 1 ,x 2 ,…,x L}, use the pre-trained model BERT to obtain the words x in the sentence i The corresponding semantic feature representation As shown in formula (1):

[0103] {h 1 ,h 2 ,…,h L}=BERT{x 1 ,x 2 ,…,x L} (twenty four)

[0104] S2.1.3: After using the pre-trained model BERT to obtain the semantic feature representation of the sentence, the encoded semantic feature representation is further abstracted through four linear layers to suit different task requirements. Specifically, the semantic representation of the i-th word of the four context features is recorded as and The calculation process is shown in formulas (2) to (5):

[0105]

[0106] in They represent the weight matrices of the specific task representation of the global detection module, the specific representation of the head entity recognition in the entity pair recognition module, the specific representation of the tail entity recognition in the entity pair recognition module, and the specific task representation of the relation extraction module. They represent the corresponding bias vectors respectively.

[0107] S2.2: Construct an entity pair recognition module based on bidirectional marking. The entity pair recognition module based on bidirectional marking will extract all entity pairs with potential relationships from the head entity-tail entity and tail entity-head entity directions to ensure the completeness of entity pair extraction;

[0108] In this embodiment, the entity pair recognition module based on bidirectional marking first identifies the head entity and the tail entity from two directions respectively, then marks the corresponding tail entity and the head entity, and finally forms an entity pair. This working mode can naturally solve the overlapping problem of shared entities, namely: SEO and SOO, and ensure the completeness of entity pair recognition. Since the entity pair recognition in the two directions is the same in internal structure and working principle, the specific introduction is based on first identifying the head entity and then marking the corresponding tail entity as a representative:

[0109] S2.2.1: For the head entity tagger in the sentence, a binary tagging strategy is adopted, marking the start and end positions of the head entity with "1" and other tags with "0";

[0110] S2.2.2: The purpose of the head entity tagger is to identify all head entities from the input sentence. The tag prediction probability of each word in the sentence is calculated as shown in formulas (6) to (7):

[0111]

[0112] in, and They represent the probability that the i-th word is the starting position or the ending position of the head entity. The weight matrix representing the head entity start position marker or the head entity end position marker, is the corresponding bias, and σ represents the activation function.

[0113] S2.2.3: After obtaining the tagging result of the head entity tagger, the tail entity tagger tags the corresponding tail entity based on the head entity. Specifically, for a given head entity, the average pooling operation is first used to extract the features of the semantic representations corresponding to each word contained in the head entity to obtain the vector representation of the head entity. As shown in formula (8);

[0114]

[0115] in Represents the semantic representation of each word contained in the k-th head entity, and Avgpool represents the average pooling operation.

[0116] S2.2.4: The head entity vector representation is fused with the sentence through the Hadamard product. Based on the fusion process, the binary tagging strategy is also used to perform tail entity recognition, as shown in formulas (9) and (10):

[0117]

[0118] in and They represent the predicted probabilities of the i-th word under the current head entity as the start and end position of the corresponding tail entity. ° represents the Hadamard product, represents the weight matrix of the tail entity start position marker or the tail entity end position marker, represents the corresponding bias, and σ represents the activation function.

[0119] S2.2.5: Combine all extracted head entities with their corresponding tail entities to form entity pairs with potential relationships.

[0120] S2.3: Build a global detection module. The global detection model will judge the extracted entity pairs, filter out entity pairs with low confidence, and ensure the accuracy of entity pair extraction.

[0121] In this embodiment, the global detection module uses the global detection matrix to filter entity pairs with low confidence according to the recognition of entity pairs in S2.2 to ensure the accuracy of entity pair recognition, including the following steps:

[0122] S2.3.1: After obtaining all entity pairs with potential relationships in the sentence, in order to ensure the accuracy of entity pair recognition, use the global detection matrix to eliminate incorrect entity pairs. The detailed process is as follows: for all entity pairs, check the score value corresponding to each entity pair in the global detection matrix. If the score value exceeds the set threshold, retain the entity pair, otherwise filter it out;

[0123] S2.3.2: If Figure 2 As shown in the green matrix in , given a sentence containing n words, the shape of the global detection matrix is ​​R n×n , each element of the matrix corresponds to the starting position of the head entity and the tail entity in each entity pair. The value of the element represents the confidence of the entity pair. The larger the value, the higher the confidence of the corresponding entity pair. The calculation formula of each element value in the global detection matrix is ​​shown in (11):

[0124]

[0125] in, are the encoded representations of the i-th and j-th tokens of the head and tail entity pairs in the input sentence. represents the weight matrix of the global detection task, b d ∈R represents the corresponding bias, and σ represents the activation function.

[0126] S2.3.3: Filter low-confidence entity pairs through the global detection matrix to complete the screening of entity pairs.

[0127] S2.4: Construct a relationship extraction module based on entity attention. This module uses the entity attention network to fully integrate entity and context information to complete the extraction of the relationship between the head and tail entities;

[0128] In this embodiment, firstly, the sentence is used as global information to further semantically enhance the vector representations of the head entity and the tail entity in the entity pair; secondly, the enhanced representations of the head entity and the tail entity are fused to obtain a unified vector representation of the entity pair; finally, the input is input into the classifier to complete the extraction of the semantic relationship between the head entity and the tail entity, including the following steps:

[0129] S2.4.1: According to the entity pairs retained in S2.3, average pooling operations are performed on the semantic representations of the words contained in the head entity and the tail entity to obtain the vector representations of the head entity and the tail entity. and and, the formula description is shown in (12)(13):

[0130]

[0131] in, It is the semantic representation of the words contained in the k-th head or tail entity, and Avgpool represents the average pooling operation.

[0132] S2.4.2: After obtaining the vector representation of the head entity and the tail entity, the sentence is used as global information to enhance the head entity and the tail entity respectively, and the attention mechanism is used to calculate the relevance between the i-th word in the sentence and the entity respectively. As shown in formula (14):

[0133]

[0134] in, represents the weight matrix, v T express The transpose of .

[0135] S2.4.3: The importance of each word in the sentence is normalized by softmax, and the entity pair attention weight α corresponding to the sentence is obtained. 1 ,α 2 ,…,α n}, where the attention weight of the i-th word The calculation is shown in formula (15):

[0136]

[0137] S2.4.4: Use the output h of the encoding layer in S2.1.3 r Multiply it with the attention to calculate the sentence vector representation of the head entity and the tail entity respectively. and And, as shown in formula (16):

[0138]

[0139] S2.4.5: Concatenate the vector representations of the head entity and the tail entity with their corresponding sentence representations to obtain the enhanced entity representation, as shown in formulas (17) and (18):

[0140]

[0141] Among them, concat represents a concatenation operation.

[0142] S2.4.6: Use the Add&Norm layer to normalize each vector and effectively prevent the gradient from disappearing, as shown in formula (19):

[0143]

[0144] S2.4.7: Obtain the semantic representation m of the entity pair in S2.4.6 k Finally, the semantic relationship between entity pairs is predicted by multi-label classification. This classification method can handle the case where entity pairs have multiple semantic relationships (EPO). The prediction probability of the i-th relationship It can be calculated by formula (20).

[0145]

[0146] in, is the weight matrix of the tagger specific to relation i, represents the corresponding bias, and σ represents the activation function.

[0147] S3: According to the entity-relationship triples extracted in S2, loss functions are designed according to the entity pair recognition module based on bidirectional labeling, the global detection module, and the relationship extraction module based on entity attention. In order to improve the generalization of the model and accurately extract relationship triplets, the entity pair negative sampling strategy is used in the training phase. During the training process, the loss functions of the three modules are combined to complete the training and testing of the overlapping relationship triple extraction model based on global detection and bidirectional labeling, and the model is updated according to the test results.

[0148] In this embodiment, according to the entity-relationship triples extracted in S2, combined with the real labels of entities and relationships in the NYT, WebNLG, NYT* and WebNLG* data sets, the loss functions of the entity pair recognition module based on bidirectional labeling, the global detection module and the relationship extraction module based on entity attention are set, and the entity pair negative sampling strategy is used in the training stage. The loss functions of these three modules are combined during the training process to complete the training and testing of the overlapping relationship triple extraction model based on global detection and bidirectional labeling, and the model is updated according to the test results, including the following steps:

[0149] S3.1: The loss of the entity pair recognition module based on bidirectional labeling consists of two parts, namely the loss from the head entity to the tail entity extraction direction and the loss from the tail entity to the head entity extraction direction. The loss from the head entity to the tail entity direction is defined as follows:

[0150]

[0151] Where L is the number of words in the input sentence, Respectively represent the predicted probabilities of the start and end positions of the head entity and the start and end positions of the tail entity, are the true value labels indicating the start and end positions of the head entity and the tail entity, respectively, and BCE(·) represents the binary cross entropy loss.

[0152] S3.2: Similar to the loss definition described in S3.1, there are also two losses in the direction from the tail entity to the head entity, denoted as L s2 and L o2 , the calculation formula is similar to (21)(22);

[0153] S3.3: For the global detection module, its loss function is as follows:

[0154]

[0155] Where L is the number of words in the input sentence, p i,j represents the predicted probability of the global detection matrix for the i-th word and the j-th word in the input sentence as the start position of the head entity and the start position of the tail entity, y i,j is the corresponding true value label, and BCE(·) represents the binary cross entropy loss.

[0156] S3.4: For the entity attention-based relation extraction module, the loss function is as follows:

[0157]

[0158] Where R is the set of predefined relations, |R| is the total number of relation types, represents the predicted probability of the i-th relationship, is its corresponding true value label, and BCE(·) represents the binary cross entropy loss.

[0159] S3.5: Calculate the total loss value L = L using the joint loss function s1 +L o1 +L s2 +L o2 +L g +L r ;

[0160] S3.6: To improve the model's ability to discriminate negative entity pairs, an entity pair negative sampling strategy was designed and adopted. During the training process, the negative sampling strategy is used to generate incorrect entity pairs in each sentence. The correct head entity or tail entity in each true triple is replaced by other incorrect head entities or tail entities. A simple example is shown in Table 1:

[0161] Table 4 Negative sampling strategy

[0162]

[0163]

[0164] S3.7: Use the test set to test the overlapping relationship triple extraction model based on global detection and bidirectional labeling, record the model's accuracy, recall rate and F1 value as evaluation indicators of the detection model effect, and save them. The results are shown in Table 5;

[0165] Table 5 Entity relationship joint extraction results

[0166] Dataset Accuracy Recall F1 value NYT 93.0 93.0 93.0 WebNLG 92.0 89.4 90.7 NYT* 93.1 93.5 93.3 WebNLG* 94.4 94.0 94.2

[0167] S3.7: Split the NYT* and WebNLG* test sets according to the triple overlap pattern in the sentences and perform tests. Record the F1 value of the model as an evaluation indicator for detecting the model's handling of the overlap problem and save it. The results are shown in Table 6.

[0168] Table 6 Extraction results for overlapping problems

[0169] Dataset Normal SEO EPO SOO NYT* 91.2 95.2 95.2 90.8 WebNLG* 93.0 94.4 94.7 94.6

[0170] S3.9: Repeat the test on the overlapping relationship triple extraction model based on global detection and bidirectional labeling by modifying the hyperparameters, and save the hyperparameter with the highest F1 value in the test result as the final hyperparameter of the model; the hyperparameters include learning rate and number of iterations;

[0171] S3.10: Analyze the results of the overlapping relationship triplet extraction model based on global detection and bidirectional labeling, and verify the effectiveness of the model through the set F1 value.

Claims

1. A method for extracting overlapping relationship triples based on global detection and bidirectional labeling, It is characterized in that Here are the steps: Encode the input sentence and obtain special representations for different tasks; Construct an entity pair recognition module based on bidirectional marking and a global detection module. The entity pair recognition module based on bidirectional marking will extract all entity pairs with potential relationships from the direction of head entity-tail entity and tail entity-head entity to ensure the completeness of entity pair extraction; the global detection model will judge the extracted entity pairs and filter out entity pairs with low confidence to ensure the accuracy of entity pair extraction; Construct a relationship extraction module based on entity attention. The relationship extraction module fully integrates entity and context information through the entity attention network to complete the extraction of the relationship between the head and tail entities; Set the loss functions of the entity pair recognition module based on bidirectional labeling, the global detection module and the entity attention-based relationship extraction module, combine the loss functions of the three modules, and adopt the entity pair negative sampling strategy in the training process to complete the training and testing of the entity relationship joint extraction model, and update the entity relationship joint extraction model according to the test results.

2. The overlapping relationship triple extraction method based on global detection and bidirectional labeling according to claim 1, Features: The encoding of the input sentence and obtaining special representations for different tasks specifically includes the following steps: S1.1: Use the BERT tokenizer to tokenize the sentences in the dataset and get the initial encoding representation X = {x 1 ,x 2 ,…,x L }; S1.2: Initialization encoding representation of the sentence X = {x 1 ,x 2 ,…,x L }, use the pre-trained model BERT to obtain the words x in the sentence i The corresponding semantic feature representation As shown in formula (1): {h 1 ,h 2 ,…,h L }=BERT{x 1 ,x 2 ,…,x L } (1) S1.3: After obtaining the semantic feature representation of the sentence using the pre-trained model BERT, the encoded semantic feature representation is further abstracted through four linear layers to suit different task requirements; specifically, the semantic representation of the i-th word of the four context features is recorded as and The calculation process is shown in formulas (2) to (5): in They represent the weight matrices of the specific task representation of the global detection module, the specific representation of the head entity recognition in the entity pair recognition module, the specific representation of the tail entity recognition in the entity pair recognition module, and the specific task representation of the relation extraction module. They represent the corresponding bias vectors respectively.

3. The overlapping relationship triple extraction method based on global detection and bidirectional labeling according to claim 1, Features: The construction of the entity pair recognition module and the global detection module based on bidirectional marking specifically includes the following steps: S2.1: The entity pair recognition module based on bidirectional labeling first identifies the head entity and the tail entity from the head entity-tail entity and the tail entity-head entity directions respectively, then labels the corresponding tail entity and the head entity, and finally forms an entity pair; this working mode can naturally solve the overlapping problem of shared entities, namely: SEO and SOO, and ensure the completeness of entity pair recognition; since the entity pair recognition in the two directions is the same in internal structure and working principle, the specific introduction is based on first identifying the head entity and then labeling the corresponding tail entity as a representative; S2.2: The global detection module uses the global detection matrix to filter out entity pairs with low confidence based on the recognition of entity pairs in S2.1 to ensure the accuracy of entity pair recognition.

4. According to claim 3, the entity relationship joint extraction method based on entity pair attention mechanism, Features: The S2.1 specifically includes the following steps: S2.1.1: For the head entity tagger in the sentence, a binary tagging strategy is adopted, marking the start and end positions of the head entity with "1" and other tags with "0"; The purpose of the head entity tagger is to identify all head entities from the input sentence. The tagging probability of each word in the sentence is calculated as shown in formulas (6) to (7): in, and Respectively represent the predicted probability of the i-th word as the starting position or the ending position of the head entity; The weight matrix representing the head entity start position marker or the head entity end position marker, is the corresponding bias, σ represents the activation function; S2.1.2: After obtaining the tagging result of the head entity tagger, the tail entity tagger tags the corresponding tail entity based on the head entity. Specifically, for a given head entity, the average pooling operation is first used to extract the features of the semantic representations corresponding to each word contained in the head entity to obtain the vector representation of the head entity. As shown in formula (8); in Represents the semantic representation of each word contained in the k-th head entity, and Avgpool represents the average pooling operation; S2.1.3: The head entity vector representation is fused with the sentence through the Hadamard product. Based on the fusion process, the binary tagging strategy is also used to identify the tail entity, as shown in formulas (9) and (10): in and They respectively represent the predicted probability of the i-th word under the current head entity as the start position and end position of the corresponding tail entity; represents the Hadamard product, represents the weight matrix of the tail entity start position marker or the tail entity end position marker, represents the corresponding bias, σ represents the activation function; S2.1.4: Combine all extracted head entities with their corresponding tail entities to form entity pairs with potential relationships.

5. According to claim 3, the entity relationship joint extraction method based on entity pair attention mechanism, Features: The S2.2 specifically includes the following steps: S2.2.1: After obtaining all entity pairs with potential relationships in the sentence, for all entity pairs, check the score value corresponding to each entity pair in the global detection matrix. If the score value exceeds the set threshold, the entity pair is retained, otherwise it is filtered out; S2.2.2: Given a sentence with n words, the shape of the global detection matrix is ​​R n×n , each element of the matrix corresponds to the starting position of the head entity and the tail entity in each entity pair. The value of the element represents the confidence of the entity pair. The larger the value, the higher the confidence of the corresponding entity pair. The calculation formula of each element value in the global detection matrix is ​​shown in (11): in, is the encoded representation of the i-th and j-th tokens of the head and tail entity pairs in the input sentence; represents the weight matrix of the global detection task, b d ∈R represents the corresponding bias, σ represents the activation function; S2.2.3: Filter low-confidence entity pairs through the global detection matrix to complete the screening of entity pairs.

6. The overlapping relationship triple extraction method based on global detection and bidirectional labeling according to claim 1, Features: The construction of the entity attention-based relationship extraction module specifically includes the following steps: S3.1: According to the entity pairs retained in S2.2, average pooling operations are performed on the semantic representations of the words contained in the head entity and the tail entity to obtain the vector representations of the head entity and the tail entity. and The formula description is shown in (12)(13): in, It is the semantic representation of each word contained in the k-th head entity or tail entity, and Avgpool represents the average pooling operation; S3.2: After obtaining the vector representation of the head entity and the tail entity, the sentence is used as global information to enhance the head entity and the tail entity respectively, and the attention mechanism is used to calculate the relevance between the i-th word in the sentence and the entity respectively. As shown in formula (14): in, represents the weight matrix, v T express The transpose of S3.3: The importance of each word in the sentence is normalized by softmax, and the entity pair attention weight α corresponding to the sentence is obtained. 1 ,α 2 ,…,α n }, where the attention weight of the i-th word The calculation is shown in formula (15): S3.4: Use the output h of the encoding layer in S1.3 r Multiply it with the attention weight to calculate the sentence vector representation of the head entity and the tail entity respectively. and As shown in formula (16): S3.5: Concatenate the vector representations of the head entity and the tail entity with their corresponding sentence representations to obtain the enhanced entity representation, as shown in formulas (17) and (18): Among them, concat represents a concatenation operation; S3.6: Use the Add&Norm layer to normalize each vector and effectively prevent the gradient from disappearing, as shown in formula (19): S3.7: Get the semantic representation m of the entity pair in S3.6 k Finally, the semantic relationship between entity pairs is predicted by multi-label classification. This classification method can handle the situation where entity pairs have multiple semantic relationships. The prediction probability of the i-th relationship is It can be calculated by formula (20); in, is the weight matrix of the tagger specific to relation i, represents the corresponding bias, and σ represents the activation function.

7. The overlapping relationship triple extraction method based on global detection and bidirectional labeling according to claim 1, Features: The updating of the entity relationship joint extraction model specifically includes the following steps: S4.1: The loss of the entity pair recognition module based on bidirectional labeling consists of two parts, namely the loss from the head entity to the tail entity extraction direction and the loss from the tail entity to the head entity extraction direction. The loss from the head entity to the tail entity direction is defined as follows: Where L is the number of words in the input sentence, Respectively represent the predicted probabilities of the start and end positions of the head entity and the start and end positions of the tail entity, are the true value labels indicating the start and end positions of the head entity and the tail entity, respectively, and BCE(·) represents the binary cross entropy loss; S4.2: The loss definition is the same as that in S4.

1. There are also two losses in the direction from the tail entity to the head entity, denoted as L s2 and L o2 , the calculation formula is similar to (21)(22); S4.3: For the global detection module, its loss function is as follows: Where L is the number of words in the input sentence, p i,j represents the predicted probability of the global detection matrix for the i-th word and the j-th word in the input sentence as the start position of the head entity and the start position of the tail entity, y i,j is the corresponding true value label, BCE(·) represents the binary cross entropy loss; S4.4: For the entity attention-based relation extraction module, the loss function is as follows: Where R is the set of predefined relations, |R| is the total number of relation types, represents the predicted probability of the i-th relationship, is the corresponding true value label, BCE(·) represents the binary cross entropy loss; S4.5: Calculate the total loss value L = L using the joint loss function s1 +L o1 +L s2 +L o2 +L g +L r ; S4.6: To improve the model’s ability to discriminate negative entity pairs, an entity pair negative sampling strategy was designed and adopted. During the training process, the negative sampling strategy was used to generate incorrect entity pairs in each sentence, and the correct head entity or tail entity in each true triple was replaced by other incorrect head entities or tail entities. S4.7: Use the test set to test the overlapping relationship triple extraction model based on global detection and bidirectional labeling, record the model's accuracy, recall rate and F1 value as evaluation indicators of the detection model effect, and save them; S4.8: Repeat the test on the overlapping relationship triple extraction model based on global detection and bidirectional labeling by modifying the hyperparameters, and save the hyperparameter with the highest F1 value in the test result as the final hyperparameter of the model; the hyperparameters include learning rate and number of iterations; S4.9: Analyze the results of the overlapping relationship triplet extraction model based on global detection and bidirectional labeling, and verify the effectiveness of the model through the set F1 value.

Citation Information

Cited By

  • Threat intelligence triple extraction method and system based on double-pointer architecture

    CN119938930A

  • A threat intelligence triplet extraction method and system based on dual-pointer architecture

    CN119938930B