Joint extraction method of entity relations based on global pointer network

Abstract

The present invention discloses a method for joint extraction of entity relations based on a global pointer network, which solves the problem of inconsistent targets by treating the head and tail positions of an entity as a whole for discrimination. A conditional layer normalization fusion head entity information is introduced to guide the model to capture triple directional features; and a relationship classification task is added to extract the potential semantic relationship of sentences to filter out some triplets with incorrect predictions. The experimental results of the present invention on the public data set CMeIE show that the model constructed by the present invention can effectively identify relation triplets in sentences. A global pointer network is designed as a decoder to extract relation and tail entities from input sentences. The global pointer network treats the head and tail positions of an entity as a whole for discrimination, achieves consistency between training and prediction targets, enhances the performance of the entity relationship extraction model, and improves the accuracy of the model in extracting triplets.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and information extraction, and proposes an entity relationship joint extraction model based on a global pointer network. The model can be used for entity relationship extraction tasks of Chinese medical texts, and can provide technical support for the construction of knowledge graphs and their downstream fields. Background Art

[0002] With the advent of the big data era, massive amounts of unstructured data are stored on the Internet. How to use information extraction technology to mine valuable information has become a research focus in the field of natural language processing. Entity relationship extraction, as a key subtask of information extraction, is widely used in the construction of knowledge graphs and its downstream fields, such as information retrieval, recommendation systems, and intelligent question answering. This task aims to extract entity pairs with specific semantic relationships from unstructured text, expressed as relationship triples (head entity, relationship, tail entity).

[0003] At present, entity relationship extraction methods are divided into two categories: pipeline methods and joint methods. The pipeline method solves the two subtasks of entity recognition and relationship extraction independently. First, all entities in the sentence are identified using the entity model, and then the entity pairs are classified through the relationship model. The pipeline method ignores the intrinsic connection and dependency between the subtasks, and there is a serious error propagation problem. The joint method models and trains the two subtasks as a whole to achieve mutual promotion of the two subtasks. For example, the LSTM-based joint model proposed by Miwa et al. encodes sentences with BiLSTM and models the grammatical dependency between words through Tree-LSTM; Zheng et al. convert entity relationship extraction into a sequence labeling task. The model first jointly decodes the entity-relationship label of each token, and then matches the relationship triples according to the proximity principle. Compared with the pipeline method, the above joint method enhances the interaction between entities and relationships and alleviates the error propagation problem, but in terms of modeling ideas, the relationship is regarded as a discrete label on the entity pair, so it is difficult for the model to learn the correct classification features during the training process, and therefore it is impossible to effectively extract overlapping triples.

[0004] In recent years, some scholars have proposed methods based on annotation strategies to solve the problem of overlapping triples. For example, ETL-SPAN proposed by Yu et al. decomposes the entity relationship extraction task into two subtasks: head entity extraction and tail entity relationship extraction, and designs a multi-sequence annotation method that integrates span distance to achieve triple extraction; Wei et al. designed a cascade binary pointer annotation framework CasRel, which can learn the mapping function between the head entity and the tail entity under a given relationship, thereby achieving overall triple modeling. The annotation strategy-based method reduces the noise interference of redundant entities by jointly decoding entity boundaries and relationship categories, but this type of method uses a conventional pointer network to train the head and tail positions of the entity separately, without considering the entity integrity required for evaluation in the prediction stage, that is, requiring the head and tail positions to be predicted correctly at the same time, resulting in inconsistency between model training and prediction goals. Summary of the invention

[0005] To address this problem, this paper proposes a joint extraction method based on a global pointer network. The global pointer network treats the first and last positions as a whole and performs tail entity discrimination on the subsequences corresponding to the first and last positions. This means that the model is trained and evaluated with entity sequences as the basic unit, ensuring the consistency of training and prediction goals. In addition, in order to further improve the triple extraction effect, this paper introduces a conditional layer normalization method to fuse the head entity information. Compared with the traditional feature fusion method of summation and concatenation, this method can better guide the model to capture the triple directional features; it also adds a relation classification auxiliary task to extract the potential semantic relationship in the sentence based on the sentence vector. The model can filter out some incorrectly predicted triplets based on this result.

[0006] The method comprises the following steps:

[0007] Step 1: Extract features from the input sentence. Use the NEZHA pre-trained language model to extract global features from the input sentence, mine deep semantic features, and obtain encoding vectors rich in contextual information.

[0008] Step 2: Identify all head entities in the sentence. The encoding vector obtained in step 2 is marked using a pointer network to indicate whether each word in the sentence is the first and last position of an entity, and the nearest matching principle is adopted, that is, each first position marker matches the nearest tail position marker backwards, and the subsequence corresponding to the first position marker to the tail position marker is identified as the head entity.

[0009] Step 3: Introduce the conditional layer normalization method to fuse the encoding vector with the head entity features. Set the corresponding bias and weight in the layer normalization structure as a function of the head entity features, and use the obtained fusion vector as the input for relation and tail entity extraction.

[0010] Step 4: Extract the tail entity of each head entity under a specific relationship. A global pointer network is designed. According to the fusion vector output from step 4, the sentence is divided into several continuous subsequences under each predefined relationship and the subsequences are scored. The subsequences are identified as the correct tail entities based on the scores.

[0011] Step 5: After the above steps, the model may extract triples containing incorrect semantic relations. In order to alleviate this problem, the present invention uses the [CLS] vector with global semantic information in the encoding vector as the sentence vector, classifies its relations to identify the potential semantic relations in the sentence, and thereby filters out some unreasonable triples in the extraction results.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1. The present invention designs a global pointer network as a decoder to extract the relationship and tail entities of the input sentence. The global pointer network treats the first and last positions of the entity as a whole for discrimination. This means that the model is trained and evaluated with the entity sequence as the basic unit, which makes the model more global, achieves consistency between training and prediction goals, and enhances the performance of the entity relationship extraction model.

[0014] 2. The present invention introduces a conditional layer normalization method to fuse the head entity features with the encoding vector. This method can not only effectively alleviate the gradient vanishing and gradient exploding problems, but also fully capture the triple direction information in the input sentence and enhance the representation ability of entity relationship features.

[0015] 3. The present invention adds a relation classification auxiliary task to identify the potential semantic relationship in the input sentence. This task can not only guide the model to better learn the sentence vector features, but also filter out some incorrectly extracted triples based on the classification results, thereby improving the accuracy of the model in extracting triples. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the overall architecture of the entity relationship extraction model based on the global pointer network. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0018] The goal of the present invention is to extract all possible triples in a sentence. In order to more effectively extract overlapping triples and avoid noise interference from redundant entities, the present invention models and solves the entity relationship extraction task based on a method of annotation strategy, as shown in formula (1):

[0019]

[0020] Among them, h, r, and t represent the head entity, relation, and tail entity of the triple, respectively, X represents the input sentence, and Ω represents the set of all relations in the dataset.

[0021] Based on this method, the present invention decomposes the entity relationship extraction task into three subtasks: 1) identifying all head entities in the sentence; 2) for each head entity, extracting the tail entity under a specific relationship; 3) filtering some unreasonable triples according to the sentence semantics.

[0022] In order to achieve the above tasks, the overall architecture of the joint extraction method based on the global pointer network proposed in this invention is as follows: Figure 1 As shown in the figure, it is divided into three modules: 1) Encoding and head entity recognition module, which uses NEZHA to extract features of the input sentence, obtains the encoding vector and marks all the head entities in the sentence through the pointer network; 2) Relation and tail entity extraction module, which introduces conditional layer normalization to fuse the head entity features and the encoding vector, and designs a global pointer network to extract the tail entity sequence under a specific relationship; 3) Relation classification module, which extracts the latent semantic relationship of the sentence and eliminates some mispredicted triplets based on the extraction results.

[0023] 1. Encoding and Head Entity Recognition Module

[0024] Step 1: In order to fully capture the deep semantic features of the sentence, this module uses the NEZHA pre-trained language model to extract features from the input sentence and obtain the corresponding encoding vector.

[0025] NEZHA is developed based on the BERT pre-trained language model, and uses additional optimization solutions such as functional relative position encoding, full-word masking, and mixed precision training to enhance the model's ability to represent the contextual information of input sentences.

[0026] H=NEZHA(X) (2)

[0027] Where X = (χ1, χ2, …, χ n ) represents the input sentence, n is the length of the input sentence, H = [h1,h2,…,h n ] represents the encoding vector of each position in the sentence.

[0028] Step 2: Decode the encoding vector H directly to identify all possible head entities in the input sentence. The present invention trains two binary pointer taggers to mark the head and tail positions of the head entities by assigning binary labels (0 / 1) to each word, where 1 indicates that the current word is the head (tail) position of a head entity, and 0 indicates that the current word is not the head (tail) position of a head entity. The tagging process is as follows:

[0029]

[0030]

[0031] Among them, W (·) represents the training parameter matrix, b (·) represents the bias vector, and σ represents the sigmoid activation function. and They represent the probability that the word at position i in the sentence is the beginning and end of the head entity respectively.

[0032] The model will determine the binary label of each word based on the relationship between the probability and the threshold. For sentences containing multiple head entities, this module adopts the nearest matching principle, that is, each first position marker matches the nearest tail position marker backwards, and the subsequence corresponding to the first position marker to the tail position marker is identified as the head entity.

[0033] This step uses the minimization of the binary cross entropy loss function to train the model parameters as follows:

[0034]

[0035] Where L represents the sequence length, Indicates the true label of the i-th character at the start or end position.

[0036] 2. Relation and tail entity extraction module

[0037] Step 3: Different from step 2 in identifying the head entity, in addition to the encoding vector of the input sentence, the head entity features need to be considered when extracting relations and tail entities. Currently, in joint models, simple addition or concatenation methods are usually used to fuse features, which limits the expression of fused features.

[0038] by Figure 1 Take the correct triplet (chronic kidney disease, high-risk population, elderly) and the wrong triplet (elderly, high-risk population, chronic kidney disease) obtained by reversing the head and tail entities as an example. If the above fusion method is used for these two groups of triplets, the same fusion vector will be obtained. To address this unreasonable phenomenon, it is necessary to consider the directional information of the triplet during feature fusion. In this regard, the present invention refers to the idea of ​​conditional batch normalization and introduces the conditional layer normalization method CLN (Conditional LayerNormalization), setting the corresponding bias and weight in the layer normalization structure as a function of the condition to be fused. The specific calculation of CLN is as follows:

[0039]

[0040] in, represents the feature information input by the CLN method, μ represents the mean of the feature information, σ 2 represents the variance of feature information, ∈ is a positive number tending to 0, γ and θ are unconditional training parameters, c γ and c β They respectively represent two input conditional information to be fused, and W1 and W2 are the training matrices of the conditional information to be fused.

[0041] As can be seen from the above formula, the CLN structure not only aligns c through the W1 and W2 training matrices γ 、c β , γ, β dimensions, and also c γ and c β Mapped into different vector spaces to learn the directional information of the conditions to be fused.

[0042] like Figure 1 As shown, the present invention uses the CLN method to fuse the head entity features and the encoding vector output in step 1, and the c γ and c β All are set as head entity encoding. The feature fusion process is as follows:

[0043] H'=CLN(H,h head ,h head ) (7)

[0044] h head =Concat(h start ,h end ) (8)

[0045] Among them, H' represents the fusion encoding vector for relation and tail entity extraction, h head It is represented as the head entity code, and the encoding vector h corresponding to the start and end positions of the entity extracted by step 1 and step 2 start and h end Splicing obtained.

[0046] Step 4: Extract the possible tail entities in the sentence according to the fusion features under each predefined relationship. To this end, the present invention designs a global pointer network to distinguish the first and last positions of the entity as a whole, rather than marking the first and last positions of the entity separately, so that the model has a more global view and achieves consistency between model training and prediction goals.

[0047] like Figure 1As shown in the figure, the global pointer network regards the input sentence of length n as n(n+1) / 2 continuous subsequences of different lengths, represented as (i,j), where i represents the starting position and j represents the ending position. The goal of this module is to score each subsequence and identify the correct tail entity based on the score. Assuming that there are m types of relationship categories in the dataset, the model will identify the subsequences in m relationship subspaces respectively, which means that this paper converts the relationship and tail entity extraction tasks into m "n(n+1) / 2 select k" multi-label classification tasks, where k represents the number of tail entities.

[0048] First, the global pointer network uses two linear layers to perform a linear transformation on the fused encoding vector H' to obtain a vector sequence and Secondly, in order to enhance the sensitivity of the pointer network to the length and span of the tail entity, the relative position encoding RoPE is introduced to satisfy The transformation matrix R is applied to q a and k a vector; finally, use and The inner product of α (i, j), represents the score of the subsequence from i to j as a complete tail entity, and all subsequences with scores greater than the threshold are regarded as the tail entities of the current head entity under the αth relationship. The global pointer network annotation process is as follows:

[0049]

[0050]

[0051]

[0052] in, represents the training parameter matrix, Represents the bias vector. For each extracted head entity, repeat the above relation and tail entity extraction operations to extract all possible triples in the sentence.

[0053] For multi-label classification tasks, the conventional idea is to convert them into n(n+1) / 2 binary classification tasks, which will lead to serious category imbalance problems. In this regard, this module introduces Circle Loss, so that the score of each entity subsequence is not less than the score of the non-entity subsequence, and finally outputs the k subsequences with the highest scores. For the scenario where the number of tail entities in the current task is not fixed, this article adds an s0 threshold to determine the number of tail entities that the module finally outputs, so that the scores of entity subsequences are greater than s0, and the scores of non-entity subsequences are less than s0, and finally outputs all subsequences greater than the threshold s0. This step trains the model parameters by minimizing this loss function, as shown below:

[0054]

[0055] Among them, P α represents all non-tail entity subsequences of the current head entity under the αth relationship, Q α Represents all tail entity subsequences of the current head entity under the ath relationship.

[0056] 3. Relationship Classification Module

[0057] Step 5: After steps 1 to 4, the model may extract triplets containing incorrect semantic relations. To alleviate this problem, this paper adds a relation classification auxiliary task, which inputs the [CLS] vector with global semantic information in the encoding vector as the sentence vector into this module, identifies the potential semantic relations in the sentence, and uses this to filter out some unreasonable triplets in the model extraction results, as shown below:

[0058]

[0059] in, Indicates the probability that the current sentence has the kth relationship, W k represents the parameter matrix to be trained, b k represents the bias vector, and σ represents the sigmoid activation function.

[0060] In the prediction stage of the model, if the probability of a certain relationship k If it is less than a given threshold, then the sentence is deemed to not have this kind of semantic relationship k, and all triples with relationship k in the model extraction results are filtered out.

[0061] In this step, the model uses a binary cross entropy loss function to train the model parameters as follows:

[0062]

[0063] in, represents the true label of the kth relation in the current sentence, and M represents the number of predefined relations.

[0064] Step 6: The joint loss function of the present invention is obtained by adding the losses in steps 2, 4, and 5, and the parameters of the overall model are learned by minimizing the joint loss, as shown below:

[0065] Loss=Loss subject +Loss object +Loss rel (14)

[0066] The present invention uses the Adam algorithm to train the model and adopts the Exponential Moving Average (EMA) to ensure the stability of the training process.

[0067] Experimental Dataset Introduction

[0068] The present invention uses the public data set CMeIE, which comes from the CHIP2020 academic evaluation competition and is jointly provided by "Zhengzhou University", "Peking University", "Pengcheng Laboratory" and "Harbin Institute of Technology (Shenzhen)". The data comes from pediatrics and more than 100 common diseases (the pediatric training corpus comes from 518 pediatric diseases, and the common disease training corpus comes from 109 common diseases). A total of 28,000 disease statements, nearly 75,000 triple data and 53 relationship types are annotated. The data set is divided into 14,339 training set data, 3,585 validation set data, and 4,482 test set data, containing a total of 53 Schema constraints.

[0069] Zeng et al. divided the sentences in the dataset into three categories according to the degree of triple overlap: Normal, EntityPairOverlap (EPO), and SingleEntityOverlap (SEO). The Normal type means that the head entity or tail entity of any triple in the sentence does not overlap with the head and tail entities of other triples. The EPO type means that the triples in the sentence have overlapping entity pairs, that is, two entities have multiple relationships at the same time. The SEO type means that there are no overlapping entity pairs, but there are overlapping entities. The CMeIE dataset sample is shown in Table 1:

[0070] Table 1 CMeIE dataset examples

[0071]

[0072] Experimental parameters introduction

[0073] The experiment of the present invention uses a GTX 2080Ti graphics card to run the code, with a video memory of 11G. The experiment is carried out on a Linux centos platform, using Python3.6 / Keras 2.2.4 / Tensorflow 1.1.14, and the experimental parameter settings are shown in Table 2:

[0074] Table 2 Hyperparameters of the model

[0075]

[0076]

[0077] Experimental evaluation indicators

[0078] The triples extracted by the present invention are considered to be correct only when the head and tail entity boundaries are correct at the same time and the corresponding relationship categories are correct. The task evaluation indicators are the following three: Precision, Recall, and F1-Score. The details are as follows:

[0079]

[0080]

[0081]

[0082] Among them, TP represents the correct number of extracted entity relationship triples, TP+FP is the total number of extracted triples, and TP+FN represents the total number of triple labels.

[0083] Experimental results analysis

[0084] In order to verify the effectiveness of the method proposed in the present invention in improving the triple extraction effect, the present invention is compared with other joint extraction models on the CMeIE Chinese medical dataset, as shown in Table 3:

[0085] 1.InfoExtractor2.0 is the baseline model for the Baidu Information Extraction Competition 2020. A structured tagging strategy is designed to fine-tune the pre-trained language model, through which multiple overlapping SPOs can be extracted in one process.

[0086] 2. The position-assisted step-by-step labeling model first determines the main entity by labeling the head and tail positions, and then labels the corresponding object entities under the preset relationship attributes one by one. In order to improve the extraction effect, triple position auxiliary information is introduced in the labeling process, and the self-attention mechanism is combined to enhance the feature representation.

[0087] 3. MultiHead Selection replaces the LSTM encoding layer with the Bert pre-trained language model, and replaces the sequence annotation CRF with pointer annotation for the NestedNER problem. This model regards the relationship classifier as a linear classifier of entity pairs, and for each entity pair, only the last character of the current entity segment is selected for relationship prediction.

[0088] 4. Biaffine Attention uses Bert as a shared encoding layer, combines the encoding vector with the entity label embedding, and uses the Biaffine classifier to predict the relationship between tokens.

[0089] 5.CasRel uses BERT as a shared encoding layer, a cascaded binary tagging framework that uses a pointer network to extract relations and entities. It first extracts the head entity, then extracts the tail entity based on the head entity and judges the relationship.

[0090] Table 3. Comparison experiment with existing extraction models

[0091]

[0092] It can be seen from Table 3 that the performance of the present invention on the CMeIE dataset is significantly improved compared with other joint extraction models. From the analysis of the results, it can be obtained that:

[0093] (1) InfoExtractor 2.0 uses multiple sequences to jointly annotate entities and relations, which can effectively identify the overlapping of entity pairs in triples. However, there may be redundant matching of head and tail entities, resulting in low extraction accuracy.

[0094] (2) By comparing the results of MultiHead Selection and Biaffine Attention, it can be found that compared with simply linearly changing the head and tail entity sequences, using the biaffine attention matrix can better capture the mapping relationship from the head entity sequence to the tail entity sequence, thereby improving the recall rate of the model.

[0095] (3) By comparing the results of Biaffine Attention and CasRel, it can be found that considering the relationship as a function mapping the head entity to the tail entity rather than a discrete label between entity pairs can better guide model training and improve the model triple extraction performance.

[0096] Generally speaking, the more triples there are in a sentence, the more complex the sentence structure is. In order to explore the extraction performance of the model in sentences of different complexity, the present invention conducts experiments on sentences with different numbers of triples on the validation set. The experimental results are shown in Table 4. From the experimental results, we can find that the extraction performance of the present invention on sentences of five complexity levels is better than that of the baseline model CasRel. This evidence shows that the present invention can more effectively model sentences containing multiple triples and extract triples.

[0097] In order to further explore the extraction performance of models with different overlapping types, the present invention divides the validation set of CMeIE into three types: Normal, EPO, and SEO, and compares the extraction performance of the present invention and the baseline model CasRel on the three sentence patterns. The experimental results are shown in Table 5. The experimental results show that the extraction performance of the model of the present invention on the three overlapping types of sentences is better than that of the baseline model CasRel. This evidence shows that the present invention can more effectively solve the overlapping triple problem.

[0098] Table 4 Experimental results on sentences containing different numbers of triples

[0099]

[0100] Table 5 Experimental results on sentences with different overlapping types

[0101]

[0102] In order to explore the influence of model components on the overall performance of the model, by adding, deleting or replacing model components, the contribution of each component to the experimental results of the model in the CMeIE dataset was evaluated while controlling other structure and overall parameters of the model. A total of 5 groups of comparative experiments were set up, namely:

[0103] Ours: means all components of the retained model

[0104] -NEZHA: NEZHA is replaced by the Bert pre-trained language model

[0105] -Global Pointer Network: Replace the global pointer network with a multi-layer binary pointer network

[0106] -CLN(Add): Use the traditional addition method to fuse the header entity vector and the encoding vector

[0107] - Relationship classification: Remove the relationship classification auxiliary task

[0108] The experimental results are shown in Table 6:

[0109] Table 6 Model ablation experiment results

[0110]

[0111] The analysis of the experimental results is as follows:

[0112] When NEZHA is replaced with the BERT pre-trained language model, the recall rate of the model decreases significantly, indicating that the NEZHA pre-trained language model can capture richer contextual information.

[0113] When the global pointer network decoder is replaced by a multi-layer pointer network decoder, the overall performance of the model decreases, indicating that the global pointer decoder enables the model to better capture the overall information of the tail entity and the mapping relationship between relations and entities.

[0114] When the conditional layer normalization is replaced by the vector addition method, the overall performance of the model decreases, indicating that the conditional layer normalization method can learn the directional information of the triplet and guide the model to more effectively fuse the head entity features.

[0115] Without the relation classification module, the overall performance of the model has declined, which shows that the relation auxiliary task can not only enhance the model's learning of sentence-level features, but also effectively eliminate incorrect triples.

Claims

1. A joint entity relationship extraction method based on a global pointer network, characterized by: The method comprises the following steps: Step 1: Extract features from the input sentence; use the NEZHA pre-trained language model to extract global features from the input sentence, mine deep semantic features, and obtain encoding vectors rich in contextual information; Step 2: Identify all head entities in the sentence; use the pointer network to mark the encoding vector obtained in step 1 to determine whether each word in the sentence is the first and last position of the entity, and use the nearest matching principle, that is, each first position mark matches the nearest tail position mark backwards, and the subsequence corresponding to the first position mark to the tail position mark is identified as the head entity; Step 3: Introduce the conditional layer normalization method to fuse the encoding vector and the head entity features; set the corresponding bias and weight in the layer normalization structure as a function of the head entity features, and use the obtained fusion vector as the input for relation and tail entity extraction; Step 4: Extract the tail entity of each head entity under a specific relationship; design a global pointer network, divide the sentence into several continuous subsequences under each predefined relationship according to the fusion vector output in step 3, score the subsequences, and determine which subsequences are the correct tail entities according to the scores; Step 5: The model may extract triplets containing incorrect semantic relations. The [CLS] vector with global semantic information in the encoding vector is used as the sentence vector, and its relations are classified to identify the potential semantic relations in the sentence and filter out some unreasonable triplets in the extraction results. The entity relationship extraction task is modeled and solved based on the annotation strategy method, as shown in formula (1): Among them, h, r, and t represent the head entity, relation, and tail entity of the triple respectively, X represents the input sentence, and Ω represents the set of all relations in the data set; Step 4: Under each predefined relationship, extract the possible tail entity in the sentence based on the fusion features; design a global pointer network to distinguish the first and last positions of the entity as a whole, rather than marking the first and last positions of the entity separately, so that the model has a more global view and achieves consistency between model training and prediction goals; The global pointer network regards an input sentence of length n as n(n+1) / 2 continuous subsequences of different lengths, represented as (i,j), where i represents the starting position and j represents the ending position; each subsequence is scored and the correct tail entity is identified based on the score; assuming that there are m types of relationship categories in the dataset, the model will identify the subsequences in m relationship subspaces respectively, converting the relationship and tail entity extraction tasks into m "n(n+1) / 2 select k" multi-label classification tasks, where k represents the number of tail entities; First, the global pointer network uses two linear layers to perform a linear transformation on the fused encoding vector H' to obtain a vector sequence and Secondly, in order to enhance the sensitivity of the pointer network to the length and span of the tail entity, the relative position encoding RoPE is introduced to satisfy The transformation matrix R is applied to q a and k a vector; finally, use and The inner product of α (i, j), represents the score of the subsequence from i to j as a complete tail entity. All subsequences with scores greater than the threshold are regarded as tail entities of the current head entity under the αth relationship. The global pointer network annotation process is as follows: in, represents the training parameter matrix, represents the bias vector; for each extracted head entity, repeat the above relation and tail entity extraction operations to extract all possible triples in the sentence; Circle Loss is introduced to ensure that the score of each entity subsequence is not less than the score of the non-entity subsequence, and finally the k subsequences with the highest scores are output; an s0 threshold is added to determine the number of tail entities that the module finally outputs, so that the scores of entity subsequences are all greater than s0, and the scores of non-entity subsequences are all less than s0, and finally all subsequences greater than the threshold s0 are output; the model parameters are trained by minimizing this loss function, as shown below: Among them, P α represents all non-tail entity subsequences of the current head entity under the αth relationship, Q α Represents all tail entity subsequences of the current head entity under the ath relationship.

2. The entity relationship joint extraction method based on global pointer network according to claim 1 is characterized by: Step 1: Use the NEZHA pre-trained language model to extract features from the input sentence and obtain the corresponding encoding vector; NEZHA is developed based on the BERT pre-trained language model, and additionally uses functional relative position encoding, full-word masking, and mixed-precision training to enhance the model's ability to represent contextual information of input sentences: H=NEZHA(X) (2) Where X=(x1,x2,…,x n ) represents the input sentence, n is the length of the input sentence, H = [h1,h2,…,h n ] represents the encoding vector of each position in the sentence.

3. The entity relationship joint extraction method based on global pointer network according to claim 2 is characterized by: Step 2: Decode the encoding vector H directly to identify all possible head entities in the input sentence, train two binary pointer taggers, and annotate the head entity's first and last positions by assigning a binary label (0 / 1) to each word, where 1 indicates that the current word is the first / last position of a head entity, and 0 indicates that the current word is not the first / last position of a head entity; the tagging process is as follows: Among them, W (·) represents the training parameter matrix, b (·) represents the bias vector, σ represents the sigmoid activation function; and Respectively represent the probability that the word at position i in the sentence is the beginning and the end of the head entity; The model will determine the binary label of each word based on the relationship between the probability and the threshold. For sentences containing multiple head entities, the nearest matching principle is adopted, that is, each first position marker matches the nearest tail position marker backwards, and the subsequence corresponding to the first position marker to the tail position marker is identified as the head entity. The model parameters are trained by minimizing the binary cross entropy loss function as follows: Where L represents the sequence length, Indicates the true label of the i-th character at the start or end position.

4. The entity relationship joint extraction method based on global pointer network according to claim 3 is characterized by: Step 3: Different from step 2 in identifying the head entity, in addition to the encoding vector of the input sentence, the head entity features need to be considered additionally when extracting relations and tail entities; The conditional layer normalization method CLN is introduced, and the corresponding bias and weight in the layer normalization structure are set as a function of the conditions to be fused; the specific calculation of CLN is as follows: Among them, S represents the feature information input by the CLN method, μ represents the mean of the feature information, and σ 2 Represents the variance of feature information, ∈ is a positive number tending to 0, γ and β are unconditional training parameters, c γ and c β They represent two input conditional information to be fused, W1 and W2 are the training matrices of the conditional information to be fused; The CLN structure not only aligns c through the W1 and W2 training matrices γ 、c β , γ, β dimensions, and also c γ and c β Mapping to different vector spaces to learn the directional information of the conditions to be fused; Use the CLN method to fuse the head entity features and the encoding vector output in step 1, and replace the c γ and c β All are set as head entity encoding; the feature fusion process is as follows: H'=CLN(H,h head ,h head ) (7) h head =Concat(h start ,h end ) (8) Among them, H' represents the fusion encoding vector for relation and tail entity extraction, h head It is represented as the head entity code, and the encoding vector h corresponding to the start and end positions of the entity extracted by step 1 and step 2 start and h end Splicing obtained.

5. The entity relationship joint extraction method based on global pointer network according to claim 4 is characterized by: Step 5: After steps 1 to 4, the model may extract triples containing incorrect semantic relations; an auxiliary task of relation classification is added, and the [CLS] vector with global semantic information in the encoding vector is input into this module as the sentence vector to identify the potential semantic relations in the sentence and filter out some unreasonable triples in the model extraction results, as shown below: in, Indicates the probability that the current sentence has the kth relationship, W k represents the parameter matrix to be trained, b k represents the bias vector, σ represents the sigmoid activation function; In the prediction stage of the model, if the probability of a certain semantic relationship k If it is less than a given threshold, then the sentence is deemed to not have this kind of semantic relationship k, and all triples with relationship k in the model extraction results are filtered out; The model uses a binary cross entropy loss function to train model parameters as follows: in, represents the true label of the kth relation in the current sentence, and M represents the number of predefined relations.

6. The entity relationship joint extraction method based on global pointer network according to claim 5 is characterized by: Step 6: The joint loss function is obtained by adding the losses in steps 2, 4, and 5, and the parameters of the overall model are learned by minimizing the joint loss, as shown below: Loss=Loss subject +Loss object +Loss rel (15) The Adam algorithm is used to train the model, and the exponential moving average is adopted to ensure the stability of the training process.

Images