Remote supervision long-tail relation extraction method based on constraint graph

Abstract

The invention discloses a remote supervision long-tail relation extraction method based on a constraint graph. The method comprises the following steps: constructing the constraint graph; performing vector representation on the relationship in the constraint graph and the constraint relationship of the entity type; obtaining a sentence set and an entity pair of a to-be-extracted relationship; splicing each word in a word embedding-position embedding-part-of-speech embedding manner to obtain embedded representation of the sentence; obtaining a final feature extraction vector representation of the sentence according to the embedded representation of the sentence; splicing the feature extraction vector representation of the sentence and the two entity type representations to obtain a first vector representation of the sentence; obtaining the entity specific attention of the i word relative to the j entity according to the sentence set of the relation to be extracted and the entity pair; and according to the first vector representation of the sentence, the entity specific attention, the relationship of the fusion constraint graph and the constraint relationship of the entity type, performing relationship prediction on the sentence set of which the relationship is to be extracted. The prediction accuracy of the model can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer natural language processing, and in particular relates to a remote supervised long-tail relationship extraction method based on a constraint graph. Background Art

[0002] Relation Extraction (RE) aims to extract structured knowledge triples from irregular text data, which can prepare for downstream tasks such as knowledge graph construction.

[0003] In the task of relation extraction, the best model currently is based on supervised relation extraction method, but the limitation of this method lies in the quantity and quality of data sets. Many fields do not have public data sets, and text data in different fields have different characteristics. In actual production life, many companies do not want to spend too much money to annotate a large amount of high-quality data.

[0004] To solve the above problems, Mintz et al. proposed the Distant Supervision (DS) technology, the basic assumption of which is: if in a certain knowledge graph, it has been confirmed that there is a relationship R between entity h and entity t, then all sentences containing (h, t) can express the relationship R. This assumption is called the Vanilla assumption. Based on this assumption, given a knowledge graph and a text corpus, a remote supervision dataset can be quickly constructed. Remote supervision uses a knowledge base outside the text consisting of a large number of entity pairs and relationships to annotate the text with relationships. Among them, "remote" can be understood as the use of knowledge graphs outside the text, and "supervision" can be understood as providing relationship labels (i.e., supervision information).

[0005] Remote supervision methods do not require manual annotation, saving time and economic costs. They are a key step in realizing automatic relation extraction from unannotated text corpora, and can effectively solve the problem of extremely scarce training samples in the field of relation extraction. At present, remote supervision technology is widely used in sentiment analysis, part-of-speech tagging, dialogue systems, named entity recognition and other fields, and plays an unprecedented role. The application of remote supervision ideas has successfully achieved low-cost, large-scale rapid annotation, greatly reducing labor costs, making full use of knowledge graphs, and preparing for the further expansion of knowledge graphs, playing an increasingly important role in relation extraction tasks.

[0006] The strong assumption of remote supervision makes it possible to quickly annotate text corpora in large quantities. All entities contained in a sentence can be quickly obtained through named entity recognition technology or text strong matching technology, and then the relationship can be traversed from the knowledge graph based on the entity pairs to quickly expand the data set.

[0007] While distant supervision provides large-scale labeled data for relation extraction, it also brings about the long-tail distribution problem. The original corpus of long-tail relations is small, resulting in insufficient training examples and difficulty in extracting effective features of the relation. Solving the long-tail distribution problem is very important for improving the integrity of the knowledge graph.

[0008] Therefore, it is urgent to develop a relationship extraction method for the long-tail distribution problem. Summary of the invention

[0009] In order to solve one of the above technical problems, the present application provides a remote supervised long-tail relationship extraction method based on a constraint graph, which improves the accuracy of long-tail relationship extraction by incorporating the constraint relationship between entity types and relationships.

[0010] According to one aspect of the present application, a remote supervised long-tail relationship extraction method based on a constraint graph is provided, which includes:

[0011] Constructing a constraint graph, wherein the constraint graph includes constraint relationships between entity types and relationships; the constraint relationships include hot relationships and long-tail relationships;

[0012] Vector representation of the constraints of the relationships and entity types in the constraint graph;

[0013] Get the sentence set S of the relationship to be extracted q ={s1,s2,···,s n} and entity pair e j ,j∈[1,2]; each sentence in the sentence set consists of k words, among which the i-th word is x i ,i∈[1,k];

[0014] Concatenate each word using "word embedding-position embedding-part-of-speech embedding" to obtain the sentence's embedding representation; obtain the sentence's final feature extraction vector representation based on the sentence's embedding representation; concatenate the sentence's feature extraction vector representation and the two entity type representations to obtain the sentence's first vector representation;

[0015] Obtaining entity-specific attention of the i-th word relative to the j-th entity according to the set of sentences and entity pairs of the relation to be extracted; and

[0016] According to the first vector representation of the sentence and the entity-specific attention, the relations of the constraint graph and the constraint relations of the entity type are fused to perform relation prediction on the sentence set of the extracted relations.

[0017] According to the relationship extraction method of at least one embodiment of the present application, the constraint graph G is composed of an edge set ν and a node set ε, and the nodes are connected by directed arrows.

[0018] According to the relation extraction method of at least one embodiment of the present application, the entity-specific attention of the i-th word relative to the j-th entity

[0019] in, is the concatenation of words and entities, A k and r k is the learning parameter; the bilinear operator A k Determine the relationship vector r k The correlation between word embedding and entity embedding.

[0020] According to the relation extraction method of at least one embodiment of the present application, a second vector representation S of the sentence is obtained according to entity-specific attention. ea .

[0021] According to the relationship extraction method of at least one embodiment of the present application, the first vector representation and the second vector representation of the sentence are directly added to obtain a final sentence vector representation, and relationship prediction is performed based on the final sentence vector.

[0022] According to the relationship extraction method of at least one embodiment of the present application, when performing relationship prediction on a sentence set whose relationship is to be extracted, a cross entropy loss function is used as an objective function.

[0023] Technical effect of the present invention: The relationship extraction method of the present application effectively solves the problem of data scarcity caused by long-tail distribution, and shows significant advantages in improving the model's accuracy in predicting long-tail relationships. By using graph convolutional networks and segmented convolutional neural networks, this method enhances the model's ability to understand relationships and improves the accuracy of capturing entity context information and features. The introduction of the entity attention mechanism further optimizes the feature representation of the model, ensuring that the model can focus on the parts of the sentence that are most relevant to the relationship extraction task. Combining all these techniques, this method demonstrates superior performance on the NYT10 dataset, proving its effectiveness in solving the problem of long-tail relationship extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings illustrate exemplary embodiments of the present application and together with the description are used to explain the principles of the present application. These drawings are included to provide a further understanding of the present application, and the drawings are included in and constitute a part of this specification.

[0025] Figure 1 It is a structural diagram of a relationship extraction method according to an embodiment of the present application.

[0026] Figure 2 is a schematic diagram of a constraint graph according to one embodiment of the present application.

[0027] Figure 3 It is a structural block diagram of the overall architecture of the model according to an implementation method of the present application.

[0028] Figure 4 It is a structural block diagram of a sentence encoder according to one embodiment of the present application. DETAILED DESCRIPTION

[0029] The present application is further described in detail below in conjunction with the accompanying drawings and implementation methods. It is to be understood that the specific implementation methods described herein are only used to explain the relevant content, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application are shown in the accompanying drawings.

[0030] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present application can be combined with each other. The technical solution of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present application can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged in addition without departing from the technical concept of the present application.

[0032] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0033] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0034] For descriptive purposes, the present application may use spatially relative terms such as "under", "beneath", "under", "down", "over", "upper", "above", "higher", and "side (e.g., as in "sidewall")" to describe the relationship of one component to another (other) component as shown in the accompanying drawings. The spatially relative terms are intended to encompass different orientations of the device in use, operation, and / or manufacture in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, a component described as "under" or "beneath" other components or features would subsequently be positioned as "over" the other components or features. Thus, the exemplary term "under" can encompass both the "over" and "under" orientations. Furthermore, the device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0035] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0036] Figure 1 It is a flowchart of a relationship extraction method according to one embodiment of the present application.

[0037] like Figure 1 As shown, the present application provides a remote supervised long-tail relationship extraction method based on a constraint graph, which includes: constructing a constraint graph, wherein the constraint graph includes constraint relationships between entity types and relationships; the constraint relationships include hot relationships and long-tail relationships; vectorizing the constraint relationships between the relationships in the constraint graph and the entity types; obtaining a sentence set S of the relationship to be extracted q ={s1,s2,···,s n} and entity pair e j ,j∈[1,2]; each sentence in the sentence set consists of k words, among which the i-th word is x i,i∈[1,k]; concatenate each word with “word embedding-position embedding-part-of-speech embedding” to obtain the embedded representation of the sentence; obtain the final feature extraction vector representation of the sentence based on the embedded representation of the sentence; concatenate the feature extraction vector representation of the sentence and the two entity type representations to obtain the first vector representation of the sentence; obtain the entity-specific attention of the i-th word relative to the j-th entity based on the sentence set of the relationship to be extracted and the entity pair; based on the first vector representation of the sentence and the entity-specific attention, fuse the relationship of the constraint graph and the constraint relationship of the entity type, and perform relationship prediction on the sentence set of the relationship to be extracted.

[0038] The above steps will be described in detail below with reference to the accompanying drawings.

[0039] Figure 2 is a schematic diagram of a constraint graph according to one embodiment of the present application.

[0040] In this application, the constraint graph is a new type of relationship dependency structure, which explicitly models the relationship dependency path by using the constraint information of the relationship and entity type. Figure 2 As shown, each directed edge is from a relationship node to an entity type node or from an entity type node to a relationship node.

[0041] The construction of the constraint graph relies on the knowledge graph ontology structure, discarding irrelevant information such as attributes in the ontology structure and retaining only entity types and relationships. Each edge in the constraint graph is represented in the knowledge graph, and the relationship represented by the edge has at least one example data.

[0042] The constraint graph does not contain example data of entity relationships, but only contains constraint relationships between entity types and relationships, which makes the graph structure simpler and easier to model, and the vector representation of the modeling result is smaller.

[0043] The constraint graph contains a lot of direct prior information, which can alleviate the noise problem caused by the mismatch between entity types and relationship types. The constraint graph contains hot relations and long-tail relations, which are indirectly connected through entity types. When modeling long-tail relations, they are easily affected by hot relations. This knowledge transfer at the vector representation level is conducive to improving the vector representation of long-tail relations and the accuracy of long-tail relation extraction.

[0044] The constraint information between entity types and relations and the potential information contained in words in a sentence are very important for improving the performance of the model in the long-tail relation extraction task.

[0045] Figure 3 It is a structural block diagram of the overall architecture of the model according to an implementation method of the present application.

[0046] In this application, the relationship and entity type can be extracted from the constraint graph through the graph encoder and represented as a vector.

[0047] The constraint graph G consists of an edge set ν and a node set ε, and the nodes are connected by directed arrows. That is, given a constraint graph, the graph encoder first converts each node in it into an embedded vector form, and then uses a graph convolutional neural network (GCN) to extract the vector representation of each relationship node and entity type node in the constraint graph.

[0048] Specifically, first construct an adjacency matrix for each node based on its connection with the edge set The construction process is shown in formula (1):

[0049]

[0050] Among them, (vi,vj) belongs to ε, which means that i and j are connected in the graph; ν i and ν j are two edges in the edge set, and randomly initialize them with a dimension of d v The embedding vector A simple embedding of the constraint graph consists of an embedding matrix and an adjacency matrix

[0051] Graph convolutional neural networks can be used to learn feature representations of graph data and convert graph data into vectors in a low-dimensional vector space. In this way, complex graph data can be converted into an easy-to-process form, making graph data more reliable and interpretable in various applications. In a specific embodiment, a graph convolutional neural network can be used as a graph encoder. Using its neighbor aggregation mechanism, self-loops are added to the edge set.

[0052] The constraint graph embedding representation is used as the input of GCN to obtain its vector representation for the i-th node in the k-th layer, as shown in formula (2):

[0053]

[0054] Among them, W (k) represents the weight matrix of the kth layer, b (k) is the bias vector of the kth layer, and ρ(·) represents the nonlinear function Tanh. The output of GCN is a matrix Each row is an abstract representation of a node, where d h is the dimension of the sentence vector.

[0055] The matrix of the constraint graph includes abstract representations of entity nodes and relationship nodes. (2) Segmentation to obtain relationship representation and entity type representation

[0056] Each node in the constraint may be connected to multiple other nodes, so the vector representation of the node needs to combine the features of all nodes connected to it. In this application, the average of the vector representations of adjacent nodes is used as the vector representation of the node. The data set used in this application contains a special relationship "NA", which means that there is no relationship between two entities. The vector representation of this relationship node is the average of all entity type node representations. Finally, the relationship r i The vector representation R i and vector representations of two entity types and After concatenation, the final vector representation is obtained, as shown in formula (3):

[0057]

[0058] Get the sentence set S of the relationship to be extracted q ={s1,s2,···,s n} and entity pair e j ,j∈[1,2]; each sentence in the sentence set consists of k words, of which the i-th word is x i ,i∈[1,k].

[0059] The sentence encoder can be used to concatenate each word with "word embedding-position embedding-part-of-speech embedding" to obtain the embedded representation of the sentence; the final feature extraction vector representation of the sentence is obtained based on the embedded representation of the sentence; the feature extraction vector representation of the sentence and the two entity type representations are concatenated to obtain the first vector representation of the sentence.

[0060] Figure 4 It is a structural block diagram of a sentence encoder according to one embodiment of the present application.

[0061] like Figure 4 As shown in FIG. 1 , the sentence encoder uses piecewise dilated convolutional neural networks (PDCNN) to encode sentences. The input of the piecewise dilated convolutional neural network model is the sentence embedding representation obtained by concatenating “word embedding-position embedding-part-of-speech embedding”.

[0062] Specifically, firstly, the sentence is divided into three parts according to the positions of the two entities, and the word embedding x for each part is iPerform dilated convolution operation, align and perform maximum pooling operation to reduce the vector dimension, and use tanh function to obtain the final feature extraction vector representation X of the sentence. i .

[0063] The segmented dilated convolutional neural network model divides the sentence into three parts according to the entity position, namely before the head entity, between the head and tail entities, and after the tail entity, and uses dilated convolution to extract semantic information for each part. The calculation method of the dilated convolution kernel size K is shown in formula (4):

[0064] K=k+(k-1)×(rate-1) (4)

[0065] Where k represents the default dilated convolution kernel size, and rate represents the hole size. i After the dilated convolution process, the vector matrix incorporates more feature information, and the matrix size is shown in formula (5):

[0066]

[0067] Among them, W out Represents the size of the word vector matrix output after the dilated convolution, W in Represents word embedding x i The matrix size of , padding indicates the number of 0s added in the padding operation, and stride indicates the step size of each movement of the convolution kernel.

[0068] Since the segmented convolution operation divides the sentence, it is necessary to perform maximum pooling on the three parts dc1, dc2, and dc3 respectively to reduce the vector dimension of the entire sentence, as shown in formula (6):

[0069]

[0070] Where 1≤i≤m, m is the number of convolution kernels. i (1，2，3) is the vector after maximum pooling, and cij is the vector representation of each word.

[0071] After the pooling operation, the dimensions of the three vectors remain consistent. They are concatenated to obtain the vector representation of the entire sentence. Then, the tanh function is used for nonlinear transformation to obtain the final sentence feature extraction vector representation, as shown in formula (7):

[0072]

[0073] Where ρ(·) represents the function tanh, q 1:m From q1 to q m The concatenation vector, X iIt is the final high-dimensional feature vector representation of the sentence.

[0074] Finally, the vector representation of the sentence is X i and two entity types representing and After concatenation, the output of the segmented dilated convolutional neural network model is obtained, that is, the first vector representation of the sentence: as shown in formula (8):

[0075]

[0076] In the present application, the entity-specific attention of the i-th word relative to the j-th entity is obtained through the entity attention module according to the sentence set of the relationship to be extracted and the entity pair; thereby utilizing the additional entity information to help narrow the possibility of the relationship, and by generating attention scores for different entity pair relationships, the attention-weighted word embeddings are pooled using a segmented pooling method to generate a second vector representation of the sentence.

[0077] Specifically, in the remote supervised relation extraction task, the words near the entity contain certain information, which is helpful for the relation extraction task. This additional information limits the possibility of the relationship and is of great help to the remote supervised relation extraction task. The Entity Attention Model (EA) is a deep learning model for handling relation extraction tasks, which can model and understand entities in text, thereby improving the accuracy of text understanding and information extraction.

[0078] In this application, a sentence has k words, where the i-th word is x i ,i∈[1,k], where each is a word embedding, is the embedding representation of the two entities. The entity-specific attention e of the i-th word with respect to the j-th entity i,j The calculation is shown in formula (9):

[0079]

[0080] in, is the concatenation of words and entities, A k and r k is a learning parameter. Bilinear operator A k Determine the relationship vector r k The correlation between word embedding and entity embedding. Intuitively, for a given relationship, more attention should be paid to words related to the entity. i,j Use the softmax function to normalize and generate the attention score a for a given word i,j , as shown in formula (10):

[0081]

[0082] a i,j The weighted word embedding c is obtained by performing an element-wise multiplication with the word embedding of each word. i,j Finally, the segmented pooling method is used to pool the attention-weighted word embedding to generate the second vector representation S of the sentence ea . M is the total number of words.

[0083] In this application, the sentence vector representations generated by the sentence encoder and the entity attention module are added through the classification layer, and the constraint information in the constraint graph is integrated to perform relationship prediction on the sentence bag as a whole.

[0084] Specifically, the sentence encoder and entity attention model obtain the first vector representation G of the sentence respectively. i and the second vector represents S ea , directly add the two vector representations, and fuse the vector feature representations of the PDCNN and EA models to obtain the final sentence vector representation, as shown in formula (11):

[0085] Q i =G i +S ea (11)

[0086] The graph encoder encodes the constraint relationship between entity types and relations into a vector representation. It is necessary to represent the constraint of relation r as C r And the vector representation Q of the sentence i Fusion, to obtain the embedding representation e of relation r in the sentence i The model takes the bag as an input and calculates the attention score of the i-th sentence in the bag to the relation r, as shown in formula (12):

[0087]

[0088] Among them, B is the number of sentences in the bag.

[0089] After obtaining the attention scores of all sentences in the bag for relation r, multiply them with the sentence vector representation to obtain the sample representation of the sentence, and then calculate the weighted sum of all sentences in the bag, as shown in formula (13):

[0090]

[0091] Calculate the vector representation of the sentence bag-to-relation classification result, as shown in formula (14):

[0092] o i =W i z i +b i (14)

[0093] Among them, W i Represents the weight matrix of the relationship category for the sentence example, b i Represents the offset. Then the conditional probability of each relationship is calculated by the softmax function, as shown in formula (15):

[0094]

[0095] Among them, i∈(1,2,···,n r ), M represents the set of all sentences, n r Indicates the number of relationships.

[0096] In order to train the best model for extracting multiple relationship categories, in this application, the objective function uses the cross entropy loss function, which is specifically defined as shown in formula (16):

[0097]

[0098] Among them, θ is the parameter obtained through training, and the parameter gradient calculated by the back propagation algorithm will be used to update the optimization parameter. Among all the predicted relationships, the relationship with the highest probability is calculated using the argmax function as the final prediction result, as shown in formula (17):

[0099] r i =argmaxP(i|M,θ) (17)

[0100] The following is a specific example to further illustrate the relationship extraction method of the present application.

[0101] The dataset used in this application is the NYT10 dataset, which is the most commonly used dataset in the field of remote supervised relationship extraction. The dataset was proposed by Riedel et al. in 2010. Its text comes from the corpus annotated by The New York Times, which contains more than 1.7 million news articles and 4 million relationships between entities, covering different types of entities, including people, organizations, places, jobs, movies, etc. Among them, the relationships involved include common family members, employment, establishment, birthplace, etc., as well as some special relationships, such as the relationship of "appointed as". Named entities are annotated using the Stanford NER tool in combination with the Freebase knowledge base. The relationship between named entity pairs is obtained by linking and referencing the relationship in the external Freebase knowledge base combined with the remote supervision method. The dataset contains a total of 53 relationships, including a special "NA" relationship, which means that the sentence cannot reflect the relationship between the two entities.

[0102] The training and testing of this experiment were run on the Ubuntu 18.04 operating system. The CPU used was an Intel(R) Xeon(R) Platinum 8358P CPU@2.60GHz, the GPU was an NVIDIA GeForce RTX 3090, the Python version was py3.8, and it was based on the Pytorch 1.11.0 version of the deep learning framework, and the CUDA version was 11.6.

[0103] In the graph encoder stage, the pre-trained model Word2Vec is used to initialize word embeddings. If the entity name consists of multiple words, these words will be combined into one word for processing. In the rest of the network, all embedding matrices and weight matrices are initialized using the Xavier method, and the bias vector is initialized to 0 by default.

[0104] The word vector training is carried out at the sentence encoder stage, using the Skip Gram + negative sampling method of the Word2Vec model. To avoid overfitting, the model uses the Dropout strategy before the classification layer. After multiple experimental verifications, the final hyperparameter settings of the model are shown in Table 1.

[0105] Table 1 Model hyperparameter settings

[0106]

[0107] The evaluation index used in this application is the Hits@k index which is the most commonly used in the field of long-tail relationship extraction.

[0108] Hits@k refers to the probability that a true label exists in the top k highest-scoring options of the predicted label for each pair of entities in a subset selected from the test set. (The k values ​​selected in this paper are 10, 15, and 20). The calculation method is shown in formula (18). The larger the value, the better the effect of the long-tail relationship extraction algorithm.

[0109]

[0110] Among them, E represents the instance set, |E| represents the total number of instance sets, and rank i represents the ranking of the true alignment instance of the i-th instance among the candidate instances, I(·) represents the exponential function, and the calculation method is shown in formula (19):

[0111]

[0112] In the experiments in this chapter, relations with less than 100 or 200 training samples are defined as long-tail relations. For comparative experiments, we constructed two subsets from the test set, both of which contain only long-tail relations. All relations in the first test subset have less than 100 training samples, while all relations in the second test subset have less than 200 training samples.

[0113] In order to verify the effect of the CGEA model in the field of long-tail relationship extraction, 9 remote supervision relationship extraction models were selected for comparative experiments with the CGEA model proposed in this paper. The introduction of the relevant models is as follows:

[0114] 1) Mintz: In 2009, Mintz et al. proposed the idea of ​​remote supervision to solve the serious lack of supervised relationship extraction data sets. They used knowledge graphs with relatively complete relationship information to quickly annotate the corpus and perform relationship classification on the constructed data set.

[0115] 2) APCNN: A sentence-level attention-based relation extraction model proposed by Lin et al. in 2016, which uses convolutional neural networks to embed the semantics of sentences and reduces the weight of noisy instances by building sentence-level attention on multiple instances.

[0116] 3) DistantEBL: A method based on explanation learning proposed by Gui et al. in 2016, which can effectively learn relation extraction rules using unlabeled data.

[0117] 4) PCNN+HATT: A method proposed by Han et al. in 2018 that fully utilizes the relational structure information in the ontology and uses an attention mechanism from coarse-grained to fine-grained to select valid examples.

[0118] 5) PCNN+KATT: In 2019, Zhang et al. used the head relation knowledge with richer example data to improve the performance of tail relation extraction, utilized the implicit relation knowledge between category labels from knowledge graph embedding, and used graph convolutional networks to learn explicit relation knowledge, and integrated the relation knowledge into the relation extraction model through a coarse-to-fine knowledge-aware attention mechanism.

[0119] 6) CoRA: In 2018, Jat et al. proposed a collaborative relation-enhanced attention model to simultaneously handle mislabeled and long-tail relations. Based on the relation-enhanced attention network, they introduced cooperative relation features shared between relations in the hierarchy to promote the relation enhancement process and balance the training data of long-tail relations.

[0120] 7) PA-TRP: In 2020, Cao et al. proposed a general method for learning relation prototypes from unannotated text, which promotes long-tail relation extraction by transferring knowledge from relation types with sufficient training data.

[0121] 8) PCNN+HRS: A long-tail relationship extraction model based on path search extracted by Wang et al. in 2021, which regards relationship extraction as a path search task on the relationship hierarchy tree.

[0122] 9) CGRE: In 2021, Liang Tianming first proposed the concept of constraint graph, and applied constraint graph in the field of long-tail relationship extraction, using segmented convolutional neural network to encode sentences and perform relationship extraction.

[0123] This paper reproduces the above model on the NYT10 dataset and compares it with the CGEA model mentioned in this chapter. The comparison results of the Hits@k index of the relation extraction model are shown in Table 2.

[0124] Table 2 Model comparison experimental results

[0125]

[0126] Neither the Mintz nor the APCNN model considers the existence of the long-tail distribution problem, and mainly performs noise reduction on the problem of incorrect labeling, making it difficult for the model to achieve good results in the long-tail relationship extraction task. Although the use of the sentence-level attention mechanism effectively enhances the noise reduction ability of the remote supervision relationship extraction model, it does not solve the problem of the lack of long-tail relationship example sentences in the dataset, and is not very helpful in improving the effect of the long-tail relationship extraction model.

[0127] The PCNN+HATT, PCNN+KATT, PCNN+HRS, PA-TRP and CoRA models are all methods based on relational hierarchical trees. These methods design different structures to enhance the feature representation of long-tail relations based on the characteristics of long-tail distribution. Compared with the denoising model for the problem of incorrect labeling, they have more obvious improvements in effect, indicating that the external knowledge of sentences can help supplement the feature representation of long-tail relations and enhance the effect of the long-tail relationship extraction model.

[0128] Both CGRE and CGEA use the constraints between entity types and relations as external knowledge of sentences. Compared with the relational hierarchical tree, they are significantly ahead of other models in Hits@k. The mismatch between entity types and relation types in the sentence bag causes certain noise problems. The constraint information not only removes the adverse effects of this part of the noise on the extraction results, but also the knowledge transfer of popular relations at the vector representation level can help improve the vector representation of long-tail relations. Experiments have proved that the constraint graph is helpful for the extraction of long-tail relations. Although CGRE uses the constraint graph, the sentence encoder is only composed of PCNN, which has a poor effect on the information extraction of sentences. By using PDCNN and EA models, the CGEA model has more reasonable sentence feature extraction and has a certain improvement in the Hits@k index.

[0129] To further verify the contribution and impact of different components in the CGEA model on the long-tail relation extraction task, this study conducted an ablation experiment on the benchmark model, CGRE, and explored the impact of using PDCNN and the impact of the entity attention mechanism on the model effect. The CGRE model first proposed the concept of constraint graph in the field of remote supervised relation extraction, and used GCN to encode the constraint graph, used PCNN to encode the sentence, and finally used the constraint-aware attention module to combine the results of the graph encoder and the sentence encoder to obtain the final relation extraction result. CGRE+PDCNN means replacing the sentence encoder of CGRE from PCNN to PDCNN. CGRE+EA means integrating the entity attention mechanism on the basis of CGRE. The comparison results of the Hits@k index of the relation extraction model are shown in Table 3.

[0130] Table 3. Comparison results of ablation experiments

[0131]

[0132] CGRE +PDCNN In terms of Hits@10, it is the same as the CGRE model. This is because in the PDCNN model, PCNN has a greater impact on the performance of relation extraction. The segmented convolution operation can well capture the contextual information and features of entities, capture information at different semantic levels, and model the relationship between different entity pairs. However, in terms of Hits@15 and Hits@20, CGRE +PDCNN The model has been improved to a certain extent. This is because the segmented dilated convolutional neural network can capture farther distances than the segmented convolutional neural network, has better robustness to interference factors such as noise, and can perform more stable recognition in complex sentences.

[0133] The CGRE+EA model outperforms the baseline model in all three indicators of Hits@k. Entity attention pays more attention to the word information in the sentence, and assigns different weights according to the influence of different words in the sentence on the final relationship extraction results. It can more accurately understand the relationship between entities, thereby improving the accuracy of the remote supervision long-tail relationship extraction task. Experiments show that the entity attention mechanism is helpful for the long-tail relationship extraction task.

[0134] Compared with CGRE and CGRE, CGEA model +PDCNN The Hits@k index of the CGRE+EA model has been improved, indicating that the PDCNN module and the EA module have positively helped the long-tail relation extraction task. Under the influence of the dilated convolution operation, the DCNN model retains more feature information for the sentence, while the entity attention model can obtain a more reasonable sentence vector representation by giving higher weights to the key words, thereby improving the overall performance of the relation extraction task.

[0135] In summary, the CGEA model can better ensure the accuracy of model prediction compared with other remote supervision long-tail relationship extraction models and meet the requirements of practical applications.

[0136] In order to solve the long-tail distribution problem caused by remote supervision, the remote supervision long-tail relationship extraction method based on constraint graph proposed in this application makes full use of the constraint relationship between different relationships and entity types and the potential information contained in words in sentences. First, the neighbor aggregation mechanism of GCN is used to promote information propagation between different relationship nodes; secondly, PDCNN is used to encode sentences, and the segmented convolution operation can better capture the contextual information and features of entities, and the dilated convolution network can capture more sentence features; then the entity attention mechanism is introduced to obtain the feature representation of the sentence; finally, the sentence encoding of PDCNN and entity attention module is added, and the constraint graph information is integrated to classify and predict the sentence bag. This method has achieved better results on the NYT10 dataset, proving that the constraint graph is helpful to make up for the current situation of lack of long-tail relationship data and improve the accuracy of relationship extraction to a certain extent.

[0137] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0138] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0139] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present application, and are not intended to limit the scope of the present application. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present application.

Claims

1. A remote supervised long-tail relationship extraction method based on constraint graph, characterized in that: include: constructing a constraint graph, wherein the constraint graph includes constraint relationships between entity types and relationships; The constraint relationships include hot relationships and long-tail relationships; Vector representation of the constraints of the relationships and entity types in the constraint graph; Get the sentence set S of the relationship to be extracted q ={s1,s2,···,s n } and entity pair e j ,j∈[1,2]; each sentence in the sentence set consists of k words, among which the i-th word is x i ,i∈[1,k]; Concatenate each word using "word embedding-position embedding-part-of-speech embedding" to obtain the sentence embedding representation; obtain the final feature extraction vector representation of the sentence based on the sentence embedding representation; concatenate the feature extraction vector representation of the sentence and the two entity type representations to obtain the first vector representation of the sentence; Obtaining entity-specific attention of the i-th word relative to the j-th entity according to the set of sentences and entity pairs of the relation to be extracted; and According to the first vector representation of the sentence and the entity-specific attention, the relations of the constraint graph G and the constraint relations of the entity type are fused to perform relation prediction on the sentence set of the extracted relations.

2. The relationship extraction method according to claim 1, characterized in that: The constraint graph G consists of an edge set ν and a node set ε, and the nodes are connected by directed arrows.

3. The relationship extraction method according to claim 1, characterized in that: Entity-specific attention of the i-th word with respect to the j-th entity in, is the concatenation of words and entities, A k and r k is the learning parameter; the bilinear operator A k Determine the relationship vector r k The correlation between word embedding and entity embedding.

4. The relationship extraction method according to claim 3, characterized in that: According to entity-specific attention, we obtain the second vector representation S of the sentence ea .

5. The relationship extraction method according to claim 4, characterized in that: The first vector representation and the second vector representation of the sentence are directly added to obtain the final sentence vector representation, and the relationship prediction is performed based on the final sentence vector.

6. The relationship extraction method according to claim 1, characterized in that: When predicting the relationship of a sentence set to be extracted, the cross entropy loss function is used as the objective function.

Images