A graph convolutional network relation extraction method guided by syntactic information attention

Abstract

The present invention discloses a graph convolutional network relationship extraction method guided by syntactic information attention, comprising the following steps: 1) word vector representation based on BERT pre-training model; 2) context encoding of each word based on bidirectional long short-term memory network to obtain vector representation of each word containing context information; 3) graph convolutional network word node feature representation integrating syntactic dependency type information; 4) relationship extraction through the feature representation of the obtained sentence vector. The present invention takes dependency type information into consideration in the modeling of the relationship extraction model, thereby improving the performance of the relationship extraction model.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, knowledge graphs, relationship extraction, etc., and in particular provides a graph convolutional network relationship extraction method guided by syntactic information attention. Background Art

[0002] A knowledge graph is a semantic network that reveals the relationships between entities. It can formally describe things in the real world and their relationships. It is now used to refer to various large-scale knowledge bases. The basic components of a knowledge graph are (entity, relationship, entity) triples, as well as entities and their related attribute value pairs. Entities are connected to each other through relationships to form a network of knowledge structures, which are widely used in intelligent search, intelligent question and answer, personalized recommendations, etc. Relationship extraction is a basic task in the construction of a knowledge graph. Its main purpose is to extract semantic relationships between entities, that is, to identify entities identified by named entity recognition, and to find semantic connections between entities to identify relationships. For example, "Bill Gates is the founder of Microsoft" contains two entities, Bill Gates and Microsoft, and the relationship between these two entities is founder.

[0003] In recent years, neural network relation extraction methods with powerful encoders (such as CNN, RNN and GCN) have advantages in capturing contextual information without the need for specially designed systems and artificially constructed features, so that relation extraction systems can better understand text and identify the relationship between entities in a given text. LSTM-RNN (Association for Computational Linguistics, 2016) uses an end-to-end neural network model to extract entities and relationships between entities. This method considers both the word order information of the sentence and the substructure information of the dependency syntactic tree. Both parts of information are modeled using bidirectional sequence LSTM-RNN. The two models are stacked so that entity-related information can be used in the process of relation extraction. BRCNN (Association for Computational Linguistics, 2016) uses two LSTMs to encode the word information and dependency information of the shortest dependency path respectively, learns the global information of the shortest dependency path, uses a convolutional layer to connect the two parts, learns the local information between words, and uses a bidirectional structure to learn the forward and backward sentence representation information of SDP respectively, enhancing the ability to classify the direction of entity relations. C-GCN (Association for Computational Linguistics, 2018) proposed a relation extraction model based on graph convolutional networks, which can effectively fuse information on arbitrary dependency structures, and proposed a new path-centric pruning technology to help the model remove irrelevant information to the greatest extent without destroying key information, thereby improving the robustness of the model. AGGCN (Association for Computational Linguistics, 2019) proposed a new attention-guided graph convolutional network to address the fact that the previous rule-based pruning strategy may delete some important information in the entire tree, and introduced a multi-head attention mechanism and dense connections into the GCN model, converting hard pruning to soft pruning.

[0004] The above relation extraction through syntactic information aims to improve relation extraction through better semantic information and analyze the contextual information related to a given entity. However, on the one hand, the current models based on dependency relationships do not take into account the importance of the type of dependency relationship for the relation extraction task and ignore the important grammatical information it contains. On the other hand, some models propose rule-based pruning strategies for syntactic dependency trees to extract relations. These pruning strategies will cause information loss to some extent and fail to achieve a balance between retaining important information and eliminating irrelevant information. Summary of the invention

[0005] In order to solve the problem that existing relation extraction methods ignore grammatical information and information loss, the present invention proposes a graph convolutional network relation extraction method guided by syntactic information attention. This method integrates dependency type information in the process of using graph convolutional neural network to represent the features of different context words in the syntactic dependency tree, and uses the attention mechanism to distinguish the importance of different dependency edges in the syntactic dependency tree for the relation extraction task. In order to solve the technical problem, the present invention provides the following technical solutions:

[0006] A graph convolutional network relation extraction method guided by syntactic information attention includes the following steps:

[0007] 1) Word vector representation based on BERT pre-trained model:

[0008] BERT is a new language model that pre-trains deep bidirectional representations by jointly adjusting the bidirectional Transformers in all layers. The BERT structure is constructed using BERT's BertModel class. The input natural language text sequence is then input into the BERT feature representation layer, and each word in the text is encoded to obtain a vector representation of the word:

[0009] x=(x1,x2,x3…x n ) (1)

[0010] 2) biLSTM context encoding:

[0011] Since the input word vector does not include the contextual information of word order and the graph convolutional network is highly dependent on the correct parse tree structure to extract key information from the sentence, but the tree structure constructed by the existing parsing algorithm is not completely correct in many cases. To address this problem, the present invention first inputs the input word vector into a bidirectional long short-term memory network to generate a contextual representation, which is then used as the input of the graph convolutional network in the model. The calculation is as follows:

[0012] i t =σ(W ix x t +W ih h t-1 +b i ) (2)

[0013] f t =σ(W fx x t +W fh h t-1 +b f ) (3)

[0014] o t =σ(W ox xt +W oh h t- 1+b o ) (4)

[0015]

[0016]

[0017]

[0018] Among them, W and b represent the weight matrix and bias vector connecting the two layers respectively, σ is the sigmoid activation function, is the dot multiplication operation, x t is the input vector, i t , f t and t They are the input gate, forget gate and output gate at time t respectively. represents the state at time t, h t Then it is the output of the hidden layer at time t;

[0019] The input sequence is calculated in order and in reverse order to obtain two different hidden layer representations. and Then, the final hidden layer representation is obtained by vector concatenation, which is the input representation of the graph convolution layer. The formula is as follows:

[0020]

[0021] 3) Graph convolutional network word node feature representation integrating syntactic dependency type information:

[0022] The present invention introduces dependency categories into the graph attention network, uses dependency types with a lot of useful information, and calculates the weights of different dependency edges according to the dependency through the attention mechanism, so that the model can distinguish the importance of different dependency edges;

[0023] 3.1) Constructing a syntactic dependency tree:

[0024] For the input sentence, the Stanford Parser tool is used to perform syntactic dependency analysis on the sentence and generate the syntactic dependency tree corresponding to the sentence;

[0025] 3.2) Construct dependency type matrix:

[0026] First, based on the constructed syntactic dependency tree, an adjacency matrix A = (A ij ) n*n To represent the dependency tree, where A ij Represents word x i and word xj Is there a dependent edge? If so, then A ij =1, if it does not exist, then A ij =0, and then construct the dependency type matrix T = (t i，j ) n*n , where t i，j Represents word x i and word x j The dependency relationship type between them is i，j Mapped to the corresponding vector representation

[0027] 3.3) Dependency edge weight calculation:

[0028] In the lth layer of the graph convolutional network GCN, word x i and word x j The weight of the dependency edge between is calculated as follows:

[0029]

[0030] where · represents the inner product operator, and is word x i and word x j The intermediate vectors are calculated as follows:

[0031]

[0032]

[0033] in, Represents a vector concatenation operation;

[0034] 3.4) Graph Convolutional Network Node Feature Representation:

[0035] Will get As word x i and word x j The weight of the dependency edge between them is obtained by graph convolution operation for each word x i The output representation is calculated as follows:

[0036]

[0037] Among them, σ, W (l) , b (l) Same as standard GCN, Represents word x that incorporates dependency type information j The calculation method is as follows:

[0038]

[0039] in, The role of is to depend on the type vector Become vectors of the same dimension;

[0040] 4) Relationship extraction:

[0041] 4.1) Maximum pooling layer dimensionality reduction:

[0042] Before performing relation extraction, first transform the sentence X = (x1, x2, ..., x n-1 , x n ) Use BERT to embed words and get the vector representation x of each word i , and input it into GCN, and after L layers of graph convolution operations, the feature vector representation corresponding to sentence X is obtained. Then the expression of the whole sentence And the representation of the two entities and Send it to a maximum pooling layer for dimensionality reduction, calculated as follows:

[0043]

[0044]

[0045]

[0046] 4.2) Relationship Prediction:

[0047] The obtained sentence is represented by h x and the representation of the two entities h E1 ,h E2 Through a trainable matrix W R The following processing is performed; finally, a softmax classifier is used to predict the relationship category of entities E1 and E2 in the sentence from the relationship set Y, and the calculation is as follows:

[0048]

[0049]

[0050] 4.3) Minimize the loss function:

[0051] The present invention uses the negative log-likelihood function with L2 regularization term as the loss function:

[0052]

[0053] Among them, m represents the number of samples, t is the one-hot vector of the relationship category, is the output category probability vector of softmax, λ is the regularization parameter, and θ is the training parameter of the relation extraction model.

[0054] The beneficial effects of the present invention are as follows:

[0055] This paper aims at how to accurately and efficiently utilize useful syntactic information in syntactic dependency trees while eliminating irrelevant information, and proposes a method for fusing syntactic information for relation extraction using an attention graph convolutional network. In this method, an attention mechanism based on a graph convolutional network is applied to different context words in a dependency tree obtained from an existing dependency parser to distinguish the importance of different word dependencies. Considering that the dependency type between words also contains important information, this paper includes dependency type information in the modeling of the relation extraction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a text instance of the present invention and its syntactic dependency tree;

[0057] Figure 2 It is the graph convolutional network relationship extraction framework of the present invention. DETAILED DESCRIPTION

[0058] by Figure 1 Take the sentence as an example and combine it with Figure 2 The present invention is further described.

[0059] A graph convolutional network relationship extraction method guided by syntactic information attention includes the following steps:

[0060] 1) Word vector representation based on BERT pre-trained language model:

[0061] Input the text sequence “James decided to join the Lakers” into the BERT pre-training layer and get the vector representation of each character: x = (x1, x2, ... x6), such as Figure 1 As shown;

[0062] 2) biLSTM context encoding:

[0063] according to Figure 2 , biLSTM receives the text sequence output from the BERT pre-trained language model, inputs it into two biLSTM units, and randomly initializes the hidden state; the vector x is input into two bidirectional LSTM units, and the forward and reverse hidden layer outputs h corresponding to each character are calculated by formula (2)-expression (7): i前 and h i后 , output h of the forward and reverse hidden layers of each character i前 ,h i后, and then concatenated into the final hidden layer representation through formula (8) That is, the input representation of the graph convolutional layer;

[0064] 3) Graph convolutional network word node feature representation integrating syntactic dependency type information:

[0065] 3.1) Constructing a syntactic dependency tree:

[0066] For the input sentence, syntactic dependency analysis is performed using the Stanford Parser tool to generate a syntactic dependency tree.

[0067] 3.2) Construct dependency type matrix:

[0068] According to the obtained syntactic dependency tree, a dependency type matrix T = (t i,，j ) n*n , where t i,j Represents word x i and word x j The type of dependency between Figure 1 (e.g. nsubj); then for each relationship type t in the matrix T i,j Mapped to the corresponding vector representation

[0069] 3.3) Dependency edge weight calculation:

[0070] In the first layer of GCN, the word x is calculated by formula (9) i and word x j The weight of the dependency edge between Using formulas (10) and (11), we can get word x i and word x j The middle vector and

[0071] 3.4) Graph Convolutional Network Node Feature Representation:

[0072] The result obtained in 3.3 As word x i and word x j The weight of the dependency edge between them is passed through the graph convolutional network layer and the formula (12) is used to obtain each word x i The output representation

[0073] 4) Relationship extraction:

[0074] 4.1) Maximum pooling layer dimensionality reduction:

[0075] The feature vector corresponding to the sentence X obtained after L layers of graph convolution operation is represented as And the representation of the two entities and Send it to a maximum pooling layer for dimensionality reduction, and obtain the sentence representation h after dimensionality reduction through formula (14)-expression (16) x and the representation of the two entities h E1 ,h E2 ;

[0076] 4.2) Relationship Prediction:

[0077] Finally, according to formula (17), a trainable matrix W is obtained R After processing, a softmax classifier is used to predict the relationship category r = "member-collection" of entities E1 and E2 in the sentence from the relationship set Y according to formula (18);

[0078] 4.3) Minimize the loss function:

[0079] According to the loss function of formula (19), during the training process, the model will be continuously updated and the value of the loss function will be minimized by the stochastic gradient descent method until convergence;

[0080] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A graph convolutional network relationship extraction method guided by syntactic information attention, characterized in that: The following steps are involved: 1) Word vector representation based on BERT pre-trained model: BERT pre-trains deep bidirectional representations by jointly adjusting the bidirectional Transformers in all layers, constructs the BERT structure using BERT's BertModel class, and then inputs the input natural language text sequence into the BERT feature representation layer, encoding each word in the text to obtain a vector representation of the word: x=(x1,x2,x3…x n ) (1) 2) biLSTM context encoding: The input word vector is first fed into a bidirectional long short-term memory network to generate a contextual representation, which is then used as the input of the graph convolutional network in the model. The calculation is as follows: i t =σ(W ix x t +W ih h t-1 +b i ) (2) f t =σ(W fx x t +W fh h t-1 +b f ) (3) o t =σ(W ox x t +W oh h t-1 +b o ) (4) Among them, W and b represent the weight matrix and bias vector connecting the two layers respectively, σ is the sigmoid activation function, is the dot multiplication operation, x t is the input vector, i t , f t and t They are the input gate, forget gate and output gate at time t respectively. represents the state at time t, h t Then it is the output of the hidden layer at time t; The input sequence is calculated in order and in reverse order to obtain two different hidden layer representations. and Then, the final hidden layer representation is obtained by vector concatenation, which is the input representation of the graph convolution layer. The formula is as follows: 3) Graph convolutional network word node feature representation integrating syntactic dependency type information: Introducing dependency categories into the graph attention network, using dependency types with a lot of useful information, and using the attention mechanism to calculate the weights of different dependency edges based on the dependencies, so that the model can distinguish the importance of different dependency edges; 3.1) Constructing a syntactic dependency tree: For the input sentence, the Stanford Parser tool is used to perform syntactic dependency analysis on the sentence and generate the syntactic dependency tree corresponding to the sentence; 3.2) Construct dependency type matrix: First, based on the constructed syntactic dependency tree, an adjacency matrix A = (A ij ) n*n To represent the dependency tree, where A ij Represents word x i and word x j Is there a dependent edge? If so, then A ij =1, if it does not exist, then A ij =0, and then construct the dependency type matrix T = (t i,j ) n*n , where t i,j Represents word x i and word x j The dependency relationship type between them is i,j Mapped to the corresponding vector representation 3.3) Dependency edge weight calculation: In the lth layer of the graph convolutional network GCN, word x i and word x j The weight of the dependency edge between is calculated as follows: where · represents the inner product operator, and is word x i and word x j The intermediate vectors are calculated as follows: in, represents a vector concatenation operation; and Represents l-1 layer word x respectively i and word x j The output representation of 3.4) Graph Convolutional Network Node Feature Representation: Will get As word x i and word x j The weight of the dependency edge between them is obtained by graph convolution operation for each word x i The output representation is calculated as follows: Among them, in the lth layer of graph convolution, W (l) is the weight matrix, b (l) is a bias vector, σ is a ReLU activation function, Represents word x that incorporates dependency type information j The calculation method is as follows: in, The role of is to depend on the type vector Become vectors of the same dimension; 4) Relationship extraction: 4.1) Maximum pooling layer dimensionality reduction: Before performing relation extraction, first transform the sentence X = (x1, x2, ..., x n-1 , x n ) Use BERT to embed words and get the vector representation x of each word i , and input it into GCN, and after L layers of graph convolution operations, the feature vector representation corresponding to sentence X is obtained. Then the expression of the whole sentence And the representation of the two entities and Send it to a maximum pooling layer for dimensionality reduction, calculated as follows: Among them, E1 and E2 represent two entities whose relations need to be extracted; 4.2) Relationship Prediction: The obtained sentence is represented by h x and the representation of the two entities h E1 ,h E2 Through a trainable matrix W R The following processing is performed; finally, a softmax classifier is used to predict the relationship category of entities E1 and E2 in the sentence from the relationship set Y, and the calculation is as follows: where b R represents a bias vector; 4.3) Minimize the loss function: The negative log-likelihood function with L2 regularization is used as the loss function: Among them, the L2 regularization term measures the absolute value of each coefficient, and adds it to the loss function as a penalty term to force the coefficients of the optimal solution to be close to 0. m represents the number of samples, and t is the one-hot vector of the relationship category. is the output category probability vector of softmax, λ is the regularization parameter, and θ is the training parameter of the relation extraction model.

Images