Entity relationship extraction method fusing semantic information and syntactic information

Abstract

The invention discloses an entity relationship extraction method fusing semantic information and syntactic information, and belongs to the technical field of language processing. According to the method, the information in the sentences is learned, so that the fusion of semantic and syntactic information is realized, and the accuracy of relationship classification between entities is improved. According to the method, an external syntax device and word vector information are combined, the influence of noise in a text is reduced by using a graph convolutional neural network of a self-attention pooling layer and a residual shrinkage network, and long-distance dependence between entities is improved. Meanwhile, loss is calculated in an interpolation prediction mode, and interaction between the BERT and the graph neural network is perfected. Experiments show that semantic vectors and syntactic structure information are fully fused, so that the relation extraction effect is remarkably improved compared with other reference models, and the problems that the recognition rate of the long-distance relation between entities in sentences is low and noise exists are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of language processing, and in particular relates to an entity relationship extraction method integrating semantic and syntactic information. Background Art

[0002] At present, the task of relation extraction is to classify the relationships between many entities in the text, so as to effectively mine the information between the texts and provide support for downstream tasks such as graph construction, question-answering systems, and search engines. However, general extraction algorithms are greatly affected by the distance between entities and it is difficult to fully mine the long-distance dependency information between entities. Although graph-based relation extraction can provide additional information for entity vectors, it is too dependent on the data structure constructed by external grammar generators, which will be mixed with noise that is not conducive to the interaction between entities, thus affecting the effect of relation extraction and affecting downstream tasks.

[0003] Therefore, a new technical solution is urgently needed in the prior art to solve this problem. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an entity relationship extraction method that integrates semantic and syntactic information to solve the technical problem that the existing extraction algorithm is greatly affected by the distance between entities, noise, etc., and the accuracy of relationship extraction is poor.

[0005] A method for extracting entity relations by integrating semantic and syntactic information comprises the following steps, which are performed in sequence:

[0006] Step 1: Establish a BERT encoding layer. The text information is encoded through the BERT encoding layer and converted into word vector information corresponding to the entity;

[0007] Step 2: Establish a message propagation module, which includes a graph neural network GCN, a graph convolution module SAG based on a self-attention pooling layer, and a residual shrinkage module. The graph neural network GCN uses an external syntax parser to construct a dependency syntax graph. The graph structure accepts the word vector information encoded by BERT, and then obtains the vector through the message propagation module and concatenates it with the entity vector encoded by BERT.

[0008] Step 3: Pass the concatenated vectors from step 2 through the decoding layer to generate a label matrix of the predicted entity, and obtain the relationship between the predicted entity pairs;

[0009] Step 4: Introduce the loss function, calculate the loss value of the relationship between specified entities through the interpolation prediction method, introduce the entity type to limit the classification of the relationship between entities, use the pre-defined entity type relationship, and specify the classification vector between irrelevant entities. Set up the mask mechanism to calculate and obtain the loss value of the relationship between specified entities, and obtain the entity relationship that integrates semantic and syntactic information.

[0010] The specific working steps of the BERT encoding layer in step 1 are:

[0011] The text information is passed through the encoder of the BERT encoding layer to obtain a sequence encoding with a digital expression of the text, and then passes through the multi-layer Transformer structure of the BERT encoding layer, using a multi-head self-attention mechanism and a forward propagation module to learn and obtain the vector representation of each word; the self-attention mechanism uses three learnable parameter matrices to obtain the corresponding query matrix Q, key matrix K and value matrix V from the input sequence L, as shown in formula (1):

[0012] Q=L·W Q , K=L·W K , V=L·W V (1);

[0013] Among them, W Q , W K , W V Corresponding to the learnable parameters of the three matrices;

[0014] Calculate the attention score of each head in the value vector and multiply it with the value matrix V to get a vector output representation of the head; after calculating multiple heads and splicing them together, output the vector representation of the sequence:

[0015]

[0016] Among them, A i represents the output vector of the i-th head; T represents the transpose operation; d k Indicates the length of the sequence; Q i represents the query matrix in the i-th head; K i represents the key matrix in the i-th head; V i represents the i-th head value matrix;

[0017] After obtaining the vectors of the text information, we use the text position {i,j}∈(0,d k ) to get the corresponding entity vector, and finally after the average operation, the entity vector expression is obtained:

[0018] e k =mean([M;<i,j> ]) (3);

[0019] Among them, e k Represents the entity vector at position {i,j} and index k; i and j represent the head and tail positions of the entity respectively; mean represents the average of the specified dimension of the matrix; M represents the sentence vector.

[0020] The following methods are used in the dependency grammar graph in step 2 to reduce the error propagation caused by the external grammar parser: first, entities are predefined in the model of the dependency grammar graph using a dictionary method, and the entities are set as high-frequency items so that more attention can be paid to the predefined information during word segmentation; second, a graph convolution module SAG based on a self-attention pooling layer is used to combine the attention mechanism with the convolution operation; finally, a residual shrinkage module is added to perform soft threshold processing on the internal vector of the node using a learned threshold.

[0021] The graph convolution module SAG based on the attention pooling layer uses the neighbor information of the node to update the node representation. The representation of each node is obtained by fusing the features of its neighbor nodes. The fused node features are expressed as follows:

[0022]

[0023] Among them, H l+1 ∈R n×d Represents the node vector of the l+1th layer; for The degree matrix of l is the weight matrix of the lth layer; represents the sum of the adjacency matrix and the identity matrix; H l A node vector representing layer l.

[0024] The specific method of combining the attention mechanism with the convolution operation is:

[0025] First, each node vector generated by the BERT encoding layer is averaged to obtain the input vector. Then, the node representation and adjacency matrix of the dependency grammar graph are input into the graph convolutional network and multiplied with the learnable attention parameter matrix M to obtain the node importance matrix that integrates node information and structural information, as shown in formula (5):

[0026]

[0027] A l =A l +I

[0028] Among them, tanh activation function; A l is the adjacency matrix of the graph layer l; I is the identity matrix, I∈{0,1} n×n ;M l+1 is the attention matrix, M l+1 ∈R d×1 ; rank is the ranking of node importance coefficients, rank∈R n×1 ;

[0029] In order to discard irrelevant node features and increase the feature representation of entity nodes, the rows and columns of the adjacency matrix A where the nodes whose rank sequence values ​​are less than the set threshold are located are set to 0, so as to achieve the purpose of not affecting the node information in the next layer of global message transmission process, as shown in formula (6):

[0030] id=top_rank(rank,[k·n]) (6);

[0031] A l+1 =change(A l ,id)

[0032] Among them, top_rank is the ranking function; k is the ratio coefficient of the current layer nodes to the number of nodes in the previous layer; n is the number of nodes; change is the adjacency matrix transformation function; A l+1 Represents the adjacency matrix of the next layer of graph propagation; finally, the next layer output after the graph convolution module is expressed as:

[0033]

[0034] The decoding steps of the decoding layer in step 3 are:

[0035] First, the node vector of the entity is found in the dependency syntactic graph using the id index, and then concatenated with the word vector information of the BERT encoding layer to obtain the final entity vector. Secondly, the feature information between entities is obtained by concatenating entities in pairs, and the linear layer is used to map the feature information between entities to the relationship classification. The specific formula is as follows:

[0036]

[0037] Among them, e a, b represents the entity feature vector with index a,b; g a,b and h a,b Respectively represent the node vector and word vector in the graph; [;] represents the concatenation between vectors; M a,b Represents the classification value inside the decoding matrix; W and b are the parameters of the linear layer, W∈R d×n , b∈R d×n .

[0038] The loss function in step 4 is obtained by setting a mask mechanism, as shown in formula (10):

[0039] loss = w·CEL GCN (p1,l)+(1-w)CEL BERT (p2,l) (10);

[0040] Among them, w is the weight coefficient; CEL represents the cross entropy loss function CrossEntropy; p1 and p2 represent the predicted values ​​output by the encoding layer of the graph neural network GCN and BERT respectively; l is the true value label;

[0041] in,

[0042] CEL GCN (p1,l)=CEL(p1·id_mask,l·id_mask)

[0043] id_mask=R<entity i ,entity j > (11);

[0044] In the formula, id_mask is the mask between entities; R is the function to determine whether it is a predefined entity type; entity i is the entity i category; j is the entity category j.

[0045] Through the above design scheme, the present invention can bring the following beneficial effects:

[0046] The present invention proposes an entity relationship extraction method based on graph neural network. By learning the information in the sentence, it realizes the fusion of semantic and syntactic information and improves the accuracy of entity relationship classification. The present invention mainly combines the external grammar device with the word vector information, uses the graph convolutional neural network with the self-attention pooling layer and the residual shrinkage network to reduce the influence of noise in the text and improve the long-distance dependency between entities. At the same time, the loss is calculated by interpolation prediction to improve the interaction between BERT and graph neural network. Experiments show that the present invention fully integrates the semantic vector and syntactic structure information, so that the relationship extraction effect is significantly improved compared with other benchmark models, and the low recognition rate of long-distance relationships between entities in sentences and the noise contained in them are alleviated.

[0047] This paper demonstrates the potential of improving relation extraction by combining the BERT encoding layer with graph neural networks and fusing semantic and syntactic information, providing valuable reference and direction for future research. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0049] Figure 1 A flowchart of an entity relationship extraction method integrating semantic and syntactic information according to the present invention;

[0050] Figure 2 A working principle diagram of a message propagation module in an entity relationship extraction method integrating semantic and syntactic information of the present invention;

[0051] Figure 3 This is an example diagram of constructing a dependency grammar graph in an entity relationship extraction method that integrates semantic and syntactic information according to the present invention;

[0052] Figure 4 A working principle diagram of a graph convolution module SAG based on a self-attention pooling layer in an entity relationship extraction method that integrates semantic and syntactic information in the present invention;

[0053] Figure 5 A working principle diagram of a residual shrinkage module in an entity relationship extraction method integrating semantic and syntactic information according to the present invention;

[0054] Figure 6 A comparison result diagram of the relationship extraction capabilities of an entity relationship extraction method that integrates semantic and syntactic information under different data sets in the present invention;

[0055] Figure 7 A comparison diagram of confusion matrices generated in an embodiment of an entity relationship extraction method integrating semantic and syntactic information of the present invention;

[0056] Figure 8 This is a comparison chart of three indicators under different sentence lengths under two data sets in an embodiment of an entity relationship extraction method that integrates semantic and syntactic information of the present invention. DETAILED DESCRIPTION

[0057] The entity relationship extraction method integrating semantic and syntactic information disclosed in the present invention mainly includes three parts: BERT encoding layer, message propagation module and decoding layer. Figure 1 As shown in the figure. First, the text information is encoded through the BERT encoding layer and converted into the corresponding word vector information. Then, an external syntax parser is used to build a dependency syntax graph. The graph structure accepts the word segmentation vector from the BERT encoding, obtains the vector through a specific message passing mechanism, and then concatenates it with the entity vector encoded by BERT. Finally, the label matrix of the predicted entity is generated through the decoding layer.

[0058] 1. BERT-based entity encoding layer

[0059] The purpose of the entity encoding layer is to obtain the vector representation corresponding to the entity so as to calculate the relationship classification between entities later. n}, and the corresponding entity vector representation E = {e1, e2, e3, ..., e kSpecifically, the text is encoded by the encoder part of BERT to obtain a sequence with a digital expression of the text, and then passes through a multi-layer Transformer structure, using a multi-head self-attention mechanism and forward propagation modules to learn the vector representation of each word. The self-attention mechanism can be expressed as follows: for the input sequence L, three learnable parameter matrices are used to obtain the corresponding Q (query), K (key), and V (value) matrices, as shown in formula (1):

[0060] Q=L·W Q , K=L·W K , V=L·W V (1)

[0061] Among them, W Q , W K , W V Corresponding to the learnable parameters of the three matrices.

[0062] In addition, the attention score of each head in the value vector is calculated and multiplied with the value matrix V to obtain a vector output representation of the head. Similarly, after calculations on multiple heads, they are finally spliced ​​together to output the vector representation of the sequence. Formula (2) represents the output vector of the i-th head.

[0063]

[0064] Where T represents the transposition operation; d k Indicates the length of the sequence;

[0065] After obtaining the text vector, we use the text position {i,j}∈(0,d k ) to get the corresponding entity vector, and finally after the average operation, the entity vector expression can be obtained. Formula (3) represents the entity e at position {i, j} k vector.

[0066] e k =mean([M;<i,j> ]) (3)

[0067] Among them, e k Represents the entity vector with index k; i and j represent the head and tail positions of the entity respectively; mean represents the average of the specified dimension of the matrix.

[0068] 2. Message propagation module based on dependency syntactic graph

[0069] Compared with the traditional vector extraction method, constructing a dependency syntax graph can greatly reflect the internal structure and grammatical information of the sentence. However, some irrelevant nodes, such as "," and ".", will introduce irrelevant information between entities during the message passing process of the graph neural network, making it difficult for the entity nodes to learn key features. In order to reduce the error propagation problem caused by relying on external grammar generators, the present invention proposes three methods to mitigate this impact. First, use a dictionary method to predefine entities in the model, and set the entities as high-frequency items so that more attention can be paid to the pre-defined information during word segmentation; secondly, Figure 3 As shown in the figure, a graph convolution module SAG based on a self-attention pooling layer is used to combine the attention mechanism with the convolution operation; finally, a residual shrinkage module RS is added to perform soft threshold processing on the internal vector of the node using a learnable threshold. Figure 3 As shown, the node vector in the figure is finally obtained.

[0070] (1) Dependency grammar graph construction

[0071] The grammatical dependency graph represents the dependency relationship between words in a natural language sentence. Each word segmentation result is represented as a node and connected through directed edges (such as subject-predicate relationship, verb-object relationship, etc.). As a graphical structure, the dependency graph contains rich semantic and grammatical structure information. When extracting n-ary relations in a sentence, introducing such features can effectively utilize structural information and provide more entity-level semantic relations.

[0072] The construction of the dependency grammar graph is completed by an external dependency parser. These parsers automatically infer the dependency relationship between words based on the grammatical rules and contextual information between words in a sentence, thereby generating a corresponding dependency grammar graph. The present invention uses an external Chinese grammar parser pyLTP and an English grammarizer spacy model to construct a sentence dependency grammar graph. An example of sentence dependency grammar analysis is as follows Figure 3 shown.

[0073] A piece of input text is first segmented by LTP, the part-of-speech tagging is performed on the segmentation results, and then the dependency syntax analysis is performed to obtain the result shown in the figure above. The above figure mainly includes three parts: ROOT node, segmentation result, and dependency relationship. When assigning an embedding vector to a node, the average vector of the corresponding node is obtained from the output vector of BERT by marking the position of the node, which is used as the embedding vector of each node, and the dependency relationship is used as the directed edge of the dependency syntax graph. In addition, the vector of the ROOT node is used as the CLS vector in the BERT output vector, and the dependency syntax graph is finally generated.

[0074] (2) Graph convolution module based on self-attention pooling layer

[0075] As a deep learning model based on graph structure, the graph neural network GCN can preserve the topological structure and node characteristics in the graph and comprehensively consider the node and structure information. The core idea of ​​GCN is to use the node's neighbor information to update the node representation. Each node representation is obtained by fusing the features of its neighbor nodes. The fused node features are shown in the following formula (4):

[0076]

[0077] Among them, H l+1 ∈R n×d Represents the node vector of the l+1th layer; for The degree matrix of l is the weight matrix of the lth layer; represents the sum of the adjacency matrix and the identity matrix; H l A node vector representing layer l.

[0078] In order to guide the model to remove the propagation of irrelevant node information, the present invention uses a self-attention graph convolution module to assign a weight value to each node to achieve the purpose of removing noise. Figure 4 As shown in , each node vector generated by the BERT encoding layer is first averaged to obtain the input vector of the module. Then, the node representation and adjacency matrix of the grammatical dependency graph are input into the graph neural network and multiplied with the learnable attention parameter matrix M to obtain the node importance matrix that integrates node information and structural information. As shown in formula (5),

[0079]

[0080]

[0081] Among them, tanh activation function; A l is the adjacency matrix of the graph layer l; I is the identity matrix, I∈{0,1} n×n ;M l+1 is the attention matrix, M l+1 ∈R d×1 ; rank is the ranking of node importance coefficients, rank∈R n×1 .

[0082] In order to discard irrelevant node features and increase the feature representation of entity nodes, the rows and columns of the adjacency matrix A where the nodes with smaller values ​​in the rank sequence are located are set to 0, so as to achieve the purpose of not affecting the node information when the next layer of global message transmission is achieved. Specifically, as shown in formula (6):

[0083] id=top_rank(rank,[k·n]) (6)

[0084] A l+1 =change(A l ,id)

[0085] Among them, top_rank is the ranking function; k is the ratio coefficient of the current layer nodes to the number of nodes in the previous layer; n is the number of nodes; change is the adjacency matrix transformation function; A l+1 Represents the adjacency matrix for the next layer of graph propagation.

[0086] Finally, the next layer output after the graph convolution module can be expressed as:

[0087]

[0088] (3) Residual shrinkage module

[0089] After graph convolution, self-attention and sorting operations, the graph structure after each layer of filtering can be obtained. In order to further reduce the influence of irrelevant node noise during graph message transmission, the present invention introduces a residual shrinkage module RS to perform soft threshold processing on the internal vector of each node, as shown in Figure (5).

[0090] First, the absolute value of the node vector is processed to make the feature tend to be positive, and then the average value of each node vector is obtained after the global average pooling operation. Secondly, the threshold features are learned using two fully connected layers and the corresponding activation functions, and mapped to the corresponding dimension vector to obtain the threshold feature matrix x, which can be specifically expressed as:

[0091] x=sigmod(W2(relu(W1(Pooling(|H l |)+b1)+b2));

[0092] Among them, H l is the vector of the node after convolution, H l ∈R n×d ; Pooling is the global average pooling function; W a and b a Represent the learnable parameter matrix and bias of the linear layer, W a ∈R 1×1 , b a ∈R 1×1 ; sigmod is the activation function that maps nodes to between 0 and 1;

[0093] In order to prevent the problem of too large a threshold, it is necessary to compare x with the absolute value of H l The corresponding positions of the vectors are multiplied to obtain the final set of thresholds δ. Then the original node vector is soft-thresholded, as shown in formula (8):

[0094]

[0095] Among them, h l and Respectively represent H l Internally update each node vector before and after;

[0096] The following pseudo code shows an example of a batch-based message propagation process based on a dependency syntax graph:

[0097]

[0098] 3. Decoding layer

[0099] For each entity pair (e a ,e b ), we need to use a linear combination of features to predict the relationship between entity pairs. First, we use the id index to find the node vector of the entity in the dependency syntactic graph, and concatenate it with the BERT word vector information to get the final entity vector. Secondly, we use the pairwise concatenation method to obtain the feature information between entities, and use the linear layer to map the feature information between entities to the relationship classification. The specific calculation method is as follows:

[0100]

[0101] Among them, e a,b represents the entity feature vector with index a and b; g a,b and h a,b Represents the node vector and word vector in the graph; [;] represents the concatenation between vectors; M a,b Represents the classification value inside the decoding matrix; W∈R d×n and b∈R d×n are the parameters of the linear layer.

[0102] 4. Loss Function

[0103] Due to the different structures of the BERT model and the graph neural network, the output dimension of the BERT model is too large, resulting in the inability to update the pre-trained model parameters in a timely manner during back propagation, and the inability to directly combine them to make the model converge quickly

[30] . Therefore, the present invention uses a predictive interpolation method to update the pre-trained model in real time. Specifically, the vector output by the BERT model is divided into two parts, one as the input of the graph neural network, and the other for separate prediction. The weights between the two parts are connected by a weight coefficient, as shown in formula (10):

[0104] loss = w·CEL GCN (p1,l)+(1-w)CEL BERT (p2,l) (10);

[0105] Among them, w is the weight coefficient; CEL represents the cross entropy loss function; p1 and p2 represent the predicted values ​​​​of GCN and BERT output respectively; l is the true value label.

[0106] At the same time, the model has an imbalance of positive and negative samples when outputting the label matrix, that is, there are few relationships between text entities, resulting in a large label matrix, but a small number of internal positive samples. To address this phenomenon, the types of entities are introduced to limit the classification of relationships between entities. By pre-defining the entity type relationship, setting a mask mechanism allows the model to calculate the loss value of the relationship between specified entities and specify the classification vector between irrelevant entities, as shown in formula (11):

[0107] id_mask=R<entity i ,entity j > (11);

[0108] CEL GCN (p1,l)=CEL(p1·id_mask,l·id_mask)

[0109] Among them, id_mask is the mask between entities; R is the function to determine the pre-entity type; entity i is the entity i category; j is the entity category j.

[0110] Example:

[0111] The experiment uses two datasets for verification. 1) Baidu DUIE 2.0 relation extraction dataset, which includes more than 30,000 text data, 130,000 triple data, and 31 predetermined relation types, including training set, test set, and validation set. Unknown means that there is no relationship between entities. 2) SemEval2010 task8 dataset, with 8,000 and 2,717 data in the training set and test set respectively, including 10 relation types, among which Other means that there is no relationship between entities.

[0112] (1) Baseline comparison

[0113] In order to verify the effectiveness of the method proposed in this invention, the selected baseline methods are compared and introduced as follows:

[0114] 1) R-BERT: Directly use the position index of the entity in the sentence, use BERT to extract the entity vector and CLS vector aggregation for relationship classification.

[0115] 2) BERT-LSTM: Based on BERT, the LSTM sequence characteristics are used to extract deeper features and aggregate vector information for classification.

[0116] 3) BERT-GAT: It builds a graph based on entities, uses attention to assign different weights to the edges between entities, then uses graph propagation to update the entity vector, and finally decodes the relationship between two entities.

[0117] 4) GP-GNN: Use GloVe to extract vectors and combine them with the location information to obtain entity vectors using the LSTM model. It builds a fully connected graph structure about the entities and predicts the relationship classification between entities.

[0118] 5) AGGCN: The original dependency tree is converted into a fully connected weighted graph. Based on the graph convolutional neural network, the correlation strength between nodes is used to learn the node representation, and finally the result between entities is obtained.

[0119] Based on the above method, the present invention tests the model's ability to extract relations under different data sets through F1, precision, and recall curves, as follows Figure 6 As shown:

[0120] Experimental results show that the performance of the proposed model exceeds that of most models in the relationship extraction task, and the overall performance reaches 93.28% and 88.41% in the DUIE 2.0 and SemEval2010 task8 datasets respectively, which proves the effectiveness of the proposed model.

[0121] exist Figure 6 Models such as BERT-BERT and BERT-LSTM are all encoded using the BERT pre-training model, and the encoded vectors are identified through different operations, but a single word vector is difficult to fully understand the meaning between entities and omits the analysis of other features. BERT-GAT and others use the training method of graph neural networks, focusing on the interaction between entities after encoding, but lack the guidance of the grammatical and semantic structure of the sentence itself. The model of the present invention is based on BERT encoding, and uses a variety of methods to fully integrate the dependency syntactic graph information, thereby improving the performance of relationship extraction.

[0122] (2) Noise situation analysis

[0123] The method proposed in the present invention is mainly based on constructing a dependency syntactic graph to reduce the propagation of irrelevant node noise. In order to more intuitively demonstrate the impact of adding the graph convolution module SAG based on the self-attention pooling layer and the residual shrinkage module RS on the model denoising effect, we perform denoising visualization in the SemEval2010 task8 dataset. First, the neural network is trained without reducing noise to ignore the impact of noise on the model. Then, the graph convolution module SAG based on the self-attention pooling layer and the residual shrinkage module RS are added in turn, and the impact of the modules on the overall performance is more intuitively observed by generating a confusion matrix.

[0124] like Figure 7 As shown in the figure, the diagonal elements in the matrix represent the correctly predicted relationship categories, and the lighter the color of the other irrelevant elements, the stronger the noise. Compared with the other two confusion matrices, the graph neural network GCN has more noise outside the diagonal, while GCN-SAG-RS has the least noise. This shows that the use of the SAG module learns the importance score of each node through the attention mechanism and removes useless nodes; the use of the residual shrinkage module RS learns the soft threshold through the residual shrinkage mechanism and updates the node vector. These two methods significantly improve the performance of the model.

[0125] (3) The impact of different text lengths on dependency relationships

[0126] In order to intuitively see the improvement of the model proposed by the present invention in the dependency relationship between entities, as shown in Figure 8 As shown, the BERT without graph neural network is compared with the text model under different sentence lengths in two data sets. In DUIE, it can be seen that the indicators of sentence length in the interval (0, 50) are not much different, the indicators of sentence length in the interval (50, 100) increase slightly, and in the interval (100-256), the three indicators F1, P, and R increase by 1.34%, 1.7%, and 0.98% respectively. In the SemEval 2010 data set, the F1 indicator is significantly higher than the former model in the three intervals of sentence length, increasing by 0.19%, 1.11%, and 2.17% respectively. That is, the longer the text length, the more obvious the performance improvement of the model of the present invention in the relationship extraction task, which shows that the introduction of graph neural network on the basis of dependency syntactic graph can increase the dependency relationship between entities and improve the relationship extraction effect of the model.

[0127] (3) Ablation experiment

[0128] In order to verify the effectiveness of the proposed method, 8 groups of ablation experiments were conducted on two sets of data sets, and the results are shown in Table 1 below. The corresponding modules are added on the basis of the previous module: using a single BERT relation extraction (BERT), adding a graph neural network (GCN), adding a graph pooling layer (SAG) with a self-attention mechanism, and adding a residual shrinkage module (RS). The experimental results show that in some DUIE data sets, adding a dependency syntax tree will reduce the F1 value of the model by 0.39%. This is because the noise is too large, which has a great impact on the overall model. After the two denoising methods, the F1 value of the model increased by 0.12% and 0.23% respectively. In the SemEval 2010 data set, after adding the dependency syntax tree, the overall effect of the model was improved by 0.16% after the graph propagation of GCN. Compared with the DUIE Chinese data set, the noise of the English syntax tree has less impact on the model. After the SAG and RS modules, the F1 value increased by 0.11% and 0.15% respectively.

[0129] Table 1 Ablation experiment

[0130]

[0131] In addition, when the experimental parameters are the same, the present invention analyzes the influence of different parameters on the model performance. Table 2 shows the performance comparison of different parameters on the DUIE 2.0 dataset.

[0132] Table 2 Effect of different parameters on model performance

[0133]

[0134] As can be seen from Table 2, when other parameters are the same, the learning rate has a greater impact on the model. This is because the model is difficult to reach the optimal value within the specified number of iterations due to the small learning rate. The text length, number of GCN layers, and SAG coefficient all have different effects on the classification effect of the model. When the number of GCN layers of the graph neural network is 3 and the SAG coefficient is 0.2, the model effect is optimal.

Claims

1. A method for extracting entity relations by integrating semantic and syntactic information, characterized by: The process includes the following steps, which are performed in sequence: Step 1: Establish a BERT encoding layer. The text information is encoded through the BERT encoding layer and converted into word vector information corresponding to the entity; Step 2: Establish a message propagation module, which includes a graph neural network GCN, a graph convolution module SAG based on a self-attention pooling layer, and a residual shrinkage module. The graph neural network GCN uses an external syntax parser to construct a dependency syntax graph. The graph structure accepts the word vector information encoded by BERT, and then obtains the vector through the message propagation module and concatenates it with the entity vector encoded by BERT. Step 3: Pass the concatenated vectors from step 2 through the decoding layer to generate a label matrix of the predicted entity, and obtain the relationship between the predicted entity pairs; Step 4: Introduce the loss function, calculate the loss value of the relationship between specified entities through the interpolation prediction method, introduce the entity type to limit the classification of the relationship between entities, use the pre-defined entity type relationship, and specify the classification vector between irrelevant entities. Set up the mask mechanism to calculate and obtain the loss value of the relationship between specified entities, and obtain the entity relationship that integrates semantic and syntactic information.

2. The entity relationship extraction method integrating semantic and syntactic information according to claim 1 is characterized by: The specific working steps of the BERT encoding layer in step 1 are: The text information is passed through the encoder of the BERT encoding layer to obtain a sequence encoding with a digital expression of the text, and then passes through the multi-layer Transformer structure of the BERT encoding layer, using a multi-head self-attention mechanism and a forward propagation module to learn and obtain the vector representation of each word; the self-attention mechanism uses three learnable parameter matrices to obtain the corresponding query matrix Q, key matrix K and value matrix V from the input sequence L, as shown in formula (1): Q=L·W Q ,K=L·W K ,V=L·W V (1); Among them, W Q , W K , W V Corresponding to the learnable parameters of the three matrices; Calculate the attention score of each head in the value vector and multiply it with the value matrix V to get a vector output representation of the head; after calculating multiple heads and splicing them together, output the vector representation of the sequence: Among them, A i represents the output vector of the i-th head; T represents the transpose operation; d k Indicates the length of the sequence; Q i represents the query matrix in the i-th head; K i represents the key matrix in the i-th head; V i represents the i-th head value matrix; After obtaining the vectors of the text information, we use the text position {i,j}∈(0,d k ) to get the corresponding entity vector, and finally after the average operation, the entity vector expression is obtained: e k =mean([M;<i,j>]) (3); Among them, e k Represents the entity vector at position {i,j} and index k; i and j represent the head and tail positions of the entity respectively; mean represents the average of the specified dimension of the matrix; M represents the sentence vector.

3. The entity relationship extraction method integrating semantic and syntactic information according to claim 1 is characterized by: The following methods are used in the dependency grammar graph in step 2 to reduce the error propagation caused by the external grammar parser: first, entities are predefined in the model of the dependency grammar graph using a dictionary method, and the entities are set as high-frequency items so that more attention can be paid to the predefined information during word segmentation; second, a graph convolution module SAG based on a self-attention pooling layer is used to combine the attention mechanism with the convolution operation; finally, a residual shrinkage module is added to perform soft threshold processing on the internal vector of the node using a learned threshold.

4. The entity relationship extraction method integrating semantic and syntactic information according to claim 3 is characterized by: The graph convolution module SAG based on the attention pooling layer uses the neighbor information of the node to update the node representation. The representation of each node is obtained by fusing the features of its neighbor nodes. The fused node features are expressed as follows: Among them, H l+1 ∈R n×d Represents the node vector of the l+1th layer; for The degree matrix of l is the weight matrix of the lth layer; represents the sum of the adjacency matrix and the identity matrix; H l A node vector representing layer l.

5. The entity relationship extraction method integrating semantic and syntactic information according to claim 4 is characterized by: The specific method of combining the attention mechanism with the convolution operation is: First, each node vector generated by the BERT encoding layer is averaged to obtain the input vector. Then, the node representation and adjacency matrix of the dependency grammar graph are input into the graph convolutional network and multiplied with the learnable attention parameter matrix M to obtain the node importance matrix that integrates node information and structural information, as shown in formula (5): Among them, tanh activation function; A l is the adjacency matrix of the graph layer l; I is the identity matrix, I∈{0,1} n×n ;M l+1 is the attention matrix, M l+1 ∈R d×1 ; rank is the ranking of node importance coefficients, rank∈R n×1 ; In order to discard irrelevant node features and increase the feature representation of entity nodes, the rows and columns of the adjacency matrix A where the nodes whose rank sequence values ​​are less than the set threshold are located are set to 0, so as to achieve the purpose of not affecting the node information in the next layer of global message transmission process, as shown in formula (6): Among them, top_rank is the ranking function; k is the ratio coefficient of the current layer nodes to the number of nodes in the previous layer; n is the number of nodes; change is the adjacency matrix transformation function; A l+1 Represents the adjacency matrix of the next layer of graph propagation; Finally, the next layer output after the graph convolution module is expressed as:

6. The entity relationship extraction method integrating semantic and syntactic information according to claim 1 is characterized by: The decoding steps of the decoding layer in step 3 are: First, the node vector of the entity is found in the dependency syntactic graph using the id index, and then concatenated with the word vector information of the BERT encoding layer to obtain the final entity vector. Secondly, the feature information between entities is obtained by concatenating entities in pairs, and the linear layer is used to map the feature information between entities to the relationship classification. The specific formula is as follows: Among them, e a,b represents the entity feature vector with index a and b; g a,b and h a,b Respectively represent the node vector and word vector in the graph; [;] represents the concatenation between vectors; M a,b Represents the classification value inside the decoding matrix; W and b are the parameters of the linear layer, W∈R d ×n , b∈R d×n .

7. The entity relationship extraction method integrating semantic and syntactic information according to claim 1 is characterized by: The loss function in step 4 is obtained by setting a mask mechanism, as shown in formula (10): loss=w·CEL GCN (p1,l)+(1-w)CEL BERT (p2,l) (10); Among them, w is the weight coefficient; CEL represents the cross entropy loss function Cross Entropy; p1 and p2 represent the predicted values ​​output by the encoding layer of the graph neural network GCN and BERT respectively; l is the true value label; in, THE GCN (p1,l)=CEL(p1·id_mask,l·id_mask) id_mask=R<entity i ,entity j > (11); In the formula, id_mask is the mask between entities; R is the function to determine whether it is a predefined entity type; entity i is the entity i category; j is the entity category j.

Images