An entity relation extraction method fusing semantic and syntactic information

Abstract

The application discloses an entity relation extraction method fusing semantic and syntactic information, and belongs to the technical field of language processing.The application realizes the fusion of semantic and syntactic information by learning information in a sentence, and improves the accuracy of relation classification between entities.The application mainly uses a graph convolutional neural network with a self-attention pooling layer and a residual shrinkage network to reduce the influence of noise in the text by combining an external grammar and word vector information, and improves long-distance dependence between entities.Meanwhile, an interpolation prediction method is used to calculate a loss, and the interaction between BERT and the graph neural network is improved.Experiments show that the application fully fuses semantic vectors and syntactic structure information, so that the relation extraction effect is significantly improved compared with other benchmark models, and the long-distance relation recognition rate between entities in a sentence and the noise problem contained are reduced.

Description

Technical Field

[0001] This invention belongs to the field of language processing technology, and in particular relates to a method for extracting entity relations by integrating semantic and syntactic information. Background Technology

[0002] Currently, relation extraction involves classifying the relationships between numerous entities in text to effectively mine information within the text and support downstream tasks such as graph construction, question answering systems, and search engines. However, general extraction algorithms are heavily influenced by the distance between entities, making it difficult to fully extract long-distance dependencies. While graph-based relation extraction can provide additional information about entity vectors, its over-reliance on data structures built using external syntax generators can introduce noise that hinders entity interactions, thus affecting the effectiveness of relation extraction and impacting downstream tasks.

[0003] Therefore, there is an urgent need for a technical solution to address this problem among existing technologies. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide an entity relation extraction method that integrates semantic and syntactic information to solve the technical problem that existing extraction algorithms are greatly affected by the distance between entities, noise, etc., and the accuracy of relation extraction is poor.

[0005] An entity relation extraction method that integrates semantic and syntactic information includes the following steps, which are performed sequentially:

[0006] Step 1: Establish a BERT encoding layer. Text information is encoded through the BERT encoding layer and converted into word vector information corresponding to entities.

[0007] Step 2: Establish a message propagation module. The message propagation module includes a graph neural network (GCN), a graph convolution module (SAG) based on a self-attention pooling layer, and a residual shrinking module. In the graph neural network (GCN), an external syntax parser is used to construct a dependency syntax graph. This graph structure receives word vector information encoded by BERT, and then obtains the vector through the message propagation module, which is then concatenated with the entity vector encoded by BERT.

[0008] Step 3: The concatenated vector from Step 2 is passed through the decoding layer to generate a label matrix for the predicted entities, thereby obtaining the relationship between the predicted entity pairs.

[0009] Step 4: Introduce a loss function and calculate the loss value of the relationship between specified entities using interpolation prediction. Introduce entity types to restrict the classification of relationships between entities. Through predefined entity type relationships and specify the classification vector between unrelated entities, set a masking mechanism to calculate and obtain the loss value of the relationship between specified entities, and obtain entity relationships that integrate semantic and syntactic information.

[0010] The specific working steps of the BERT coding layer in step one are as follows:

[0011] The text information is encoded by the encoder of the BERT encoding layer to obtain a sequence encoding with digital text representation. Then, it is processed by the multi-layer Transformer structure of the BERT encoding layer, and the vector representation of each word is learned and obtained using a multi-head self-attention mechanism and a forward propagation module. The self-attention mechanism uses three learnable parameter matrices to obtain the corresponding query matrix Q, key matrix K and value matrix V from the input sequence L, as shown in formula (1):

[0012] Q = L·W Q K = L·W K V = L·W V (1);

[0013] Among them, W Q W K W V Learnable parameters corresponding to the three types of matrices;

[0014] Calculate the attention score for each head on the value vector and multiply it by the value matrix V to obtain a vector output representation of that head; after calculations for multiple heads, concatenate them together to output the vector representation of the sequence.

[0015]

[0016] Among them, A i The output vector of the i-th head is represented by T; T represents the transpose operation; d k Q represents the length of the sequence; i K represents the query matrix in the i-th header; i V represents the key matrix in the i-th head; i Represents the i-th head value matrix;

[0017] After obtaining the vectors of the text information, the text position of the k-th entity {i,j}∈(0,d) is used. k The corresponding entity vectors are obtained, and finally, after averaging, the entity vectorization representation is obtained:

[0018] e k =mean([M;<i,j> ]) (3);

[0019] Among them, e k This represents the entity vector located at {i,j} with index k; i and j represent the beginning and end positions of the entity, respectively; mean represents the average of the matrix over a specified dimension; M represents the sentence vector.

[0020] In step two, the dependency syntax graph employs the following methods to reduce error propagation caused by the external parser: First, entities are predefined in the dependency syntax graph model using a dictionary approach, and these entities are set as high-frequency items so that more attention is paid to the predefined information during word segmentation; second, a graph convolution module based on self-attention pooling layers (SAG) is used to combine the attention mechanism with convolution operations; finally, a residual shrinking module is added, using a learned threshold to perform soft thresholding on the vectors inside the nodes.

[0021] The graph convolutional module SAG based on the attention pooling layer updates the representation of a node using the neighbor information of the node. The representation of each node is obtained by fusing the features of its neighbor nodes. The fused node features are represented by the following formula (4):

[0022]

[0023] Among them, H l+1 ∈R n×d The node vector representing the (l+1)th layer; for The degree matrix; W l It is the weight matrix of the l-th layer; H represents the sum of the adjacency matrix and the identity matrix; l This represents the node vector of layer l.

[0024] The specific method for combining the attention mechanism with convolution operations is as follows:

[0025] First, the node vectors generated by the BERT encoding layer are averaged to obtain the input vector. Then, the node representations and adjacency matrices of the dependency syntax graph are input into the graph convolutional network and multiplied with the learnable attention parameter matrix M to obtain the node importance matrix that integrates node information and structural information, as shown in formula (5).

[0026]

[0027] A l =A l +I

[0028] Wherein, the tanh activation function; A l Let I be the adjacency matrix of layer l in Figure 1; I is the identity matrix, I∈{0,1} n×n M l+1 Let M be the attention matrix. l+1 ∈R d×1 rank is the ranking of node importance coefficients, rank∈R n×1 ;

[0029] To discard irrelevant node features and add feature representations for entity nodes, the adjacency matrix A containing nodes with values ​​less than a set threshold in the rank sequence is set to 0. This ensures that node information is not affected during the next layer of global message passing, as shown in equation (6).

[0030] id=top_rank(rank,[k·n]) (6);

[0031] A l+1 =change(A l ,id)

[0032] Where top_rank is the sorting function; k is the ratio of the current layer nodes to the previous layer nodes; n is the number of nodes; change is the adjacency matrix transformation function; A l+1 This represents the adjacency matrix for the next layer of graph propagation; finally, the output of the next layer after the graph convolution module is represented as:

[0033]

[0034] The decoding steps of the decoding layer in step three are as follows:

[0035] First, the node vector of the entity is found in the dependency parsing graph using the id index, and then concatenated with the word vector information of the BERT encoding layer to obtain the final entity vector. Second, the feature information between entities is obtained by concatenating them pairwise, and a linear layer is used to map the feature information between entities to relation classification. The specific formula is as follows:

[0036]

[0037] Among them, e a, b represents the feature vector of the entity with indices a and b; g a,b and h a,b These represent the node vector and word vector in the graph, respectively; [;] represents the concatenation of vectors; M a,b Represents the classification value within the decoding matrix; W and b are parameters of the linear layer, W∈R d×n b∈R d×n .

[0038] In step four, the loss function is obtained by setting a masking mechanism, as shown in formula (10):

[0039] loss = w·CEL GCN (p1,l)+(1-w)CEL BERT (p2,l) (10);

[0040] Where w is the weight coefficient; CEL represents the cross-entropy loss function; p1 and p2 represent the predicted values ​​output by the encoding layers of the graph neural network GCN and BERT, respectively; l is the ground truth label.

[0041] in,

[0042] CEL GCN (p1,l)=CEL(p1·id_mask,l·id_mask)

[0043] id_mask = R < entity i entity j > (11);

[0044] In the formula, id_mask is the mask between entities; R is a function to determine whether it is a predefined entity type; entity i For entity i, the category is: entity j For entity j, the category is j.

[0045] Through the above design scheme, the present invention can bring the following beneficial effects:

[0046] This invention proposes an entity relation extraction method based on graph neural networks. By learning information from sentences, it achieves the fusion of semantic and syntactic information, improving the accuracy of entity relation classification. This invention mainly combines external grammar tools and word vector information, using a graph convolutional neural network with self-attention pooling layers and a residual shrinking network to reduce the impact of noise in the text and improve long-distance dependencies between entities. Simultaneously, it uses interpolation prediction to calculate the loss, improving the interaction between BERT and the graph neural network. Experiments show that this invention fully integrates semantic vectors and syntactic structure information, resulting in a significant improvement in relation extraction performance compared to other benchmark models, alleviating the low recognition rate and noise issues associated with long-distance entity relations within sentences.

[0047] This invention demonstrates the potential to improve relation extraction performance by combining BERT encoding layers with graph neural networks and fusing semantic and syntactic information, providing valuable reference and direction for future research. Attached Figure Description

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0049] Figure 1 This is a flowchart of an entity relation extraction method that integrates semantic and syntactic information according to the present invention;

[0050] Figure 2 This is a schematic diagram illustrating the working principle of the message propagation module in an entity relation extraction method that integrates semantic and syntactic information according to the present invention.

[0051] Figure 3 This is an example diagram illustrating the construction of a dependency syntax graph in an entity relation extraction method that integrates semantic and syntactic information according to the present invention.

[0052] Figure 4 This is a schematic diagram illustrating the working principle of the graph convolutional module SAG based on the self-attention pooling layer in the entity relation extraction method that integrates semantic and syntactic information of the present invention.

[0053] Figure 5 This is a schematic diagram illustrating the working principle of the residual shrinkage module in an entity relation extraction method that integrates semantic and syntactic information according to the present invention.

[0054] Figure 6 This is a comparison of the relation extraction capabilities of the entity relation extraction method that integrates semantic and syntactic information in this invention under different datasets.

[0055] Figure 7 This is a comparison diagram of the confusion matrix generated in an embodiment of the entity relation extraction method that integrates semantic and syntactic information according to the present invention;

[0056] Figure 8 This is a comparison chart of three metrics under different sentence lengths in two datasets, as part of an embodiment of an entity relation extraction method that integrates semantic and syntactic information according to the present invention. Detailed Implementation

[0057] The entity relation extraction method that integrates semantic and syntactic information disclosed in this invention mainly comprises three parts: a BERT encoding layer, a message propagation module, and a decoding layer, as detailed below. Figure 1 As shown, firstly, the text information is encoded by the BERT encoding layer, converting it into corresponding character vector information. Then, an external parser is used to construct a dependency syntax graph. This graph structure accepts the segmented word vectors from the BERT encoding, obtains the vectors through a specific message passing mechanism, and concatenates them with the entity vectors from the BERT encoding. Finally, a decoding layer generates a label matrix for the predicted entities.

[0058] 1. BERT-based entity encoding layer

[0059] The purpose of the entity encoding layer is to obtain vector representations of entities for subsequent calculation of entity relationship classification. For an input text X = {x1, x2, x3, ..., x...} n After pre-training the BERT model, the corresponding entity vector representation E = {e1, e2, e3, ..., e} is obtained. kSpecifically, the text is encoded by the encoder of BERT to obtain a sequence with digital representation of the text. Then, it goes through a multi-layer Transformer structure and uses multi-head self-attention mechanism and forward propagation to learn the vector representation of each character. The self-attention mechanism can be expressed as follows: for the input sequence L, the corresponding Q (query), K (key), and V (value) matrices are obtained using three learnable parameter matrices, as shown in formula (1):

[0060] Q = L·W Q K = L·W K V = L·W V (1)

[0061] Among them, W Q W K W V Learnable parameters corresponding to the three types of matrices.

[0062] Furthermore, the attention score of each head on the value vector is calculated and multiplied by the value matrix V to obtain the vector output representation of a head. Similarly, after calculations by multiple heads, the vector representation of the sequence is finally concatenated. Equation (2) represents the output vector of the i-th head.

[0063]

[0064] Where T represents the transpose operation; d k Indicates the length of the sequence;

[0065] After obtaining the text vector, use the text position {i,j}∈(0,d) of the k-th entity. k The corresponding entity vector can be obtained by ), and finally, after averaging, the entity vectorization expression can be obtained. Equation (3) represents the entity e at position {i,j}. k vector.

[0066] e k =mean([M;<i,j> (3)

[0067] Among them, e k This represents the entity vector with index k; i and j represent the head and tail positions of the entity, respectively; mean represents the average of the matrix over a specified dimension.

[0068] 2. Message propagation module based on dependency parsing graph

[0069] Compared to traditional vector extraction methods, constructing dependency parsing graphs can greatly reveal the internal structure and syntactic information of sentences. However, some irrelevant nodes, such as commas and periods, can introduce irrelevant information between entities during message passing in graph neural networks, making it difficult for entity nodes to learn key features. To reduce the error propagation problem caused by relying on external grammarians, this invention proposes three methods to mitigate this impact. First, entities are predefined in the model using a dictionary approach, and entities are set as high-frequency items so that more attention is paid to predefined information during word segmentation; second, such as Figure 3 As shown, a graph convolution module SAG based on a self-attention pooling layer is used to combine the attention mechanism with the convolution operation; finally, a residual shrinking module RS is added to perform soft thresholding on the vectors inside nodes using a learnable threshold. Figure 3 As shown, the final node vector in the graph is obtained.

[0070] (1) Dependency syntax graph construction

[0071] Dependency graphs, as representations of dependency relationships between words in natural language sentences, represent each segmented word as a node, connected by directed edges (such as subject-verb, verb-object, etc.). As a graphical structure, dependency graphs contain rich semantic and syntactic structural information. Introducing such features when extracting n-gram relations from sentences can effectively utilize structural information and provide more entity-level semantic relationships.

[0072] The construction of dependency grammar graphs is accomplished through external dependency parsers. These parsers automatically infer the dependency relationships between words in a sentence based on grammatical rules and contextual information, thereby generating the corresponding dependency grammar graph. This invention uses the external Chinese parser pyLTP and the English parser spacy model to construct the dependency grammar graph of a sentence. An example of sentence dependency parsing is as follows. Figure 3 As shown.

[0073] An input text is first segmented using LTP (Local Time Protocol), and the segmented results are then tagged with part-of-speech tags. Finally, dependency parsing (DP) yields the result shown in the diagram above. The diagram mainly consists of three parts: the ROOT node, the segmented results, and the dependencies. When assigning embedding vectors to nodes, the average vector of the corresponding node is obtained from the BERT output vector by marking the node's position, serving as the embedding vector for each node. Dependencies are represented as directed edges in the dependency parsing graph. Furthermore, the vector of the ROOT node is used as the CLS vector in the BERT output vector, ultimately generating the dependency parsing graph.

[0074] (2) Graph convolution module based on self-attention pooling layer

[0075] Graph Neural Networks (GCNs), as a deep learning model based on graph structures, can preserve the topological structure and node characteristics of graphs, comprehensively considering both node and structural information. The core idea of ​​GCNs is to use the neighbor information of a node to update the node representation. Each node representation is obtained by fusing the features of its neighboring nodes. The fused node features are shown in the following formula (4):

[0076]

[0077] Among them, H l+1 ∈R n×d The node vector representing the (l+1)th layer; for The degree matrix; W l It is the weight matrix of the l-th layer; H represents the sum of the adjacency matrix and the identity matrix; l This represents the node vector of layer l.

[0078] To guide the model in removing the propagation of irrelevant node information, this invention uses a self-attention map convolution module to assign weight values ​​to each node, thereby achieving the purpose of noise removal. For example... Figure 4 As shown, firstly, the node vectors generated by the BERT encoding layer are averaged to obtain the input vector of the module. Then, the node representations and adjacency matrices of the syntax dependency graph are input into the graph neural network and multiplied with the learnable attention parameter matrix M to obtain the node importance matrix that integrates node information and structural information. As shown in formula (5),

[0079]

[0080]

[0081] Wherein, the tanh activation function; A l Let I be the adjacency matrix of layer l in Figure 1; I is the identity matrix, I∈{0,1} n×n M l+1 Let M be the attention matrix. l+1 ∈R d×1 rank is the ranking of node importance coefficients, rank∈R n×1 .

[0082] To discard irrelevant node features and add feature representations for entity nodes, the adjacency matrix A containing nodes with smaller values ​​in the rank sequence is set to 0, so as not to affect node information during the next layer of global message passing. Specifically, as shown in equation (6):

[0083] id=top_rank(rank,[k·n]) (6)

[0084] A l+1 =change(A l ,id)

[0085] Where top_rank is the sorting function; k is the ratio of the current layer nodes to the previous layer nodes; n is the number of nodes; change is the adjacency matrix transformation function; A l+1 This represents the adjacency matrix for propagation in the next layer of the graph.

[0086] Finally, the output of the next layer after the graph convolution module can be represented as:

[0087]

[0088] (3) Residual shrinkage module

[0089] After graph convolution, self-attention, and sorting operations, the filtered graph structure of each layer can be obtained. In order to further reduce the influence of irrelevant node noise during graph message transmission, this invention introduces a residual shrinkage module RS to perform soft thresholding on the internal vector of each node, as shown in Figure (5).

[0090] First, the absolute values ​​of the node vectors are processed to make the features tend to be positive. Then, a global average pooling operation is performed to obtain the average value of each node vector. Next, threshold features are learned using two fully connected layers and their corresponding activation functions, and mapped to the corresponding dimensional vectors to obtain the threshold feature matrix x, which can be specifically represented as:

[0091] x=sigmod(W2(relu(W1(Pooling(|H l |)+b1)+b2));

[0092] Among them, H l H is the vector obtained after convolution of the nodes. l ∈R n×d Pooling is the global average pooling function; W a and b a W represents the learnable parameter matrix and bias of the linear layer, respectively. a ∈R 1×1 b a ∈R 1×1 ; sigmoid is an activation function that maps nodes to the range of 0-1;

[0093] To prevent the threshold from being too large, x needs to be compared with H in absolute form. l The vectors are multiplied at corresponding positions to obtain a final set of thresholds δ. The original node vectors are then subjected to soft thresholding, as shown in equation (8):

[0094]

[0095] Among them, h l and H respectively l The vector of each node before and after the internal update;

[0096] The pseudocode below illustrates an example of a message propagation process based on a dependency parsing graph in batch processing:

[0097]

[0098] 3. Decoding layer

[0099] For each entity pair (e) in the sentence a ,e b This requires using a linear combination of features to predict the relationship between entity pairs. First, the node vector of the entity is found in the dependency parsing graph using the ID index, and then concatenated with the BERT word vector information to obtain the final entity vector. Second, the feature information between entities is obtained by concatenating them pairwise, and a linear layer is used to map the feature information between entities to relationship classification. The specific calculation method is as follows:

[0100]

[0101] Among them, e a,b This represents the feature vector of the entity with indices a and b; g a,b and h a,b This represents the node vector and word vector in the graph; [;] represents the concatenation of vectors; M a,b Represents the classification value within the decoding matrix; W∈R d×n and b∈R d×n These are the parameters of the linear layer.

[0102] 4. Loss Function

[0103] Because the BERT model and the graph neural network have different structures, the output dimension of the BERT model is too large, which causes the pre-trained model parameters to not be updated in time during backpropagation, and they cannot be directly combined to make the model converge quickly

[30] . Therefore, this invention uses a prediction interpolation method to make the pre-trained model update in real time. Specifically, the vector output by the BERT model is divided into two parts, one as the input of the graph neural network and the other for prediction. The weights between the two parts are connected by weight coefficients, as shown in formula (10):

[0104] loss = w·CEL GCN (p1,l)+(1-w)CEL BERT (p2,l) (10);

[0105] Where w is the weight coefficient; CEL represents the cross-entropy loss function; p1 and p2 represent the predicted values ​​output by GCN and BERT, respectively; and l is the true value label.

[0106] Meanwhile, the model exhibits an imbalance between positive and negative samples when outputting the label matrix, meaning there are few relationships between text entities, resulting in a large number of label matrices but a small number of positive samples. To address this issue, entity types are introduced to restrict the classification of relationships between entities. By pre-defining entity type relationships, a masking mechanism is set to enable the model to calculate the loss value of the relationship between specified entities and specify the classification vector between unrelated entities, as shown in formula (11):

[0107] id_mask = R < entity i entity j > (11);

[0108] CEL GCN (p1,l)=CEL(p1·id_mask,l·id_mask)

[0109] Where id_mask is the mask between entities; R is a function to determine the pre-defined entity type; entity i For entity i, the category is: entity j For entity j, the category is j.

[0110] Example:

[0111] The experiment used two datasets for validation. 1) The Baidu DUIE 2.0 relation extraction dataset, which contains over 30,000 text data points, 130,000 triplet data points, and 31 predefined relation types, including training, test, and validation sets. "Unknown" indicates that there is no relation between entities. 2) The SemEval2010 task8 dataset, with 8000 training data points and 2717 test data points, contains 10 relation types, where "Other" indicates that there is no relation between entities.

[0112] (1) Baseline Comparison

[0113] To verify the effectiveness of the method proposed in this invention, baseline methods were selected for comparison, as described below:

[0114] 1) R-BERT: It directly uses the position index of entities in the sentence, uses BERT to extract entity vectors and CLS vectors to perform relation classification.

[0115] 2) BERT-LSTM: Based on BERT, it utilizes the sequence characteristics of LSTM to extract deeper features and aggregates vector information for classification.

[0116] 3) BERT-GAT: Based on entity graph construction, it uses attention to assign different weights to the edges between entities, then uses graph propagation to update entity vectors, and finally decodes the relationship between two entities.

[0117] 4) GP-GNN: Use GloVe to extract vectors and combine them with location information to obtain entity vectors using an LSTM model. Construct a fully connected graph structure about entities and predict the relationship classification between entities.

[0118] 5) AGGCN: It transforms the original dependency tree into a fully connected weighted graph, and learns the node representation based on the correlation strength between nodes using graph convolutional neural networks, and finally obtains the results between entities.

[0119] Based on the above method, this invention uses F1, accuracy, and recall curves to test the model's relation extraction capability on different datasets, as follows: Figure 6 As shown:

[0120] Experimental results show that the model of this invention outperforms most models in relation extraction tasks, achieving overall performance of 93.28% and 88.41% on the DUIE 2.0 and SemEval2010 task8 datasets, respectively, demonstrating the effectiveness of the model proposed in this invention.

[0121] exist Figure 6 Models such as R-BERT and BERT-LSTM use BERT pre-trained models for encoding, and then perform different operations on the encoded vectors for recognition. However, single word vectors are insufficient to fully understand the meaning between entities, and they miss the analysis of other features. BERT-GAT and other models use graph neural networks for training, focusing on the interaction between encoded entities, but lack guidance on the grammatical and semantic structure of the sentence itself. The model in this invention is based on BERT encoding and uses multiple methods to fully integrate dependency syntactic graph information, thereby improving relation extraction performance.

[0122] (2) Noise situation analysis

[0123] The method proposed in this invention mainly reduces the propagation of noise from irrelevant nodes based on the construction of a dependency syntax graph. To more intuitively demonstrate the impact of adding a graph convolutional module (SAG) based on self-attention pooling layers and a residual shrinking module (RS) on the model's denoising performance, we visualized the denoising on the SemEval2010 task8 dataset. First, the neural network was trained without noise reduction to ignore the influence of noise on the model. Then, the graph convolutional module (SAG) based on self-attention pooling layers and the residual shrinking module (RS) were added sequentially, and the impact of the modules on the overall performance was observed more intuitively by generating a confusion matrix.

[0124] like Figure 7 As shown, the diagonal elements in the matrix represent the correctly predicted relation categories, and lighter colors in the remaining irrelevant elements indicate stronger noise. Compared to the other two confusion matrices, the Graph Neural Network (GCN) has more noise outside the diagonal, while the GCN-SAG-RS has the least. This indicates that using the SAG module, which learns the importance score of each node through an attention mechanism and removes useless nodes, and using the residual shrinking module RS, which learns a soft threshold through a residual shrinking mechanism and updates the node vectors, both methods significantly improve the model's performance.

[0125] (3) The impact of different text lengths on dependency relationships

[0126] To visually demonstrate the improvement in inter-entity dependencies achieved by the model proposed in this invention, such as... Figure 8 As shown, the BERT model without graph neural networks and the text model are compared across two datasets with different sentence lengths. In the DUIE dataset, the performance metrics are similar within the (0, 50) sentence length range, slightly increase between (50, 100) lengths, and rise by 1.34%, 1.7%, and 0.98% respectively within the (100-256) range. In the SemEval 2010 dataset, the F1 score is significantly higher than the former model across all three sentence length ranges, improving by 0.19%, 1.11%, and 2.17% respectively. This indicates that the longer the text length, the more significant the performance improvement of the model in relation extraction, demonstrating that introducing graph neural networks on top of dependency syntax graphs can increase the dependencies between entities and improve the model's relation extraction performance.

[0127] (3) Ablation test

[0128] To verify the effectiveness of the proposed method, eight ablation experiments were conducted on two datasets, and the results are shown in Table 1 below. The following modules were added to the previous module: using a single BERT relation extraction (BERT), adding a graph neural network (GCN), adding a graph pooling layer with self-attention mechanism (SAG), and adding a residual shrinking module (RS). Experimental results show that in some DUIE datasets, adding a dependency syntax tree reduces the model's F1 score by 0.39%. This is because excessive noise has a significant impact on the overall model. After the two denoising methods, the model's F1 score improved by 0.12% and 0.23%, respectively. In the SemEval 2010 dataset, adding a dependency syntax tree and then using graph propagation through GCN improved the overall model performance by 0.16%. Compared to the DUIE Chinese dataset, the noise in constructing the English syntax tree has a smaller impact on the model. After using the SAG and RS modules, the F1 score improved by 0.11% and 0.15%, respectively.

[0129] Table 1 Ablation Experiment

[0130]

[0131] In addition, under the same experimental parameters, this invention analyzed the impact of different parameters on model performance. Table 2 shows the performance comparison of different parameters on the DUIE 2.0 dataset.

[0132] Table 2. Effects of different parameters on model performance

[0133]

[0134] As shown in Table 2, with other parameters remaining the same, the learning rate has a significant impact on the model. This is because a small learning rate makes it difficult for the model to reach its optimal state within a specified number of iterations. Text length, the number of GCN layers, and the SAG coefficient all have different effects on the model's classification performance. The model achieves its optimal performance when the number of GCN layers in the graph neural network is 3 and the SAG coefficient is 0.2.

Claims

1. An entity relation extraction method that integrates semantic and syntactic information, characterized by: The steps include the following steps, and the following steps are performed in sequence: Step 1: Establish a BERT encoding layer. Text information is encoded through the BERT encoding layer and converted into word vector information corresponding to entities. Step 2: Establish a message propagation module. The message propagation module includes a graph neural network (GCN), a graph convolution module (SAG) based on a self-attention pooling layer, and a residual shrinking module. In the graph neural network (GCN), an external syntax parser is used to construct a dependency syntax graph. This graph structure receives word vector information encoded by BERT, and then obtains the vector through the message propagation module, which is then concatenated with the entity vector encoded by BERT. Step 3: The concatenated vector from Step 2 is passed through the decoding layer to generate a label matrix for the predicted entities, thereby obtaining the relationship between the predicted entity pairs. Step 4: Introduce a loss function and calculate the loss value of the relationship between specified entities using interpolation prediction. Introduce entity types to restrict the classification of relationships between entities. Through predefined entity type relationships and specify the classification vector between unrelated entities, set a masking mechanism to calculate and obtain the loss value of the relationship between specified entities, and obtain entity relationships that integrate semantic and syntactic information. The dependency syntax graph in step two employs the following methods to reduce error propagation caused by the external parser: First, entities are predefined in the dependency syntax graph model using a dictionary approach, and these entities are set as high-frequency items so that more attention is paid to the predefined information during word segmentation; second, a graph convolution module based on self-attention pooling layers (SAG) is used to combine the attention mechanism with convolution operations; finally, a residual shrinking module is added, using a learned threshold to perform soft thresholding on the vectors inside the nodes. The Graph Convolutional Module (SAG) based on the attention pooling layer updates the node representation using the neighbor information of the node. The representation of each node is obtained by fusing the features of its neighbor nodes. The fused node features are represented by the following formula: ； in, Representing the Layer node vectors; , for The degree matrix; It is the first Layer weight matrix; This represents the sum of the adjacency matrix and the identity matrix; express Layer node vectors; The specific method for combining the attention mechanism with convolution operations is as follows: First, the node vectors generated by the BERT encoding layer are averaged to obtain the input vector. Then, the node representations and adjacency matrices of the dependency syntax graph are input into the graph convolutional network and multiplied with the learnable attention parameter matrix M to obtain the node importance matrix that integrates node information and structural information, as shown in the following equation: ； in, For activation functions; For the image The adjacency matrix of the layer; It is the identity matrix. ; For attention matrix, ; It is a ranking of node importance coefficients. ; To discard irrelevant node features and add feature representations for entity nodes, The adjacency matrix of nodes whose values ​​in the sequence are less than a set threshold The row and column are set to 0 to ensure that node information is not affected during the global message passing process at the next level, as shown in the following formula: ； in, It is a sorting function; This is the ratio of the number of nodes in the current layer to the number of nodes in the previous layer. The number of nodes; This is the adjacency matrix transformation function; This represents the adjacency matrix used for propagation in the next layer of the graph. Finally, the output of the next layer after the graph convolution module is represented as: 。 2. The entity relation extraction method integrating semantic and syntactic information according to claim 1, characterized in that: The specific working steps of the BERT coding layer in step one are as follows: Text information is encoded by the BERT encoder to obtain a sequence with digital text representation. This sequence then passes through a multi-layer Transformer structure within the BERT encoder, using a multi-head self-attention mechanism and a forward propagation module to learn and obtain the vector representation of each character. The self-attention mechanism uses three learnable parameter matrices to derive the corresponding query matrix from the input sequence L. Key matrix Sum matrix As shown in the following formula: ； in, , , Learnable parameters corresponding to the three types of matrices; Calculate the attention score for each head on the value vector and multiply it by the value matrix V to obtain a vector output representation of that head; after calculations for multiple heads, concatenate them together to output the vector representation of the sequence. ； in, This represents the output vector of the i-th head; Indicates the transpose operation; Indicates the length of the sequence; This represents the query matrix in the i-th header; This represents the key matrix in the i-th head; Represents the i-th head value matrix; After obtaining the vectors of the text information, the first vector is used... The text location of each entity The corresponding entity vectors are obtained, and finally, an averaging operation is performed to obtain the entity vectorized representation: ； in, The representative position is at And the index is entity vectors; and These represent the beginning and end positions of the entity, respectively. This represents the average across a specified dimension of the matrix. Represents a sentence vector.

3. The entity relation extraction method integrating semantic and syntactic information according to claim 1, characterized in that: The decoding steps of the decoding layer in step three are as follows: First, the node vector of the entity is found in the dependency parsing graph using the id index, and then concatenated with the word vector information of the BERT encoding layer to obtain the final entity vector. Second, the feature information between entities is obtained by concatenating them pairwise, and a linear layer is used to map the feature information between entities to relation classification. The specific formula is as follows: ； in, This represents the feature vector of the entity with indices a and b; and These represent the node vectors and word vectors in the graph, respectively. This represents the concatenation of vectors. Represents the classification value within the decoding matrix; , These are all parameters of the linear layer. , .

4. The entity relation extraction method integrating semantic and syntactic information according to claim 1, characterized in that: In step four, the loss function is obtained by setting a masking mechanism, as shown in the following formula: ； in, These are the weighting coefficients; Let CrossEntropy represent the cross-entropy loss function; and These represent the predicted values ​​output by the coding layers of the graph neural network GCN and BERT, respectively. Labels for real values; in, ； ； In the formula, A mask between entities; A function to determine whether an entity is a predefined entity type; For entities category; For entities category.

Images