A method for optimizing a BERT model in combination with knowledge graph entity correlation degree

Abstract

The application discloses a BERT model optimization method combined with knowledge graph entity correlation. The method firstly obtains an entity set through entity linking according to a knowledge graph and a text data set, and calculates the correlation between entities according to the shortest path length between the entities on the knowledge graph in the entity set, then inputs the text data set into the BERT model as a training sample, trains the BERT model by using the loss function in the BERT model, and optimizes the attention distribution of the corresponding entity part in the text data under the multi-head attention of the BERT model according to the correlation between the entities during the training process, obtains the optimized BERT model, and applies the BERT model to processing a downstream task. The application guides the training of the BERT model through the knowledge graph entity correlation, and effectively improves the effect of the BERT model in processing the downstream task.

Description

Technical Field

[0001] This invention relates to pre-trained language models in the field of natural language processing, specifically to an optimization method for the BERT model that combines entity association in knowledge graphs. Background Technology

[0002] In recent years, natural language processing (NLP) technology has made significant progress. Among them, the pre-trained model BERT (based on a Transformer bidirectional encoder structure, it is a pre-trained language model) provides strong support for solving downstream tasks in the field of NLP due to its excellent task performance and general applicability, and is an important foundation for research and application in the field of NLP.

[0003] Existing research mainly focuses on training the BERT model for downstream tasks based on multi-head attention before processing those tasks. The drawback of this method is that multi-head attention is ignorant of and ignores the entity knowledge in the text data, neglecting the relationships between entities. During training, attention cannot be allocated according to the degree of relationship between entities, so the BERT model is not fully learned and cannot fully realize its performance in processing downstream tasks. Summary of the Invention

[0004] To address the above issues, this invention proposes a BERT model optimization method that combines knowledge graph entity association. First, entity sets are obtained by linking entities based on the knowledge graph and text dataset. Then, the association degree between entities is calculated based on the shortest path length between entities in the entity set. Next, the text dataset is used as training samples and input into the BERT model. The model is trained using the loss function within the BERT model. During the training process, the attention distribution of corresponding entity parts in the text data under multi-head attention in the BERT model is optimized based on the association degree between entities, allowing the BERT model to learn fully and thus fully leverage its performance in processing downstream tasks.

[0005] The specific details of the BERT model optimization method combining knowledge graph entity association degree proposed in this invention are as follows:

[0006] S1. Obtain an entity set by linking entities based on the knowledge graph and text dataset.

[0007] S2. Calculate the degree of association between entities based on the shortest path length between entities in the entity set on the knowledge graph.

[0008] S3. Use the text dataset as input to the BERT model, train it using the loss function inside the BERT model, and optimize the corresponding entity parts in the text data according to the correlation between entities during the training process. Under the multi-head attention of the BERT model, the attention distribution is optimized to obtain the BERT model.

[0009] S4. Use the optimized BERT model to process downstream tasks.

[0010] As a preferred approach, step S1 involves linking the text content within the text dataset T to the corresponding entities in the knowledge graph to obtain the entity set ES.

[0011] As a preferred option, step S2 is based on the entity set ES in step (1), and the specific process is as follows:

[0012] (2-1) Extract an entity from ES as the target entity, and calculate the shortest path length and average shortest path length between the target entity and other entities:

[0013] E,Avg=Dij(m,N)

[0014] Where m represents the target entity, N represents other entities besides the target entity, Dij() represents the algorithm for calculating the shortest path length, E represents the vector formed by the shortest path lengths between the target entity and other entities, and Avg represents the average of the shortest path lengths between the target entity and other entities.

[0015] (2-2) Construct a vector from the average of the shortest path lengths and the ratio of the shortest path lengths of each entity to the target entity, and use normalization techniques to map the values ​​of each dimension of the vector to the interval [0,1]:

[0016]

[0017] Here, Normalization() is the normalization function. The final result is the set R of association degrees corresponding to each entity in the set ES.

[0018] (2-3) Extract the next entity from ES as the target entity, repeat steps (2-2) and (2-2) until all entities in ES have been traversed, and construct the set of association degrees between all target entities and other entities into the set of association degrees between all entities RT.

[0019] As a preferred embodiment, step S3 is based on the text dataset T in step (1) and the correlation set RT in step (2), and the specific process is as follows:

[0020] (3-1) For set T, the text data in it is used as the input of the BERT model. The loss function inside the BERT model is used for training, and the parameters of the BERT model are trained with the goal of reducing the loss.

[0021] (3-2) For each piece of text data in the set T involved in step (1) during the BERT model training process, it consists of several words, some of which are entities. An N×N correlation matrix S is constructed with length N, containing the correlation between all words in the text data. Specifically, for any two words in the text data, if they are the same, the correlation between the two words is 1. Otherwise, if they are different and at least one of them is not an entity in the knowledge graph, the correlation between the two words is 0. The correlation of the remaining parts of the correlation matrix S is determined by the set RT.

[0022] (3-3) Based on the correlation matrix S, the attention distribution under multi-head attention in the BERT model is optimized to obtain the optimized BERT model. Specifically, the attention distribution is composed of attention weights. First, the attention weights before optimization are adjusted based on the Hadamard product of the correlation matrix S, then fused with the attention weights before optimization, and finally the optimized attention distribution is obtained through the attention distribution calculation function.

[0023]

[0024]

[0025] Where ⊙ represents the Hadamard product, which refers to the element-wise multiplication of two matrices; score is the result of fusing knowledge graph relevance and semantic relevance; Q, K, and V are the query, key, and value parameter matrices involved in the attention distribution under multi-head attention within the BERT model, respectively. T The attention weights before optimization, d k Let Q and K be the dimensions of the row vectors of matrices Q and K, softmax be the probability distribution function, and MultiAttention be the function for calculating the attention distribution under multi-head attention.

[0026] Beneficial Effects: The BERT model optimization method proposed in this invention, which combines knowledge graph entity correlation, first obtains an entity set by linking entities based on the knowledge graph and text dataset. Then, it calculates the correlation between entities based on the shortest path length between entities in the knowledge graph. Next, it inputs the text dataset as training samples into the BERT model and trains it using the loss function inside the BERT model. During the training process, it optimizes the attention distribution of corresponding entity parts in the text data under multi-head attention in the BERT model based on the correlation between entities. This allows the BERT model to allocate attention according to the degree of correlation between entities during training, focusing attention on entities with higher correlation, thereby effectively improving the performance of the BERT model in processing downstream tasks. Attached Figure Description

[0027] Figure 1 This is a flowchart of the present invention;

[0028] Figure 2 This is a schematic diagram illustrating a specific example of the present invention. Detailed Implementation

[0029] A specific process for optimizing a BERT model by incorporating entity association from a knowledge graph is as follows: Figure 1 As shown, the specific steps include the following:

[0030] 1. Obtain an entity set by linking entities based on the knowledge graph and text dataset.

[0031] 2. Calculate the degree of association between entities based on the shortest path length between entities in the entity set on the knowledge graph.

[0032] 3. Use the text dataset as input to train the BERT model. During training, optimize the attention distribution of corresponding entity parts in the text data under the multi-head attention of the BERT model based on the correlation between entities to obtain the optimized BERT model.

[0033] 4. Use the optimized BERT model to process downstream tasks.

[0034] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the accompanying drawings of this invention are further described in detail below, with specific details as follows:

[0035] Example 1:

[0036] like Figure 2 As shown, the present invention processes semantic matching tasks, mainly including the following steps:

[0037] (1) Entity sets are obtained by linking entities based on the synonym knowledge graph and the text dataset T, where each record in the text dataset T is represented as a triple. <r,r + r - > represents a sample data point, a positive sample data point, and a negative sample data point for the semantic matching task, respectively. The text dataset is linked with the thesaurus knowledge graph to obtain the entity set ES corresponding to the semantic matching task.

[0038] (2) Calculate the association degree between entities in the semantic matching task based on the shortest path length between entities in the entity set ES on the synonym knowledge graph. The specific process is as follows:

[0039] (2-1) Extract an entity from ES as the target entity, and calculate the shortest path length and average shortest path length between the target entity and other entities:

[0040] E,Avg=Dij(m,N)

[0041] Where m represents the target entity, N represents other entities besides the target entity, Dij() represents the algorithm for calculating the shortest path length, E represents the vector formed by the shortest path lengths between the target entity and other entities, and Avg represents the average of the shortest path lengths between the target entity and other entities.

[0042] (2-2) A vector is formed by the ratio of the average shortest path length to the shortest path length of each entity to the target entity, and normalization is used to map the values ​​of each dimension of the vector to the interval [0,1].

[0043]

[0044] Here, Normalization() is the normalization function. It yields the set R of association degrees corresponding to each entity in the set ES.

[0045] (2-3) ES extracts the next entity as the target entity, and repeats the above steps until all entities in ES have been traversed, and merges the set of association degrees between all target entities and other entities into the set of association degrees between all entities RT.

[0046] (3) The text dataset T is used as the input of the BERT model. Triplet Loss is used as the loss function when the BERT model is trained for the semantic matching task. During the training process, the attention distribution of the corresponding entity parts in the text data under the multi-head attention of the BERT model is optimized according to the correlation between entities, so as to obtain the BERT model optimized for the semantic matching task.

[0047] (3-1) For set T, all sample data in all records are used as input to the BERT model. Then, during training, the Triplet Loss function is used to calculate the cosine distance between a sample and its positive and negative samples respectively, bringing the sample closer to its positive samples and pushing the sample further away from its negative samples. The parameters of the BERT model are trained with the goal of reducing the Triplet Loss.

[0048]

[0049] Where m is a hyperparameter, with a default value of 1, d is the cosine distance function, and p is the sample data. + Let p be the positive sample data. - For p, there are negative sample data.

[0050] (3-2) For each piece of text data in the set T involved in step (1) during the BERT model training process, it consists of several words, some of which are also entities in the knowledge graph of step (1). Construct an N×N correlation matrix S of length N. S contains the correlation between all words in the text data (i.e., not just the correlation between entities). Specifically, for any two words in the text data, if they are the same, the correlation between the two words is 1. Otherwise, if they are different and at least one of them is not an entity in the knowledge graph, the correlation between the two words is 0. The correlation of the remaining part of the correlation matrix S is determined by the set RT.

[0051] (3-3) Based on the correlation matrix S, the attention distribution under multi-head attention in the BERT model is optimized to obtain the optimized BERT model. Specifically, the attention distribution is composed of attention weights. First, the original attention weights in the correlation matrix S are adjusted based on the Hadamard product, then fused with the original attention weights, and finally the optimized attention distribution is obtained through the attention distribution calculation function.

[0052]

[0053]

[0054] Where ⊙ represents the Hadamard product, which refers to the element-wise multiplication of two matrices; score is the result of fusing knowledge graph relevance and semantic relevance; Q, K, and V are the query, key, and value parameter matrices involved in the attention distribution under multi-head attention within the BERT model, respectively. T For the original attention weights, d kLet Q and K be the dimensions of the row vectors of matrices Q and K, softmax be the probability distribution function, and MultiAttention be the function for calculating the attention distribution under multi-head attention.

[0055] Table 1

[0056] accuracy F1 BERT 87.6％ 95.8％ This invention 89.6％ 97.2％

[0057] Table 1 shows the experimental results of the BERT model and the optimized BERT model (i.e., BERT(our)) in the semantic matching task. Compared with the original BERT model, the BERT model optimized by the BERT model optimization method combining knowledge graph entity association proposed in this invention has significant performance improvements in all indicators (accuracy and recall F1).

Claims

1. A method for optimizing the BERT model by combining entity association in a knowledge graph, characterized in that, Includes the following steps: S1. Link entities based on the knowledge graph and text dataset to obtain an entity set; S2. Calculate the association degree between entities based on the shortest path length between them on the knowledge graph. The specific process is as follows: S2.1 Extract an entity from Elasticsearch as the target entity, and calculate the shortest path length and average shortest path length between the target entity and other entities: E,Avg=Dij(m,N) Where m represents the target entity, N represents other entities besides the target entity, Dij() represents the algorithm for calculating the shortest path length, E represents the vector formed by the shortest path lengths between the target entity and other entities, and Avg represents the average of the shortest path lengths between the target entity and other entities. S2.

2. Construct a vector from the average of the shortest path lengths and the ratio of the shortest path lengths of each entity to the target entity, and normalize the values ​​of each dimension of the vector to the interval [0,1]. Normalization() is the normalization function; it yields the set R of association degrees for each entity in ES. S2.3 Extract the next entity from ES as the target entity, and repeat S2.1 and S2.2 until all entities in ES have been traversed, and construct the set of association degrees between all target entities and other entities into the set of association degrees between all entities RT. S3. Use the text dataset as input to train the BERT model. During training, optimize the attention distribution of corresponding entity parts in the text data under the multi-head attention of the BERT model based on the correlation between entities, to obtain the optimized BERT model. The specific process is as follows: S3.1 For a text dataset T, use the text data as input to the BERT model and train it using the loss function inside the BERT model; S3.2 Each piece of text data in the text dataset T consists of several words, and the entity is a part of those words; construct an N×N correlation matrix S of length N, which contains the correlation between all words in the text data; The specific construction process of the correlation matrix S is as follows: for any two words in the text data, if they are the same, the correlation between the two words is 1; otherwise, if they are different and at least one of them is not an entity in the knowledge graph, the correlation between the two words is 0; the correlation of the remaining part of the correlation matrix S is determined by the set RT. S3.

3. Based on the correlation matrix S, optimize the attention distribution under multi-head attention in the BERT model to obtain the optimized BERT model. The specific process is as follows: The attention distribution is composed of attention weights. First, adjust the attention weights before optimization based on the Hadamard product in the correlation matrix S, then fuse them with the attention weights before optimization, and finally obtain the optimized attention distribution through the attention distribution calculation function. Where ⊙ represents the Hadamard product, which refers to the element-wise multiplication of two matrices; score is the result of fusing knowledge graph relevance and semantic relevance; Q, K, and V are the query, key, and value parameter matrices involved in the attention distribution under multi-head attention within the BERT model, respectively. T The attention weights before optimization, d k Let Q and K be the dimensions of the row vectors of matrices Q and K, softmax be the probability distribution function, and MultiAttention be the function for calculating the attention distribution under multi-head attention. S4. Use the optimized BERT model to process downstream tasks.

2. The BERT model optimization method combining knowledge graph entity association degree according to claim 1, characterized in that, Step S1 is as follows: By linking the text content in the text dataset T to the corresponding entities in the knowledge graph, we obtain the entity set ES.

Images