BERT knowledge graph completion method and system integrating entity types

Abstract

The present invention discloses a BERT knowledge graph completion method and system that integrates entity types. The method uses the external knowledge of entity information, namely the entity type, as a supplement to the entity information to better obtain the semantic information of the entity and improve the text enhancement representation at the input end of the model. The method uses a Transformer encoder with a multi-head attention mechanism to realize feature acquisition and sequence encoding, and uses a small number of labeled triplets for fine-tuning to effectively realize the mining of implicit relationships and achieve fast and accurate cleaning of multi-source heterogeneous data.

Description

Technical Field

[0001] The present invention relates to the field of multi-source heterogeneous data processing, and in particular to a BERT knowledge graph completion method and system that integrates entity types. Background Art

[0002] Multi-source, heterogeneous data often suffers from data quality issues such as missing data and inconsistencies. Data error correction and gap filling are important means of ensuring data quality. Existing data error correction and gap filling technologies rely on a large amount of external knowledge to guide the cleaning process, but this efficiency is limited by the small size and inefficient construction of this external knowledge. Summary of the Invention

[0003] The present invention provides a BERT knowledge graph completion method and system that integrates entity types to solve the technical problem of low cleaning efficiency of multi-source heterogeneous data.

[0004] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0005] A BERT knowledge graph completion method integrating entity types includes the following steps:

[0006] Construct a BERT knowledge graph completion model: The BERT knowledge graph completion model includes an input layer, an encoding layer and an output layer. The input layer is used to construct a representation sequence of the head entity, head entity type, relationship, tail entity and tail entity type of the input text; and splice the representation sequence of the head entity, head entity type, relationship, tail entity and tail entity type into an input sequence; the encoding layer is used to encode the input sequence, extract semantic features of different levels of the encoded input sequence, and splice the semantic features of different levels; the output layer is used to calculate the predicted probability of the spliced ​​semantic features, and calculate the distance between the model predicted probability and the expected probability to adjust the network parameters of the BERT knowledge graph completion model.

[0007] The BERT knowledge graph completion model is trained, and the trained BERT knowledge graph completion model is used to perform classification prediction on the input text.

[0008] Preferably, the encoding layer uses a bidirectional Tansformer encoder to encode the input sequence, and uses a multi-head attention mechanism to extract semantic features of different levels of the encoded input sequence, and splices the semantic features of different levels; the output layer uses a sigmoid function to calculate the predicted probability of the spliced ​​semantic features, and calculates the distance between the model predicted probability and the expected probability, and determines whether the distance exceeds a preset threshold. If the distance exceeds the threshold, the network parameters of the completion model are adjusted; the output layer uses the final hidden state of [CLS] as the deep representation of the input sequence and the output of the model.

[0009] Preferably, the input sequence is expressed as:

[0010]

[0011] Among them, S represents the input sequence, [CLS] is the classification label, is the representation sequence of the head entity; A sequence representing the head entity type; is a sequence of representations of relations; is the representation sequence of the tail entity; It is a representation sequence of the tail entity type, and [SEP] is a separator.

[0012] Preferably, the encoding layer includes an input embedding layer and a position embedding layer, wherein the input embedding layer is used to map the input text into an input vector; the position embedding layer is used to construct a position vector of the input text, and the input vector and the position vector are spliced ​​together to form an input representation of the input sequence.

[0013] Preferably, the input vector includes a word vector and a segment embedding vector, and each character s in the input sequence i The input is represented as:

[0014] V i =E c +E s +E i

[0015] V i For character s i The input representation, E c For character s i The word vector, E s For character s i The segment embedding vector, E i For character s i The position vector of .

[0016] Preferably, the encoding layer includes a multi-head attention layer, a first residual & normalization layer, a feedforward network layer, and a second residual & normalization layer.

[0017] The multi-head attention layer is used to perform attention calculation on the input vector to obtain the attention sequence b i .

[0018] The first residual & normalization layer is used to combine the input vector with the attention sequence b i Add and get b' i , and b' i Perform layer normalization.

[0019] The feedforward network layer uses a linear rectification unit to normalize the layer b' i Perform a linear transformation.

[0020] The second residual & normalization layer is used to linearly transform b' i Perform layer normalization and convert the layer normalized b' i Mapped to a sequence of representations.

[0021] Preferably, the coding layer further comprises the following steps:

[0022] For each word in the input sequence, define each word position vector to be the same size as the word vector, and then sum the input vector and position vector of each word. The summation formula is as follows:

[0023]

[0024]

[0025] Among them, i represents the sequence number, 2i represents an even multiple of i, 2i1 represents one of the even multiples of i, PE (pos,2i) 、 They are respectively represented as the current position and the position vector of one of the dual numbers, pos represents the position, d model Represents a word model.

[0026] Preferably, training the BERT knowledge graph completion model includes:

[0027] Pre-training stage: Use a large amount of unlabeled data to pre-train the BERT knowledge graph completion model. In the pre-training stage, the masked word pre-training and next sentence prediction methods are combined to obtain the initial pre-training parameters.

[0028] Fine-tuning stage: Without changing the internal structure of BERT, an output layer is added after the encoding layer, and a small number of labeled triplets are used to adjust the network parameters of the BERT knowledge graph completion model; finally, the encoding result specially marked by [CLS] is used as the output of the model.

[0029] Preferably, the distance between the predicted probability and the expected probability is calculated using a cross entropy loss function, specifically:

[0030]

[0031]

[0032] in, is the expected probability, and is a two-dimensional real vector, which is the predicted probability, and and are the sets of valid and invalid triples respectively, Depend on Constructed; is the distance; (h, r, t) represents a triplet, h represents the distance, r represents the angle, t represents the time, represents the time conversion function, is the sequence representation of the model input; H is the hidden state size of the pre-trained BERT; is the classification layer weight.

[0033] A computer system comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0034] The present invention has the following beneficial effects:

[0035] The BERT knowledge graph completion method and system that integrates entity types in the present invention uses external knowledge of entity information, namely entity types, as a supplement to entity information, better obtains the semantic information of entities, improves the text enhancement representation at the input end of the model, uses the Transformer encoder with a multi-head attention mechanism to realize feature acquisition and sequence encoding, uses a small number of labeled triples for fine-tuning, effectively realizes the mining of implicit relationships, and achieves fast and accurate cleaning of multi-source heterogeneous data.

[0036] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 This is a simplified structural diagram of the BERT knowledge graph completion model integrating entity types according to a preferred embodiment of the present invention;

[0039] Figure 2 It is the input representation of the BERT knowledge graph completion model;

[0040] Figure 3 This is a schematic diagram of feature extraction based on the multi-head self-attention mechanism;

[0041] Figure 4 It is a schematic diagram of sequence encoding based on Transformer encoder. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways as defined and covered by the claims.

[0043] To address the scarcity of labeled triples for battlefield information in uncertain environments, this paper uses BERT to complete the knowledge graph for battlefield information in uncertain environments. Furthermore, entity types, as external knowledge, enhance entity semantics. Therefore, entity type information is incorporated into triples as input to the model to enhance textual semantics.

[0044] Specifically, the BERT knowledge graph completion method integrating entity types in the present invention includes the following steps:

[0045] 1. Construction of BERT Knowledge Graph Completion Model with Entity Type Integration

[0046] Inspired by the KG-BERT model, this paper proposes a BERT knowledge graph completion framework that integrates entity types to model triples, which references the original implementation of the multi-layer bidirectional Transformer encoder similar to Vaswani et al., such as Figure 1 As shown in the figure, the model consists of three parts: input layer, encoding layer, and output layer. The input layer represents the head entity, head entity type, relationship, tail entity, and tail entity type as a sequence, and uses the sentence / word consisting of the five sequences as the input of BERT. The encoding layer uses a bidirectional Tansformer encoder to encode the input sequence and uses a multi-head attention mechanism to splice the calculation results of text representations at different levels. The output layer uses the sigmoid function to calculate the score of the triple. Finally, the distance between the model's predicted probability and the expected probability is calculated. The final hidden state of [CLS] is used as the deep representation of the input sequence and the output of the model.

[0047] The BERT-KGC model uses triples in the knowledge graph as input to introduce external knowledge. Entity types are applied to the model input as text semantic enhancement to obtain supplementary information about entities. In order to obtain a vector representation of a sequence, each input sequence of the BERT-KGC model always starts with a special classification label [CLS] and contains 5 sentences or words to form a sequence. The input of the head and tail entities can be entity description sentences or the name of the entity itself. The first sentence representation contains the words marked as The second sentence contains the head entity of The third sentence indicates that the entity contains the header of The fourth sentence contains the relationship marked as The fifth sentence contains the tail entity marked as The input for these sequences can be entity / entity type / relationship description sentences or entity / entity type / relationship names themselves. The five texts / words are marked with [SEP] to distinguish them.

[0048] In order to better capture word order information, BERT can represent an input sequence by summing the corresponding token embeddings, segment embeddings, and position embeddings. The input sequence representation can be visualized as Figure 2 .

[0049] For token embedding, for Chinese text, this study uses word embedding, and each input Chinese character is represented as a word vector E c .

[0050] For the segment embedding E s , the words in the same sentence have the same segment embedding, and the words in different sentences have different segment embeddings. The segment embedding is used to assist the model in distinguishing the position vector representation of the words in each sentence. This chapter represents different segment embeddings as the following tokens: The segment embedding of the word contained in the head entity sentence is the position vector E h , the segment embedding of the word contained in the relation is the position vector E r , the segment embedding of the word contained in the tail entity is the position vector E t , the segment embeddings of the words contained in the sentences describing the head entity and tail entity types are the position vectors E hl and E tl .

[0051] For position embedding, since the order of the input sequence cannot be encoded by the Transformer's multi-head attention mechanism, in order to understand the order information of each word in the sequence, each input sequence is merged, and each word at each position corresponds to the corresponding position embedding. The corresponding position of each word is represented as a position vector E i A vector is learned at each position to encode the information of the sequence order, allowing BERT to learn the sequential features of the input.

[0052] Finally, the input representation is recorded as Where m is the length of the sequence, that is, the sum of the lengths of the five sentences or words input by the model. m, the classification label (1 [CLS]) and the separation marker (4 [SEP]) form a sequence S of length m+1+4. Each character s in S i The input is represented by V i =E c +E s +E i .

[0053] BERT is based on a multi-layer, bidirectional Transformer encoder. Each encoder uses a multi-headed attention mechanism to incorporate contextual information around words and establish the strength (weight) of connections between words. A simple linear model can be directly stacked on top of BERT and fine-tuned to integrate with specific tasks. The Transformer completely eliminates recursive and convolutional networks, and its global information can be acquired through parallel training of the Attention mechanism. To focus on syntactic or semantic features at different positions in a sentence, the Transformer uses a multi-headed self-attention mechanism to capture semantic features between words in a sentence, thereby improving model performance.

[0054] Feature acquisition based on multi-head self-attention mechanism. Transformer adopts multi-head attention mechanism, which uses stacked heads to expand the scope of information acquisition. Each head focuses on the information within its own range, and multiple heads work together to achieve full attention to the information within all ranges, while effectively realizing parallel computing. The structure of the multi-head self-attention mechanism is as follows: Figure 3 As shown, let Q represent the query, d q Indicates the feature dimension of the query; K represents the key, d k Represents the characteristic dimension of the key; V represents the value, d v Represents the feature dimension of Value; q represents the query sequence, k represents the key sequence, and v represents the value sequence.

[0055] Use linear projection to project the query, key and value, and then calculate the similarity. This process is iterated A times to get d q , d k and d v Dimension, each linear transformation uses a different parameter matrix W, then, on each Query, Key and Value result, perform the attention function to get d v The dimensional results are concatenated and then projected to produce the output value of the multi-head attention.

[0056] Each q only operates with the k in the same position, that is, q i,1 Only with k i,1 and k j,1 Perform the operation, i.e. q i,1 =W q, 1 q i , and get the corresponding Then, we can calculate b with the corresponding v. i,1 …b i,6, and then reduce the dimension to get b i As can be seen, multiple heads work together to achieve full attention to all information within the range, and by repeating the calculation h times, the model focuses on key features. The calculation process is shown in Equations (1) and (2). In the multi-head attention mechanism, each head performs its own duties and focuses on different points. The attention matrices obtained by multiple heads are spliced ​​and linearly transformed.

[0057]

[0058]

[0059] Among them, Concat represents the splicing operation, head i W represents the dot product self-attention calculation result after the Q, K and V of the i-th sub-layer pass through the linear projection layer, and W represents the weight of the linear projection layer passed by Q, K and V of the i-th sub-layer in turn; 0 Represents the weight of the linear projection layer after the concatenation result.

[0060] Sequence encoding based on Transformer encoder. The BERT-KGC model proposed in this study is based on the original architecture of Transformer. In the BERT-KGC model, Transformer is used to encode the character representation in the triple input sequence that integrates entity type information. Its structure is as follows Figure 4 .

[0061] The Transformer encoder consists of six identical layers, each of which is divided into an upper and lower part. The upper part is a fully connected feedforward network, while the lower part is a multi-head attention mechanism. A residual connection and a normalization layer are added to the periphery of each of the upper and lower parts.

[0062] (1) Input layer: After the model input is accepted by the encoder, it is mapped into the input vector a by the input embedding layer i , plus the position vector e i Then an input sequence is generated as the input of the multi-head attention mechanism layer.

[0063] (2) Multi-head attention layer: In the multi-head attention mechanism layer, the input vector is calculated to obtain b i sequence.

[0064] (3) Residual & Normalization Layer: In the residual link layer, the input sequence a of the multi-head attention mechanism layer is converted to i and output b i Add and get b' i , and b' iPerform layer normalization processing, and the normalized output of each sublayer is LayerNorm(x+Sublayer(x)), where Sublayer(x) is the sublayer output result.

[0065] (4) Feedforward network layer: This is a fully connected feedforward neural network. For different positions, corresponding fully connected networks are set up. The positions are independent of each other and the parameters are also different. At the same time, two linear transformations are used to achieve parallel calculations. The activation function of the linear transformation uses the rectified linear unit (ReLU), see formula (3).

[0066] FFN(x)=max(0,xW1+b1)W2+b2 (3)

[0067] Among them, W1 and W2, b1 and b2 represent the weights and biases of the two linear layers respectively.

[0068] After each input vector is processed in the feedforward network layer, the residual connection and layer normalization are performed again. Finally, the input sequence is mapped into a representation sequence, which is continuous.

[0069] (5) Position vector: The position information of the word is not considered in the Self Attention layer. For each word, the same operation is performed on the words adjacent to it or the words farther away from it. This is obviously not what we want. Position embedding is added to the word embedding. In the following operations, the word order information and position information can be extracted through the distance between the position vector and the word. In order to consider the word order in the input sequence, position embedding is introduced to number the position of each word. Each position has a unique position vector e i , define the position vector to be the same size as the word vector, and then sum the input embedding and the position embedding, see Equations (4) and (5).

[0070]

[0071]

[0072] Among them, d model is the dimension of the system model, pos represents the position information;

[0073] 2. Training of BERT Knowledge Graph Completion Model

[0074] The implementation of the BERT framework consists of two steps: pre-training and fine-tuning. First, pre-training is performed using a large amount of unlabeled data, known as unsupervised pre-training. BERT's initialization relies on parameters from the pre-training process. Using unlabeled data for pre-training significantly reduces corpus overhead. In downstream tasks, a small number of labeled triplets are used to adjust the initialization parameters, known as supervised fine-tuning.

[0075] The main idea of ​​the knowledge graph completion task based on pre-training and fine-tuning is: first, a large amount of unlabeled data is used for pre-training. In the pre-training stage, the masked word pre-training and the next sentence prediction method are combined to obtain the initial pre-training parameters; then, without changing the internal structure of BERT, an output layer is added after the encoding layer, and a small number of labeled triplets are used for parameter adjustment; finally, the encoding result specially marked by [CLS] is used as the output of the model.

[0076] Based on unsupervised BERT model pre-training, this approach combines the BERT model's masked word prediction training method (Masked LM) with the next sentence prediction method (Predict the next sentence) to achieve better pre-training results. Since most parameters are sufficiently capable of extracting higher-level features after pre-training, more parameters are retained in downstream applications, and parameter adjustments are performed within a smaller range, thereby improving model speed.

[0077] During the pre-training phase, the trained BERT model can capture high-level, abstract features in sentences. During the fine-tuning phase, BERT's internal structure does not need to be altered; instead, a specific output network layer is added above the encoding layer to implement the task output. This involves using a small amount of annotated data and all the parameters from the BERT pre-training phase to train the model's output layer parameters. This study considers the knowledge graph completion problem of battlefield information in uncertain environments as a classification problem. Five sentences consisting of triples and head and tail entity types are used as model inputs. The encoded vectors are used as sentence representations, and the final hidden state encoding result of [CLS] is used as the model output.

[0078] Use the final hidden state of the first token [CLS] (i.e., the encoding result of Transformer) as the deep representation of the input sequence to obtain the sequence representation of the model input, and use To express, represents the final hidden vector of the i-th input, where H is the hidden state size of the pre-trained BERT. The state C associated with the final hidden layer special token [CLS] is used as the total sequence representation of the triplet score.

[0079] The only new parameters introduced during the tuning phase are the classification layer weights The sigmoid function takes a continuous value between 0 and 1, which can sensitively perceive the impact of subtle changes in text on the results. Therefore, the sigmoid function is used as the activation function in the output layer, and the score function of the triple (h, r, t) is expressed as formula (6).

[0080]

[0081] in, is a two-dimensional real vector, and

[0082] and are the sets of valid and invalid triples respectively, Depend on Constructed, where The pseudo triples in The valid triples h or t in Randomly replace an entity in Finally, the cross entropy loss function is used to describe the distance between the actual and expected output probabilities, as shown in Equation (7).

[0083]

[0084] in, is the expected output probability, and

[0085] After using the cross entropy loss function to achieve the distance between the actual and expected output probabilities, it is determined whether the distance exceeds a preset threshold. If the distance exceeds the threshold, the network parameters of the completion model are adjusted.

[0086] In addition, in this embodiment, a computer system is also disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0087] In summary, the present invention adopts knowledge graph completion technology to propose a BERT knowledge graph completion method that integrates entity types. It uses the external knowledge of entity information, namely the entity type, as a supplement to the entity information, to better obtain the semantic information of the entity, improve the text enhancement representation at the input end of the model, and use the Transformer encoder with a multi-head attention mechanism to realize feature acquisition and sequence encoding. It uses a small number of labeled triplets for fine-tuning to effectively realize the mining of implicit relationships.

[0088] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A BERT knowledge graph completion method integrating entity types, characterized in that: The following steps are involved: Constructing a BERT knowledge graph completion model: The BERT knowledge graph completion model includes an input layer, an encoding layer, and an output layer. The input layer is used to construct a representation sequence of the head entity, head entity type, relationship, tail entity, and tail entity type of the input text; and splicing the representation sequence of the head entity, head entity type, relationship, tail entity, and tail entity type into an input sequence; the encoding layer is used to encode the input sequence, extract semantic features at different levels of the encoded input sequence, and splice the semantic features at different levels; the output layer is used to calculate the predicted probability of the spliced ​​semantic features, and calculate the distance between the model predicted probability and the expected probability to adjust the network parameters of the BERT knowledge graph completion model; Training the BERT knowledge graph completion model, and using the trained BERT knowledge graph completion model to perform classification prediction on the input text; The encoding layer uses a bidirectional Tansformer encoder to encode the input sequence, and uses a multi-head attention mechanism to extract semantic features at different levels of the encoded input sequence, and then splices the semantic features at different levels; the output layer uses a sigmoid function to calculate the predicted probability of the spliced ​​semantic features, and calculates the distance between the model's predicted probability and the expected probability, and determines whether the distance exceeds a preset threshold. If the distance exceeds the threshold, the network parameters of the completion model are adjusted; the output layer uses the final hidden state of [CLS] as the deep representation of the input sequence and the output of the model; The encoding layer includes a multi-head attention layer, a first residual & normalization layer, a feedforward network layer, and a second residual & normalization layer; The multi-head attention layer is used to perform attention calculation on the input vector to obtain the attention sequence b i ; The first residual & normalization layer is used to combine the input vector with the attention sequence b i Add to get b ′i , and b ′i Perform layer normalization; The feedforward network layer uses a linear rectification unit to normalize the layer b ′i Perform linear transformation; The second residual & normalization layer is used to linearly transform b ′i Do layer normalization and normalize the layer b ′i Mapped to a representation sequence; Training the BERT knowledge graph completion model includes: Pre-training phase: Use a large amount of unlabeled data to pre-train the BERT knowledge graph completion model. In the pre-training phase, the masked word pre-training and next sentence prediction methods are combined to obtain the initial pre-training parameters. Fine-tuning stage: Without changing the internal structure of BERT, an output layer is added after the encoding layer, and a small number of labeled triplets are used to adjust the network parameters of the BERT knowledge graph completion model; finally, the encoding result specially marked by [CLS] is used as the output of the model.

2. The BERT knowledge graph completion method for integrating entity types according to claim 1 is characterized in that: The input sequence is represented as: Among them, S represents the input sequence, [CLS] is the classification label, is the representation sequence of the head entity; A sequence representing the head entity type; is a sequence of representations of relations; is the representation sequence of the tail entity; It is a representation sequence of the tail entity type, and [SEP] is a separator.

3. The BERT knowledge graph completion method for integrating entity types according to claim 1 is characterized in that: The encoding layer includes an input embedding layer and a position embedding layer. The input embedding layer is used to map the input text into an input vector; the position embedding layer is used to construct a position vector of the input text and concatenate the input vector and the position vector together to form an input representation of the input sequence.

4. The BERT knowledge graph completion method for integrating entity types according to claim 3 is characterized in that: The input vector includes a word vector and a segment embedding vector. For each character s in the input sequence i The input is represented as: V i =E c +E s +E i V i For character s i The input representation, E c For character s i The word vector, E s For character s i The segment embedding vector, E i For character s i The position vector of .

5. The BERT knowledge graph completion method for integrating entity types according to claim 1 is characterized in that: The coding layer further comprises the following steps: For each word in the input sequence, define each word position vector to be the same size as the word vector, and then sum the input vector and position vector of each word. The summation formula is as follows: Among them, i represents the sequence number, 2i represents an even multiple of i, 2i1 represents one of the even multiples of i, PE (pos,2i) 、 They are respectively represented as the current position and the position vector of one of the dual numbers, pos represents the position, d model Represents a word model.

6. The BERT knowledge graph completion method for integrating entity types according to claim 1, characterized in that: The distance between the predicted probability and the expected probability is calculated using the cross entropy loss function, which is: in, is the expected probability, and is a two-dimensional real vector, which is the predicted probability, and and are the sets of valid and invalid triples respectively, Depend on Constructed; is the distance; (h, r, t) represents a triplet, h represents the distance, r represents the angle, t represents the time, represents the time conversion function, is the sequence representation of the model input; H is the hidden state size of the pre-trained BERT; is the classification layer weight.

7. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images