A knowledge graph entity recognition method based on ELECTRA and neural network model

Abstract

The present invention discloses a knowledge graph entity recognition method based on ELECTRA and a neural network model, comprising: (1) text preprocessing, adding [cls] and [seq] tags to the input text as the beginning and end of a sentence; (2) processing based on the ELECTRA model, randomly masking some entities through a Mask mechanism, predicting the masked content by a generator, and judging whether the predicted content comes from the original data by a discriminator, and outputting words with part-of-speech tags; (3) bidirectional long short-term memory network BiLSTM model processing, enhancing sentence semantic information, obtaining contextual semantic features of each word, and improving the accuracy of entity recognition; (4) CRF model layer processing, outputting the label set with the highest probability by calculating the state transition feature function and label score. The method can effectively improve the accuracy of entity recognition, significantly reduce the energy consumption of model training, and greatly improve the training speed, and is suitable for entity recognition tasks of large-scale knowledge graphs.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a knowledge graph entity recognition method based on ELECTRA and a neural network model. Background Art

[0002] At present, people have entered the era of digitalization and informatization. While the amount of information is growing explosively, the quality of information is becoming increasingly prominent. Poor-quality data usually has quality problems such as inaccuracy, missing data, obsolete data, and redundant data, which greatly reduces the availability of information. It not only fails to provide users with usable knowledge, but also leads to wrong decisions. Among the multiple dimensions of data quality, entity identity is one of the important dimensions. To solve the entity identity problem of poor-quality data, entity recognition is particularly important. Existing entity recognition technologies usually use dictionaries and rules, statistical machine learning, and deep learning methods for entity recognition. The main limitations are as follows: (1) Although the existing entity recognition methods based on dictionaries and rules are simple to use, the establishment of dictionaries and rule bases requires a lot of manpower costs, and there are shortcomings such as poor transferability; (2) The existing entity recognition based on statistical machine learning lacks consideration of context information and is not suitable for large sample data; (3) The existing entity recognition based on deep learning usually has problems such as insufficient consideration of semantic context information, easy gradient vanishing and gradient explosion, and requires a large amount of data labeling. Summary of the invention

[0003] Technical issues to be solved

[0004] In order to solve the problem of named entity recognition in text information, the present invention proposes a knowledge graph entity recognition method based on ELECTRA and a neural network model, thereby solving the aforementioned problems existing in the prior art.

[0005] Technical Solution

[0006] A knowledge graph entity recognition method based on ELECTRA and a neural network model, characterized by the following steps:

[0007] Step 1: Text preprocessing. To avoid the problem of blurred entity boundaries, add the [cls] tag as the beginning of the sentence and the [seq] tag as the end of the sentence to the input text information.

[0008] Step 2: Based on the processing of the ELECTRA model, the sentence beginning and ending tags added in step 1 are used as the input of the ELECTRA model. Based on the Mask mechanism, some entities are randomly masked to obtain random Mask text. The generator predicts the masked content. Assuming that the predicted masked content is xt, the output sentence predicted by the generator flows to the discriminator. The discriminator determines whether the predicted xt comes from the data itself rather than generated by the generator, and outputs the word with part-of-speech tag;

[0009] Step 3: Based on the processing of the BiLSTM model, the output of step 2 is used as the input of the BiLSTM layer. Through the BiLSTM layer, the semantic information of the sentence can be enhanced, and each word in the sentence obtains the semantic features of the context;

[0010] Step 4: CRF model layer processing, the output of step 3 is used as the input of the CRF model layer, the state transition feature function is obtained, the score of the label is calculated, and the output sequence label is the label set with the largest output probability.

[0011] Beneficial Effects

[0012] The present invention proposes a knowledge graph entity recognition method based on ELECTRA and a neural network model. To avoid the problem of fuzzy entity boundaries, the output text data is preprocessed, a [cls] tag is added as the beginning of a sentence, and a [seq] tag is added as the end tag of a sentence. The preprocessed data is processed by the ELECTRA model, and some entities are randomly masked based on the Mask mechanism. The masked content is predicted by a generator, and the predicted sentence is judged by a discriminant to determine whether it comes from the data itself rather than generated by the generator, and the words with part-of-speech tags are output, and the words with part-of-speech tags are processed by a CRF model layer, and the scores of the labels are calculated, and the output sequence labels are the label set with the maximum output probability. The present invention preprocesses the output text data, effectively avoiding the problem of fuzzy entity boundaries, the preprocessed data is processed by the ELECTRA model layer, and the semantic information of the sentence is further enhanced by the BiLSTM layer, and the scores of the labels are calculated by the CRF model layer, and the output sequence labels are the label set with the maximum output probability. The method of the present invention can not only effectively improve the accuracy of entity recognition, but also help to reduce energy consumption and improve training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 System overall framework diagram

[0014] Figure 2 ELECTRA structure diagram

[0015] Figure 3 BiLSTM Flowchart DETAILED DESCRIPTION

[0016] The present invention will now be further described with reference to the embodiments and the accompanying drawings:

[0017] This paper proposes a knowledge graph entity recognition method based on ELECTRA and neural network model. The overall framework diagram of knowledge graph entity recognition is shown in Figure 1 The technical solutions adopted to solve its technical problems include the following:

[0018] 1. Text preprocessing

[0019] First, the text data X = (X1 X2 ... Xn) is preprocessed by adding a [cls] tag as the beginning of a sentence and a [seq] tag as the end of a sentence. The result after processing is X' = (cls X1 X2 ... Xn seq).

[0020] 2. Processing based on the ELECTRA model

[0021] The generator and the discriminator are two major components of the ELECTRA model. The generator predicts the 15% of words that are mined, and the discriminator judges the predicted value. True is the original word, and false is the replacement word. The input to the generator is a mask mechanism that randomly masks some entities, but in order to solve the entity boundary problem, the present invention adopts the following operations: first read the data set, then delete the extra spaces and other special symbols in the data set, and finally separate a certain paragraph of the data set with cls and seq.

[0022] After receiving the random Mask text, the generator generates a specific x t The probability calculation formula is shown in formula (1). Where x represents the input text, h(x) represents the context vector transformed from x, and x t Indicates the location where there is a mask.

[0023]

[0024] The output sentence predicted by the generator flows to the discriminator, which discriminates the predicted x t Whether it comes from the data itself rather than the generator. This is a binary classification task, so the loss is the cross entropy loss, and the calculation formula is shown in formula (2).

[0025] D(x, t) = sigmoid(w T h D (x) t ) (2)

[0026] The generator and discriminator have a loss effect on the ELECTRA model respectively, so the loss functions of the two are summed to obtain the final loss function of the ELECTRA model. The calculation formula is shown in Equation 3.

[0027]

[0028] In formula 3, The calculation formula is as follows: The calculation formula is as follows:

[0029]

[0030]

[0031] In formula 4, x i represents the i-th word in the mask list, x masked represents the word that replaces x at the i-th position. In formula 5, represents the original unreplaced sequence, represents the original replaced sequence, where represents the loss value of x being replaced at position t in the X sequence.

[0032] The pre-annotated text data in BIO format is input into the ELECTRA module, and after being processed by the module, the output is words with part-of-speech tags.

[0033] 3. Processing based on the BiLSTM model

[0034] The essence of LSTM is a recurrent neural network of unidirectional long short-term memory units, which is designed to alleviate the gradient vanishing problem during training. LSTM is composed of the input word at time t, cell state, temporary cell state, hidden layer state, forget gate, memory gate and output gate.

[0035] Calculation of forget gate:

[0036] f t =σ(W f ·[h t-1 , x t ]+b f ) (6)

[0037] Memory gate calculation:

[0038] i t =σ(W i ·[h t-1 , x t ]+b i ) (7)

[0039]

[0040]

[0041] Output gate calculation:

[0042] σ t =σ(W o ·[h t-1 , x t ]+b o ) (10)

[0043] h t =o t *tanh(C t ) (11)

[0044] However, LSTM can only obtain information about the previous context of a sentence, but not about the following context. Therefore, the LSTM that obtains the previous context is reversed to obtain the following context information, and then the forward LSTM is combined with the reverse LSTM to form a BiLSTM. For example, if the input sentence is "cold and fever", the semantic information of the sentence can be enhanced through the BiLSTM layer, and each word in it obtains the semantic features of the context. The process is as follows: Figure 3 shown.

[0045] 4.CRF model layer processing

[0046] Conditional random fields (CRFs) can solve the problem of entity annotation prediction errors and fully consider the combination relationship between labels. The transition matrix in the label represents the probability of transitioning from one state to another. It is used to verify whether the label combination conforms to the BIO annotation principle, thereby determining whether the combination between adjacent annotated entities is correct. Therefore, in CRF, what is considered is not a single optimal value, but the global optimal value of the sentence composed of all words.

[0047] For a sentence input X=(X1,X2,X3,....,X n ), marked with a tag sequence Tag, its score is defined as:

[0048] Sorce=ES+TS (12)

[0049] In the BiLSTM module, the ES calculation result is directly input into the CRF module. The calculation formula of ES is:

[0050]

[0051] Where i represents the index of the current node in X, L represents the total number of feature functions on the current node i, and T L represents the node characteristic function, U L Denotes the characteristic function TL The corresponding weight coefficient is used to measure the trust of the characteristic function.

[0052] The second step in CRF is to find the state transition feature function, which is essentially the probability of a state y1 transferring to another state y2. It is calculated as follows:

[0053]

[0054] Where i represents the index of the current node in X, k represents the total number of feature functions on the current node i, S k represents the local feature function, which is related to the current position and the previous position, M k Indicates S k The corresponding weight coefficient.

[0055] If P is the score matrix output by the BiLSTM network, then the score of the jth label of the i-th word is P i，j For labels y=(y1,y2,y3,...,y n ), its score formula is:

[0056]

[0057] Z(x)=Σ y` exp(Score(x,y)) (16)

[0058] Since the network structure is trained using the maximum likelihood method, where Z(x) is the sum of the exponential scores of all label sequences y, the corresponding loss function is as follows:

[0059] logp(y|x)=Score(x,y)+logZ(x) (17)

[0060] The final output sequence label is the label set with the highest output probability.

Claims

1. A knowledge graph entity recognition method based on ELECTRA and neural network model, characterized in that Here are the steps: Step 1: Text preprocessing. To avoid the problem of blurred entity boundaries, add the [cls] tag as the beginning of the sentence and the [seq] tag as the end of the sentence to the input text information. Step 2: Based on the processing of the ELECTRA model, the text data obtained in step 1 is added with the sentence start marker [cls] and the sentence end marker [seq], and the marked text data is used as the input of the ELECTRA model. Based on the Mask mechanism, some entities are randomly masked to obtain random Mask text. The generator predicts the masked content. Assume that the predicted masked content is x t , the output sentence predicted by the generator flows to the discriminator, and the discriminator judges the predicted x t Whether it comes from the data itself rather than the generator, and outputs words with part-of-speech tags; Step 3: Based on the processing of the BiLSTM model, the output of step 2 is used as the input of the BiLSTM layer. The semantic information of the sentence is enhanced through the BiLSTM layer, and each word in the sentence obtains the semantic features of the context; Step 4: CRF model layer processing, the output of step 3 is used as the input of the CRF model layer, the state transition feature function is obtained, the score of the label is calculated, and the output sequence label is the label set with the largest output probability.

2. According to the knowledge graph entity recognition method based on ELECTRA and neural network model in claim 1, the text preprocessing in step 1 is specifically as follows: preprocessing the text data X=(X1 X2 ... Xn), adding the [cls] tag as the beginning of the sentence, adding the [seq] tag as the end of the sentence, and the processed result is X'=(cls X1 X2 ... Xnseq).

3. The knowledge graph entity recognition method based on ELECTRA and neural network model according to claim 1 is characterized in that: The generator and the discriminator are the two major components of the ELECTRA model. The generator predicts the 15% of words that are mined, and the discriminator distinguishes the predicted values. True is the original word, and false is the replacement word. The input to the generator is the mask text that randomly masks some entities. The following operations are used to solve the entity boundary problem: first read the data set, then delete the extra spaces, carriage returns, tabs, line breaks and other blank characters in the data set, and finally separate a paragraph of the data set with cls and seq; After receiving the random Mask text, the generator generates a specific x t The probability calculation formula is shown in formula (1), where h(x) represents the context vector transformed from x, and x t Indicates the location where there is a Mask; Here P G (x t |x) means that given an input sequence x, the generator predicts that the content at the masked position t is x t The conditional probability of x represents the input text sequence, e(x t ) T Represents the word vector x at position t t The transposed matrix, h G (x) t represents the hidden layer representation of the generator for the input sequence x at position t, x′ represents the candidate word in the vocabulary, and e(x′) T Represents the word vector transposed matrix of the candidate word x′; The output sentence predicted by the generator flows to the discriminator, which discriminates the predicted x t Whether it comes from the data itself rather than generated by the generator, this is a binary classification task, so the loss is the cross entropy loss, and the calculation formula is shown in formula (2): D(x,t)=sigmoid(w T h D (x) t ) (2) Where D(x, t) represents the probability that the discriminator determines whether the word at position t is the original word, and w T represents the parameter matrix transpose of the discriminator, h D (x) t represents the hidden layer representation of the discriminator for the input sequence x at position t; The generator and discriminator have a loss effect on the ELECTRA model respectively, so the loss functions of the two are summed to obtain the final loss function of the ELECTRA model. The calculation formula is shown in Formula 3: where θ G represents the parameters of the generator, θ D represents the parameters of the discriminator, represents the loss function of the generator, λ represents the weight coefficient of the discriminator loss function, Represents the loss function of the discriminator: In formula 3, The calculation formula is as follows: The calculation formula is formula (5): In formula 4, x i represents the i-th word in the mask list, x masked represents the word that replaces x at the i-th position, M represents the set of masked positions, represents the mathematical expectation, P G (x i |x masked ) represents the generator. In formula 5, n represents the sequence length, x c represents the replaced sequence, t represents the position index in the sequence, and D(xc, t) represents the discriminator's response to the sequence x c The predicted probability at position t, represents the original unreplaced sequence, represents the original replaced sequence, where represents the loss value of x replaced at position t in the X sequence; The pre-annotated text data in BIO format is input into the ELECTRA module, and after being processed by the module, the output is words with part-of-speech tags.

4. The knowledge graph entity recognition method based on ELECTRA and a neural network model according to claim 1, wherein the processing based on a bidirectional long short-term memory network BiLSTM model in step 3 is specifically that LSTM is composed of an input word at time t, a cell state, a temporary cell state, a hidden layer state, a forget gate, a memory gate, and an output gate; Calculation of the forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f ) (6) where f t represents the output value of the forget gate at time t, σ represents the sigmoid activation function, W f represents the weight matrix of the forget gate, h t-1 represents the hidden state at time t-1, x t represents the input vector at time t, b f represents the bias term of the forget gate, [h t-1 , x t ] means concatenating the hidden layer state and the input vector; Memory gate calculation: i t =σ(W i ·[h t-1 ,x t ]+b i ) (7) where i t represents the input gate output value at time t, W i represents the weight matrix of the input gate, b i represents the bias term of the input gate; in represents the candidate cell state at time t, tanh represents the hyperbolic tangent activation function, W C The weight matrix representing the cell state, b C Bias term representing the cell state: Among them C t represents the cell state at time t, C t-1 represents the cell state at time t-1, * represents element-by-element multiplication operation, Calculation of output gate: the t =σ(W o ·[h t-1 ,x t ]+b o ) (10) Among them t represents the output value of the output gate at time t, W o represents the weight matrix of the output gate, b o Represents the bias term of the output gate: h t =o t *tanh(C t ) (11) where h t represents the hidden layer state output at time t, tanh represents the hyperbolic tangent activation function, C t represents the cell state at time t; However, LSTM can only obtain information about the previous context of the sentence, but not the following context. Therefore, the LSTM that obtains the previous context is reversed to obtain the following context information, and then the forward LSTM is combined with the reverse LSTM to form a BiLSTM.

5. According to the knowledge graph entity recognition method based on ELECTRA and neural network model in claim 1, the CRF model layer processing in step 4 is specifically as follows: the conditional random field CRF solves the problem of entity labeling prediction errors, fully considers the combination relationship between labels, and the transfer matrix in the label represents the probability of one state transferring to another state. It is used to verify whether the label combination complies with the BIO labeling principle, thereby judging whether the combination between adjacent labeled entities is correct. Therefore, in CRF, what is considered is not a single optimal value, but the global optimal value of the sentence composed of all words; For a sentence input X = (X1, X2, X3, ..., X n ), marked with a tag sequence Tag, its score is defined as: Score=ES+TS (12) In the BiLSTM module, the ES calculation result is directly input into the CRF module. The calculation formula of ES is: in, i represents the index of the current node in X, L represents the total number of feature functions on the current node i, T L represents the node characteristic function, y i represents the label value of the i-th position in the sequence X, U L Denotes the characteristic function T L The corresponding weight coefficient is used to measure the trust of the characteristic function. The second step in CRF is to find the state transition feature function, which is essentially the probability of a state y1 transferring to another state y2. It is calculated as follows: Where i represents the index of the current node in X, k represents the total number of feature functions on the current node i, S k represents the local feature function, which is related to the current position and the previous position, M k Indicates S k The corresponding weight coefficient, y i-1 represents the label value of the i-1th position in the sequence X, If P is the score matrix output by the BiLSTM network, then the score of the jth label of the i-th word is P i，j , for labels y=(y1,y2,y3,...,y n ), its score formula is: Where Score(x, y) represents the overall score of the input sequence x and the label sequence y: Z(x)=∑ y exp(Score(x,y)) (16) Since the BiLSTM network structure is trained using the maximum likelihood method, where Z(x) is the exponential sum of the scores of all label sequences y, the corresponding loss function is as follows: logp(y|x)=Score(x,y)+logZ(x) (17) The final output sequence label is the label set with the highest output probability.

Images