A sentence-level relation extraction method, device and storage medium

Abstract

The present invention discloses a sentence-level relation extraction method, device and storage medium, including obtaining a sentence, inserting special tags at both ends of each entity, inputting a pre-trained language model to obtain the word vector of each word in the sentence; using a plurality of randomly initialized relation vectors to calculate the relevance of each word vector, and further calculating the binary classification loss; using average pooling to calculate the sentence vector and entity vector, using maximum pooling to calculate the relation vector, and inputting these special inputs into a multi-classifier to obtain the multi-classification loss; hiding the entity in the sentence with a certain probability to obtain another sentence, and measuring the consistency loss of the output distribution of the two sentences; and jointly training all the losses in the above steps. The present invention can effectively extract key information in the text, capture the connection between entities in the text, and thus achieve good relation extraction performance.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a sentence-level relationship extraction method, device and storage medium. Background Art

[0002] In recent years, artificial intelligence has become an important driving force for a new round of scientific and technological revolution and industrial transformation. Natural language processing (NLP), as an important research direction of artificial intelligence, is the key to realizing the transition from perception to cognition of artificial intelligence. Relation extraction is a core task in the field of natural language processing. It mainly realizes the extraction and recognition of semantic relationships expressed between entity pairs in texts. It is also the basis for building knowledge graphs. It is widely used in information retrieval, machine reading comprehension, human-computer dialogue, personalized recommendation, intelligent search and other fields. With the widespread application of deep learning in the field of artificial intelligence, research results have shown that deep learning can capture effective feature information using large amounts of data. Many researchers have also applied deep learning technology to relationship extraction, and have made significant progress compared to traditional manual feature extraction methods.

[0003] Due to the diversity of textual expressions, relation extraction is a complex and difficult task. The extraction of relations between entity pairs depends largely on the model's ability to understand the semantics in the text, which is also extremely challenging in natural language processing. With the development of deep learning, the current relation extraction based on deep learning methods can be divided into two categories: one is based on traditional neural network models such as convolutional neural networks and recurrent neural networks. Due to the limitations of model size and difficulty in obtaining training data, such models can only achieve limited results; the other is a relation extraction model based on pre-trained language models (such as BERT, GPT). The pre-trained model was first proposed in the field of images. It uses a large amount of training data to learn rich knowledge, and then combines transfer learning to migrate to other neural networks to obtain better results. A large number of experimental results show that pre-trained language models are already ahead of traditional neural networks in many natural language processing tasks.

[0004] Although many researchers at home and abroad have completed the task of relation extraction based on pre-trained language models, some experimental results show that there is still room for improvement in pre-trained language models and there is still a lot of potential to be explored. In relation extraction and even the entire field of natural language processing, there have been a lot of studies on word vectors. Undoubtedly, the quality of word vectors is related to model performance, but few people have explored and utilized the rich semantic information in labels (relations). Summary of the invention

[0005] In order to overcome the shortcomings and deficiencies of the prior art, the present invention provides a sentence-level relation extraction method, device and storage medium, specifically a relation extraction method based on relation embedding attention mechanism. The present invention utilizes the rich semantic information in the relation to enhance the extraction of key information in the text, thereby achieving better relation extraction performance.

[0006] The present invention adopts the following technical solution:

[0007] A sentence-level relation extraction method includes the following:

[0008] A sentence is obtained, wherein the sentence includes two entities, special tags are inserted at both ends of each entity, and a pre-trained language model is input to obtain a word vector for each word in the sentence;

[0009] Use random initialization to get the relation embedding, use the dot product attention mechanism with the word vector of each word to get its relevance, and further get the attention representation set of the sentence. Input each vector in the set into the binary classifier to get the prediction result, and calculate the binary classification loss;

[0010] Based on the attention representation set, the relationship vector is calculated using the maximum pooling, and the sentence vector and two entity vectors are obtained using the average pooling. The above four vectors are concatenated and input into the multi-classifier to obtain the multi-classification loss.

[0011] Hide the two entities in the sentence according to the preset probability to obtain another sentence S'; repeat the above steps for another sentence to obtain the binary loss and multivariate loss, and further obtain the consistency loss of the prediction results of the two sentences;

[0012] The binary loss, multivariate loss, and consistency loss are jointly trained, and the relationship extraction model is obtained when the loss is minimized.

[0013] Further, a sentence is obtained, wherein the sentence includes two entities, and special tags are inserted at both ends of each entity, specifically:

[0014] Get a sentence S, insert a special marker "[CLS]" at the beginning of the sentence, insert special markers "#" at both ends of the first entity, and insert special markers "$" at both ends of the second entity.

[0015] Furthermore, we use random initialization to get the relation embedding, and use the dot product attention mechanism with the word vector of each word to get its relevance, and further get the attention representation set of the sentence, which is as follows:

[0016] Randomly initialize to obtain relation embeddings, and use the dot product attention mechanism to get the relevance of each relation embedding with each word vector;

[0017]

[0018] e in the formula i represents the embedding of the i-th relation, h j The word vector representing the jth word is multiplied element by element using the obtained relevance, and the products are added to obtain the sentence attention representation set V = {v1, v2, ..., v m};

[0019]

[0020] Furthermore, each vector in the attention representation set is input into the binary classifier to obtain the prediction result, and the binary classification loss is calculated, which is:

[0021] Input each vector in the attention representation set into the binary classifier to obtain the probability of the relationship existing, and then use the cross entropy loss function according to the true label to obtain the binary loss L bin .

[0022] The true label is generated according to the following rule: for each relation r in the predefined relation set, the label is 1 if the relation exists, otherwise the label is 0.

[0023] Furthermore, the relationship vector is calculated based on the attention representation set using the maximum pooling, and the sentence vector is obtained using the average pooling. The two entity vectors are specifically:

[0024] The sentence vector is obtained according to the following formula:

[0025] h cls =W0[tanh(h0)]+b0

[0026] The two entity vectors are obtained according to the following formula:

[0027]

[0028]

[0029] The relationship vector is obtained according to the following formula:

[0030] h v =W2[tanh(MaxPooling(V))]+b2

[0031] Where i and j are the start and end subscripts of the first entity, respectively, and k and l are the start and end subscripts of the second entity, respectively.

[0032] Furthermore, the set probability is less than 0.5.

[0033] Furthermore, the consistency loss of the two sentence prediction results is obtained, where the consistency loss is:

[0034]

[0035] and Represent the multi-classification prediction results of sentences S and S' respectively.

[0036] Furthermore, the binary loss, multivariate loss, and consistency loss are jointly trained to obtain the optimal model:

[0037] L=0.5*(L mul +L′ mul +L bin +L′ bin )+L kl .

[0038] L mul , L bin , L′ mul , L′ bin They represent the multivariate loss of sentence S, the binary loss of sentence S, the multivariate loss of sentence S', and the binary loss of sentence S' respectively.

[0039] A storage medium stores a computer program, which implements the relationship extraction method when executed by a processor.

[0040] A device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the relationship extraction method is implemented when the processor executes the program.

[0041] Beneficial effects of the present invention:

[0042] (1) This invention improves the traditional method of treating relations as hard labels and makes full use of the rich semantics in relations to help the model extract key information;

[0043] (2) The present invention uses a method of jointly training auxiliary tasks and main tasks, which can learn better relationship embedding while enhancing model performance. The auxiliary task refers to a binary classification task, and the main task refers to a multi-classification task.

[0044] (3) The present invention constructs enhanced samples by means of mask entities to prevent model overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention.

[0046] Figure 2 Flow chart of the first three steps of the present invention;

[0047] Figure 3 It is a flow chart of the fourth step of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0049] Example 1

[0050] like Figure 1 and Figure 2 As shown, a sentence-level relation extraction method includes: 1) extracting information related to a specific relation in a text by learning relation embedding and using an attention mechanism; 2) designing an auxiliary task to improve model performance and jointly learn auxiliary relation embedding; 3) generating similar samples by hiding entities in the text with a certain probability as a means of data enhancement; 4) preventing the model from over-learning by minimizing the difference in probability distribution between samples and similar samples. The present invention can effectively extract key information in the text and capture the connection between entities in the text, thereby achieving good relation extraction performance.

[0051] The specific steps include:

[0052] The first step is to obtain a sentence S, which includes two entities. Special tags are inserted at both ends of each entity, and the pre-trained language BERT model is input to obtain the word vector H = {h1, h2, ..., h n},h i ∈R d , represents the word vector of the i-th word.

[0053] Specifically:

[0054] Get a sentence S = {w1,w2,…,w n , n is the number of words in the text, insert a special marker "[CLS]" at the beginning of the sentence, insert special markers "#" at both ends of the first entity, and insert special markers "$" at both ends of the second entity. For example, "[CLS]this machine is blowing#sand#into$molds$.", "sand" and "molds" are two entities.

[0055] In the second step, random initialization is used to obtain relational embedding, and the dot product attention mechanism is used with the word vector of each word to obtain its relevance, and then the attention representation set of the sentence is obtained. Each vector in the set is input into the binary classifier to obtain the prediction result, and the binary classification loss is calculated.

[0056] Specifically:

[0057] Randomly initialize to obtain relation embedding, let E = {e0,e1,…,e m},e i ∈R d , m is the number of relations, e i represents the embedding of the i-th relation, h j The word vector representing the jth word, each relation embedding is respectively associated with each word vector using the dot product attention mechanism to obtain the degree of relevance;

[0058]

[0059] Medium α ij Indicates the degree of relevance between the i-th relationship and the j-th word.

[0060] Use the obtained relevance to multiply the word vectors element by element, and add the products to get the attention representation set of the sentence;

[0061]

[0062] Repeat the operations of formula (1) and (2) for each relation, and obtain the set of relation attention representations V = {v1, v2, …, v m},v i ∈R d .

[0063] In order to strengthen the embedding of specific relations i We use the original labels to design a binary classification task and jointly train it with the main task. Each vector in V is input into the binary classifier to obtain the probability of detecting the existence of the relationship by the feature vector. in:

[0064]

[0065] Where W bin ∈R d is the parameter matrix, b bin is the bias, Represents the probability that the i-th relationship exists in the sentence.

[0066] According to the real label The corresponding binary loss can be calculated using the cross entropy loss function, which is expressed as:

[0067]

[0068]

[0069] in represents the binary loss of the i-th relationship, L binRepresents the total binary loss. Since the above processing method will result in the number of positive samples being much smaller than the number of negative samples, in formula (4) b The loss weight of the positive sample can be adjusted. It is an adjustable hyperparameter that enables the model to focus more on optimizing the positive samples during training.

[0070] The true label is generated according to the following rule: for each relation r in the predefined relation set, the label is 1 if the relation exists, otherwise the label is 0.

[0071] In the third step, the relationship vector is calculated using maximum pooling based on the attention representation set, and the sentence vector and two entity vectors are obtained using average pooling. The above four vectors are concatenated and input into the multi-classifier to obtain the multi-classification loss.

[0072] Specifically:

[0073] The output of the special tag "[CLS]" is used as the representation of the sentence; the average pooling output of the entity is used as the entity representation; the maximum pooling of the relation attention output is used as the relation representation. The formulas are respectively expressed as:

[0074] h cls =W0[tanh(h0)]+b0 (6)

[0075]

[0076]

[0077] h v =W2[tanh(MaxPooling(V))]+b2 (9)

[0078] Where i, j are the start and end subscripts of the first entity, respectively, and k, l are the start and end subscripts of the second entity, respectively. W0,W1,W2∈R d×d , b0, b1, b2 are biases. The maximum pooling operation for the relational attention representation V is defined as follows:

[0079] (v max ) i =Max{(v t ) i},t∈{0,1,…,s-1} (10)

[0080] h v =[(v max )1;(v max )2;…;(v max ) d ] (11)

[0081] Where i is the index of the feature, i.e., represents the i-th element of the vector.

[0082] Concatenate the four vectors and input them into the multi-classifier to get the classification result:

[0083]

[0084]

[0085] Calculate the multivariate cross entropy loss based on the true label Y:

[0086]

[0087] Where c is the number of categories.

[0088] The fourth step is Figure 3 As shown, the two entities in the sentence are hidden according to the preset probability to obtain another sentence; the above steps are repeated for the other sentence to obtain the binary loss and the multivariate loss, and further obtain the consistency loss of the prediction results of the two sentences.

[0089] Specifically:

[0090] Take the sentence "this machine is blowing sand into molds." as an example. The two entities are "sand" and "molds". The entity words are replaced with a special marker "[MASK]" with a certain probability p. p is an adjustable hyperparameter, which is generally not greater than 0.5. The replaced sentence may be: "this machine is blowing sand into [MASK].".

[0091] Then, the processed sentence S′ is calculated according to formulas (1)-(14) to obtain the corresponding binary loss L′ bin , multivariate loss L′ mul , predict distribution For distribution and Compute the Kullback-Leibler divergence:

[0092]

[0093] The fifth step is to jointly train the binary loss, multivariate loss, and consistency loss, and obtain the relationship extraction model when the loss is minimized.

[0094] The final loss of the model is as follows:

[0095] L=0.5*(L mul +L′ mul +Lbin +L′ bin )+L kl (16)

[0096] Use gradient descent to minimize the loss until the model converges and obtain the relationship extraction model.

[0097] The sixth step is to input the sentence to be extracted into the relation extraction model to achieve relation extraction performance.

[0098] Table 1: Performance comparison of BERT+REMB and mainstream relation extraction models on the Semeval-2010Task 8 dataset

[0099] Model Macro-F1 Att-Pooling-CNN 88.0 R-BERT 89.2 BERT+MTB 89.5 BERT+REMB 89.8

[0100] Table 1 shows the performance comparison between the BERT+REMB of the present invention and the mainstream relation extraction model on the Semeval-2010Task8 dataset. The present invention achieves the best results on this dataset.

[0101] Example 2

[0102] A storage medium stores a computer program, which implements the relationship extraction method when executed by a processor.

[0103] A sentence-level relation extraction method includes the following:

[0104] A sentence is obtained, wherein the sentence includes two entities, special tags are inserted at both ends of each entity, and a pre-trained language model is input to obtain a word vector for each word in the sentence;

[0105] Use random initialization to get the relation embedding, use the dot product attention mechanism with the word vector of each word to get its relevance, and further get the attention representation set of the sentence. Input each vector in the set into the binary classifier to get the prediction result, and calculate the binary classification loss;

[0106] Based on the attention representation set, the relationship vector is calculated using the maximum pooling, and the sentence vector and two entity vectors are obtained using the average pooling. The above four vectors are concatenated and input into the multi-classifier to obtain the multi-classification loss.

[0107] Hide the two entities in the sentence according to the preset probability to obtain another sentence S'; repeat the above steps for another sentence to obtain the binary loss and multivariate loss, and further obtain the consistency loss of the prediction results of the two sentences;

[0108] The binary loss, multivariate loss, and consistency loss are jointly trained, and the relationship extraction model is obtained when the loss is minimized.

[0109] Example 3

[0110] A device comprises a memory, a processor and the relationship extraction method stored in the memory and executable on the processor.

[0111] A sentence-level relation extraction method includes the following:

[0112] A sentence is obtained, wherein the sentence includes two entities, special tags are inserted at both ends of each entity, and a pre-trained language model is input to obtain a word vector for each word in the sentence;

[0113] Use random initialization to get the relation embedding, use the dot product attention mechanism with the word vector of each word to get its relevance, and further get the attention representation set of the sentence. Input each vector in the set into the binary classifier to get the prediction result, and calculate the binary classification loss;

[0114] Based on the attention representation set, the relationship vector is calculated using the maximum pooling, and the sentence vector and two entity vectors are obtained using the average pooling. The above four vectors are concatenated and input into the multi-classifier to obtain the multi-classification loss.

[0115] Hide the two entities in the sentence according to the preset probability to obtain another sentence S'; repeat the above steps for another sentence to obtain the binary loss and multivariate loss, and further obtain the consistency loss of the prediction results of the two sentences;

[0116] The binary loss, multivariate loss, and consistency loss are jointly trained, and the relationship extraction model is obtained when the loss is minimized.

[0117] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A sentence-level relation extraction method, characterized in that: These include: A sentence is obtained, wherein the sentence includes two entities, special tags are inserted at both ends of each entity, and a pre-trained language model is input to obtain a word vector for each word in the sentence; Use random initialization to get the relation embedding, use the dot product attention mechanism with the word vector of each word to get its relevance, and further get the attention representation set of the sentence. Input each vector in the set into the binary classifier to get the prediction result, and calculate the binary classification loss; Based on the attention representation set, the relationship vector is calculated using the maximum pooling, and the sentence vector and two entity vectors are obtained using the average pooling. The above four vectors are concatenated and input into the multi-classifier to obtain the multi-classification loss. Hide the two entities in the sentence according to the preset probability to obtain another sentence S'; repeat the above steps for another sentence to obtain the binary loss and multivariate loss, and further obtain the consistency loss of the prediction results of the two sentences; The binary loss, multivariate loss, and consistency loss are jointly trained. When the loss is minimized, the relationship extraction model is obtained to achieve relationship extraction. A sentence is obtained, wherein the sentence includes two entities, and special tags are inserted at both ends of each entity, specifically: Get a sentence S, insert a special mark "[CLS]" at the beginning of the sentence, insert special marks "#" at both ends of the first entity, and insert special marks "$" at both ends of the second entity; Use random initialization to get the relation embedding, and use the dot product attention mechanism with the word vector of each word to get its relevance, and further get the attention representation set of the sentence, specifically: Randomly initialize to obtain relation embeddings, and use the dot product attention mechanism to get the relevance of each relation embedding with each word vector; e in the formula i represents the embedding of the i-th relation, h j The word vector representing the jth word is multiplied element by element using the obtained relevance, and the products are added to obtain the sentence attention representation set V = {v1, v2, ..., v m }; The relationship vector is calculated by using the maximum pooling based on the attention representation set, and the sentence vector and two entity vectors are obtained by using the average pooling: The sentence vector is obtained according to the following formula: h cls =W0[tanh(h0)]+b0 The two entity vectors are obtained according to the following formula: The relationship vector is obtained according to the following formula: h v =W2[tanh(MaxPooling(V))]+b2 Where i and j are the start and end subscripts of the first entity, respectively, and k and l are the start and end subscripts of the second entity, respectively.

2. The relationship extraction method according to claim 1, characterized in that: Input each vector in the attention representation set into the binary classifier to obtain the prediction result and calculate the binary classification loss, which is: Input each vector in the attention representation set into the binary classifier to obtain the probability of the relationship existing, and then use the cross entropy loss function according to the true label to obtain the binary loss L bin ; The true label is generated according to the following rule: for each relation r in the predefined relation set, the label is 1 if the relation exists, otherwise the label is 0.

3. The relationship extraction method according to claim 1, characterized in that: The set probability is less than 0.

5.

4. The relationship extraction method according to claim 1, characterized in that: Obtain the consistency loss of the two sentence prediction results, where the consistency loss is: and Represent the multi-classification prediction results of sentences S and S' respectively.

5. The relationship extraction method according to claim 4, characterized in that: The optimal model is obtained by jointly training binary loss, multivariate loss, and consistency loss: L=0.5*(L mul +L′ mul +L bin +L′ bin )+L kl ; L mul , L bin , L′ mul , L′ bin They represent the multivariate loss of sentence S, the binary loss of sentence S, the multivariate loss of sentence S', and the binary loss of sentence S' respectively.

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the relationship extraction method according to any one of claims 1 to 5 is implemented.

7. A device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the relationship extraction method as described in any one of claims 1 to 5 is implemented.

Images