Entity relationship extraction model training method and entity relationship joint extraction method

Abstract

The invention discloses an entity relationship extraction model training method and an entity relationship joint extraction method, and the method comprises the steps: S1, combining a teacher model and a student model into an entity relationship extraction model with a knowledge distillation framework, inputting a training data set, and obtaining a training data set; outputting triple information of the text data by the student model; and S2, judging and comparing whether the precision of the triple information output by the current model is maximum or not, if not, screening out data needing to be manually marked, carrying out manual marking, fusing the data with the initial training data set to obtain an optimized training data set, continuously inputting the optimized training data set into the model to train the model, and finishing model training until the precision reaches the maximum. According to the method, the active learning method is combined with the entity relation joint extraction model, the active learning method is adopted, corpora are expanded, a new training set can be continuously formed to refine parameter values, and the method has good generalization ability and precision and is suitable for the vertical field with small data volume.

Description

Technical Field

[0001] The present invention relates to the technical field of self-speech processing, and in particular to an entity relationship extraction model training method and an entity relationship joint extraction method. Background Art

[0002] Natural language processing (NPL) is an important direction in the field of computer science and artificial intelligence, and information extraction is a basic research in natural language processing. Entity relationship extraction is one of the key tasks of information extraction, and its purpose is to extract entities and relationships from unstructured text data to form structured knowledge. Relational triples are usually represented in the form of a subject, a relational predicate, and an object, where the subject and the object are meaningful named entities, and the relational predicate is usually one of several predefined relation types.

[0003] There are two types of traditional joint extraction models: joint extraction models with shared parameters and joint extraction models with joint decoding. The former is achieved by sharing parameters (sharing input features or internal hidden layer states). This model usually uses a shared neural network to encode the input text, and performs entity recognition and relationship extraction based on the encoding; the latter is to strengthen the interaction between the entity model and the relationship model, and proposes a joint decoding algorithm, using two independent decoders to perform entity recognition and relationship extraction on the input text. The difficulty of joint extraction lies in how to strengthen the interaction between the entity model and the relationship model and further utilize the potential information between the two models.

[0004] With the continuous development of deep learning, entity extraction mostly adopts deep learning methods, but the training of its model requires a large amount of labeled data and is costly. Therefore, active learning is applied to the deep learning model of entity and relationship extraction to minimize the manual labeled data required for model training while maintaining the performance of the extraction model. At the same time, with the rapid development of information technology, the amount of data in various fields has shown an explosive growth trend, resulting in a large number of domain-specific terms. The information contained in these redundant and complex domain data is difficult to use. Therefore, the application of entity extraction technology in various vertical fields has important value. However, due to the small amount of data in each vertical field and the small amount of corpus labeled data, the above-mentioned traditional joint extraction model has defects such as low accuracy, poor generalization ability, and low recall rate when extracting entity relationships in vertical fields. Summary of the invention

[0005] This application proposes an entity relationship extraction model training method and an entity relationship joint extraction method, aiming to solve the problems of low accuracy and poor generalization ability in the prior art for entity relationship extraction in vertical fields.

[0006] The technical solution adopted by the invention is: a method for training an entity relationship extraction model, the method comprising: step S1, combining a teacher model and a student model into an entity relationship extraction model with a knowledge distillation framework, inputting a training data set, and outputting triple information of text data by the student model;

[0007] Step S2, determine whether the accuracy of the triple information output by the current model is the maximum. If not, filter out the data that needs to be manually labeled, and then merge it with the initial training data set to obtain an optimized training data set to continue inputting the model to train the model until the accuracy reaches the maximum and the model training is completed.

[0008] In an optional embodiment, further, it includes an iterative step of the teacher model and the student model: the teacher and student models are trained on the latest optimized training data set until the test accuracy reaches the maximum, and knowledge distillation is used to synchronize the knowledge between the teacher model and the student model in the current iteration.

[0009] Furthermore, the student model is optimized by cross-training the teacher model and the student model with the goal of minimizing the loss function, the steps include: S3.1: calculating the cross entropy of the Soft-target obtained by softmaxing the teacher model at temperature T and the Soft-prediction obtained by softmaxing the student model at the same temperature as the first loss: in, Refers to the value of the softmax output of the teacher model on the i-th category under the condition that the temperature is equal to T, Refers to the value of the softmax output of the student model on the i-th category under the condition that the temperature is equal to T;

[0010] S3.2: Calculate the cross entropy between the Hard-target and the actual value obtained by the student model at a temperature of 1 and softmax as the second loss: Among them, c i Refers to the true label value on the i-th category, c i ∈{0,1}, where 1 represents a positive label and 0 represents a negative label, Refers to the value of the softmax output of the student model on the i-th category when the temperature is equal to 1;

[0011] S3.3: Combine the first loss and the second loss, the total loss function: L = αL soft +βL hard , the student model parameters are optimized with the total loss function L as the minimum objective, where α and β are both hyperparameters.

[0012] Furthermore, step S1 also includes the step of expanding the training data set, and the specific method includes:

[0013] S1.1, divide the input training data set into training set and test set;

[0014] S1.2, input the training set of the training data set into the model for training, and test the model on the test set of the training data set, and calculate the test accuracy of the training data set;

[0015] S1.3, determine whether the test accuracy of the current training data set is greater than the test accuracy of the previous training data set, if so, proceed to step S1.4, if not, proceed to step S1.5;

[0016] S1.4, take the current teacher model as the new model and perform knowledge distillation of the student model simultaneously, and determine whether the uncertain data is exhausted. If so, stop training; if not, send the training set to the sample selection algorithm and perform manual annotation before integrating it into the current training data set to obtain an optimized training data set. Return the optimized training data set to step S1.1 to continue iterative training until the data is exhausted;

[0017] S1.5, determine whether the uncertain data is exhausted. If so, stop training. If not, send the training set to the sample selection algorithm and perform manual annotation before merging it into the current training data set to obtain an optimized training data set. Return the optimized training data set to step S1.1 to continue iterative training. Iterative training continuously feeds back to the teacher model to refine the parameter values ​​until the performance of the teacher model and the student model reaches the maximum.

[0018] A method for joint entity relationship extraction is further proposed, which is an entity relationship extraction model with a knowledge distillation framework obtained by combining a teacher model and a student model based on the entity relationship extraction model training method described in any one of claims 1 to 4.

[0019] In an optional embodiment, further, in the student model, the input mapping relationship of the entity is extracted, and in the teacher model, the entity is mapped by the input of the relationship, and the teacher model and the student model are combined into an entity relationship extraction model with a knowledge distillation framework.

[0020] Furthermore, the teacher model is a pre-trained BERT model, and the method for the teacher model to obtain triple information includes:

[0021] Step S4.1: Data preprocessing: segment the text data, convert the word sequence into a vector to obtain the text feature h of the input text, and perform subword alignment to obtain h avg, in the set of vectorized representations of the sentence h; h = BERT(s), h avg =Avgpool(h), where Avgpool is an average pooling operation to align the embedding transformation length with the original sentence length;

[0022] Step S4.2: Use the label classifier to perform multi-label binary classification on the sentence text features and relation information, and obtain a subset P of potential relations that may exist in the sentence based on the input privileged relation information. rel :P rel =σ(W r h avg +b r ), where σ represents the sigmoid function, and represent the weight and bias parameters when calculating the potential relationship subset, reflecting the existence probability of different relationships;

[0023] Step S4.3: Perform two sequence labeling operations to extract the corresponding subject and object respectively, thereby extracting the complete triple information: in and denote the probability distribution that the i-th tag is the subject or object of the j-th relation, u j is the jth relation representation that can be embedded in the matrix U, is the encoding representation of the ith token after subword alignment, W sub 、b sub and W obj It is the weight and bias parameter when calculating the relationship between subject and object.

[0024] Furthermore, the method for obtaining triple information of the student model includes:

[0025] Step S5.1: Combine GloVe embedding X g and a trainable position embedding X p , use a convolutional encoder consisting of L stacked blocks to encode the text: X = [X g ;X p ];H=Block(…(Block(X))); where [;] represents a connection operation, and each Block contains two expansion rates p i The expanded convolution, a gated unit and a residual connection are used; each block is padded to ensure that the output dimension matches the input dimension:

[0026] Y a =DilatedConv a (X);

[0027] Y b=DilatedConv b (X);

[0028]

[0029] Among them, DilatedConv represents the dilated convolution module, represents element-by-element multiplication, Y i Refers to the output of the i-th Block and the input of the (i+1)-th Block. The sentence represents H and the output Y of the last Block. L equivalence;

[0030] Step S5.2: Two different self-attention modules generate the subject auxiliary feature H through H calculation h and object auxiliary feature H t ;

[0031] Step S5.3: The two lines S→O and O→S are carried out in parallel. The sentences are connected with the corresponding auxiliary features and sent to the feedforward network to extract the subject information S and the object information O respectively. The subject information S is then used to guide the extraction of the object, and the object information O is used to guide the extraction of the subject.

[0032] A computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the entity relationship extraction model training method or the entity relationship joint extraction method.

[0033] Compared with the prior art, the present invention relies on teacher-student models to form an entity relationship extraction model with a knowledge distillation framework, and continuously optimizes the teacher model and the student model. In the case of less data, it takes the combination of manual annotation to actively expand the corpus, so that new training sets can be continuously formed, and then continuously fed back to the teacher model to refine the parameter values, which continues until the performance of the teacher model and the student model is saturated. The corpus is expanded using active learning methods, and the model effect is improved while constructing a high-quality vertical field triple data set. This makes the present invention good at solving the task of joint extraction of entity relationships in the context of low resources in vertical fields, and has good generalization capabilities in vertical fields with small data volumes and less labeled data, and can be flexibly applied in multiple vertical fields such as cultural tourism, medical care, and finance.

[0034] In addition, the present invention also adopts a bidirectional entity extraction framework for the previous framework model that extracts subject entities and object entities in sequence, so as to avoid the failure of subject entity extraction leading to the failure of extraction of the entire triple, and combines the two-step entity relationship extraction model. In the student model, the input mapping relationship of the entity is extracted, and the teacher model maps the entity through the input of the relationship. The teacher and student models are jointly trained to extract entity relationship triples; by combining the active learning method with the entity relationship joint extraction model, it can effectively cope with the situation of less annotated data in the vertical field corpus, thereby further improving the accuracy of triple extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 It is the overall structure diagram of the entity relationship joint extraction model in the present invention;

[0037] Figure 2 It is the overall framework diagram of the model of the entity relationship joint extraction model in the present invention;

[0038] Figure 3 It is the framework diagram of the teacher model in the present invention;

[0039] Figure 4 It is a framework diagram of the student model in the present invention. DETAILED DESCRIPTION

[0040] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] The present invention proposes a training method for an entity relationship extraction model, such as Figure 1As shown, the entity relationship extraction model in the present invention is mainly composed of a teacher model and a student model, which is a relationship extraction model with a knowledge distillation framework structure. Text data is input into the teacher model and the student model by a corpus data set, and the teacher model is pre-trained in combination with text data of certain samples manually annotated to obtain a teacher model with relatively high accuracy. The teacher model is synchronously trained by knowledge distillation to obtain a student model, and the student model outputs triple information of the corresponding text data. The accuracy of the triple information output by the student model is judged, and the text data that does not meet the accuracy standard is sent to the sample selection module. The sample selection module adopts a sample selection algorithm and combines active learning to iteratively update the teacher model. The updated teacher model synchronously trains the student model each time, and the updated student model continues to output triple information until the ability of the teacher model and the student model to extract relations from text data reaches the best, which will be described in detail below.

[0042] First, combine the attached Figure 1 , 2 As shown, the entity relationship extraction model in the present application is a relationship extraction model with a knowledge distillation framework composed of a teacher model and a student model, wherein knowledge distillation is a model compression technology that aims to transfer the knowledge of a complex model (i.e., its parameters and learned representations) to a simplified model so as to reduce the computational and storage requirements of the model while maintaining performance. A relatively complex model is called a teacher model, and a relatively simple model is called a student model. The teacher model is usually a complex or large model that performs well on a given task and has high performance, while the student model is a simplified model that usually has fewer parameters and computing resource requirements. The main implementation of knowledge distillation is to use the output of the teacher model as a soft label (output probability distribution) to train the student model. The student model learns by minimizing the cross-entropy or other loss function with the output of the teacher model. The goal of the student model is to fit the prediction of the teacher model as closely as possible to obtain similar generalization ability and performance.

[0043] Specifically, in an optional embodiment, the teacher model adopts the BERT model, and the student model adopts the convolutional neural network model; Figure 3 , 4It can be seen that the teacher model combines the input privileged relationship information (including but not limited to manual annotations and known relational databases) to perform relationship tagging on the input text data, and uses the entity extraction component to tag the subject and object. For example: when inputting "Mark Zuckerberg is the founder of Facebook", "founder" is the relationship tag, and the entity extraction component tags the subject "Mark Zuckerberg" and the object "Facebook" respectively.

[0044] The specific implementation steps of the teacher model are as follows:

[0045] Step S4.1: Data preprocessing, using the BERT pre-trained model to obtain text features h, using the following method and subword alignment to obtain h avg ,

[0046] h = BERT(s);

[0047] h avg =Avgpool(h),

[0048] Avgpool is an average pooling operation, the purpose of which is to align the embedding conversion length with the original length of the sentence. In this process, the input text data needs to be segmented by a tokenizer and the segmented words (tags) are obtained. The model then outputs the predicted part of speech for each word. The text sequence after segmentation also uses a label set {B, I, O} to mark the entity boundaries of the words in the sentence. B represents the beginning, I represents the inside, and O represents the outside. After marking, it is input into the model for training. By performing BIO marking before the relationship extraction task, words with B as the beginning and followed by I are spliced ​​until the next B label position, which is equivalent to separating a word, thereby completing the segmentation of the input text; then, through encoding processing, a word sequence W = {w1, w2...w n}, and then convert the word sequence into a vector to obtain the text feature h = {h1, h2...h n}, and then add [CLS] and [SEP] tags. Similarly, all sentences in the training set can be of the same length, but the length of each sentence is not fixed. It is necessary to ensure that the length of all word lists is consistent. Suppose the length of the sentence is kept at 7, and the length of the input sentence after word segmentation is 5. Then, it is necessary to add row padding tags [PAD] to the sentence to make the sentence length 7. The [CLS] tag at the beginning of the sentence is used to ensure the aggregation information of the entire complete sentence. In the vectorized representation set h of the sentence, each word vector includes three vectors: token embedding, segment embedding, and position embedding, which are merged into the embedding layer as the input of the BERT model.

[0049] Step S4.2: Use the label classifier to perform multi-label binary classification on the sentence text features and relationship information, label the relationship according to the input privileged relationship information, and combine the potential relationship prediction module to obtain the probability P of the potential relationship that may exist in the sentence. rel ,

[0050] P rel =σ(W r g avg +b r ),

[0051] Among them, σ represents the sigmoid function, W r and b r They represent the weight and bias parameters when calculating the potential relationship subset, reflecting the existence probability of different relationships. rel When it is greater than the set threshold, the relationship is marked as 1 and placed in the potential relationship set R; the probability of each relationship is calculated through the sigmoid function to obtain the subset R of potential relationships; for example, assuming there are three potential relationship labels: "founder", "located in" and "cooperation", the label classifier will output the probability of each relationship, such as founder, 0.85; located in, 0.12; cooperation, 0.03, which means that according to the content of the sentence, the model believes that the possibility of the existence of the "founder is" relationship is the highest and higher than the preset value, and the relationship "founder is" is placed in the relationship set R;

[0052] Step S4.3: Perform two sequence labeling operations, use the Softmax with a fully connected layer and the potential relationship set R to label the subject and object, and extract triple information: the fully connected layer traverses the relationship of the relationship set R for each subject, calculates whether there is a related object, and outputs a triple if there is, and outputs the triple in combination with the following formula:

[0053]

[0054]

[0055] in and denote the probability distribution that the i-th tag is the subject and object of the j-th relation, u j is the jth relation representation that can be embedded in the matrix U, is the encoding representation of the ith token after subword alignment, W sub 、b sub , W obj and b objIt is the weight and bias parameter when calculating the subject and object corresponding to the relationship. If the output probability of the subject and object for the Jth relationship is greater than the preset value, the subject and object are marked as 1. At this time, the {subject, relationship, object} triple is output for successful matching. In the above way, the subject and object tags can be successfully extracted according to the input privileged relationship, and the triple information can be obtained.

[0056] The training of the student model specifically includes the following steps:

[0057] Step S5.1: Encode the text using a convolutional encoder; in this example, we combine GloVe embedding X g and a trainable position embedding X p Concatenate to form the vector representation of the input sentence X = [X g ;X p ] In this way, we can combine word embeddings with position information to obtain a richer and more contextual representation of the input sentence. For example: suppose we use a pre-trained GloVe word embedding model to convert each word in the sentence into a 300-dimensional word vector representation. For each word, we calculate its position information relative to the subject "Mark Zuckerberg" and the object "Facebook". We can define the distance relative to the subject and the distance relative to the object. These position information can be represented by a small trainable embedding matrix. For example, each position feature is 50 dimensions. For each word, we concatenate its GloVe word embedding vector and the corresponding position embedding vector to form a vector representation with a total length of 350 dimensions, so that we can get a word-level vector representation sequence.

[0058] In this way, we can combine word embeddings and position information to obtain a vector representation of the input sentence that is richer and more contextual. Such a vector representation can be used in relation extraction models or other natural language processing tasks, helping to improve the performance and generalization ability of the model.

[0059] Step S5.2: Use self-attention to calculate the subject auxiliary features and object auxiliary features, connect the sentence representation with the auxiliary features, and send them to FFN to extract the subject information and object information respectively;

[0060] This step specifically includes the following sub-steps:

[0061] S5.2.1: Combine GloVe embedding X g and a trainable position embedding X p The obtained vector is encoded using a convolutional encoder containing L stacked blocks to obtain the text representation: H = Block (… (Block (X))), where [;] represents a connection operation and each Block contains two expansion rates p iA dilated convolution, a gating unit, and a residual connection are used to ensure that the output dimension matches the input dimension by padding for each block:

[0062] The final text representation is obtained in the following way:

[0063] Y a =DilatedConv a (X)

[0064] Y b =DilatedConv b (X)

[0065]

[0066] Among them, DilatedConv represents the dilated convolution module, represents element-by-element multiplication, Y i Refers to the output of the i-th Block and the input of the (i+1)-th Block. Obviously, the text representation H is equivalent to the output YL of the last Block.

[0067] S5.2.2: Two different self-attention modules generate the subject auxiliary feature H through the text representation H h and object auxiliary feature H t , bidirectionally and parallelly connect the text features with the corresponding auxiliary features, extract the corresponding subject information S and object information 0, and then use them to guide the extraction of object and subject information respectively;

[0068] Main auxiliary feature H h The generation of is described as an example:

[0069]

[0070] Q=W q ·H+b q

[0071] K=W k ·H+b k

[0072] V=W v .H+b v

[0073] Among them, d k is the dimension of matrix K, W q , W k , W v and b q 、b k 、b vare the weights and bias parameters used to obtain the Q, K, and V matrices respectively. t The acquisition of the same subject auxiliary feature H h , I will not go into details here.

[0074] S5.2.3: Connect the sentence with the corresponding auxiliary features and send them to the feedforward network to extract the subject information S and object information O respectively. Then use the obtained subject information S to guide the extraction of the object, and use the obtained object information O to guide the extraction of the subject.

[0075] Here, two lines S→O and O→S are used in parallel. The sentences are connected with the corresponding auxiliary features and sent to the feedforward network to extract the subject information S and the object information 0 respectively. Then the subject information S is used to guide the extraction of the object, and the object information 0 is used to guide the extraction of the subject. Here, the S→O direction is taken as an example:

[0076]

[0077]

[0078]

[0079]

[0080] Among them, [;] represents the connection operation. and They respectively indicate that the i-th token is the beginning and end of the subject with the j-th relationship, and They respectively indicate that the i-th token is the beginning and end of the object with the j-th relationship.

[0081] Step S5.3: Use the object and entity information to form triplets through relational mapping. The two lines S→O and O→S are carried out in parallel. The sentences are connected with the corresponding auxiliary features and sent to the feedforward network to extract the subject information S and the object information O respectively. The subject information S is then used to guide the extraction of the object, and the object information O is used to guide the extraction of the subject.

[0082] Here, the teacher model extracts relational features in advance and the student model extracts the subject and object in parallel. In contrast to the framework model that extracts subject entities and object entities in sequence, a bidirectional entity extraction framework is adopted to avoid the failure of extracting the entire triple due to the failure of subject entity extraction; the two-step entity relationship extraction model is combined, and the student model extracts the input mapping relationship of the entity, and the teacher model maps the entity through the input mapping of the relationship, and the teacher and student models are jointly trained to extract entity relationship triples; the active learning method is combined with the entity relationship joint extraction model to improve the model accuracy, and triples can also be extracted from two directions to improve the recognition ability of triples in unstructured corpus. Moreover, the system can obtain information from multiple angles and multiple levels. This diversified information acquisition method helps the system understand the text content more comprehensively and enhances the ability to grasp the complex relationship between subject, object and relationship.

[0083] The above are examples of text relationship extraction by the teacher model and the student model. For the situation where the vertical field is expected to be less, how to increase the amount of corpus so that the student model and the teacher model can be trained for deep learning, thereby relying on less corpus data to improve the accuracy and generalization ability of the model's relationship collection in this field is the key. The present invention effectively copes with the situation of less annotated data in the vertical field corpus by combining the active learning method with the entity relationship joint extraction model, thereby improving the accuracy of triple extraction. It is oriented to most vertical fields and has good generalization ability, and can be flexibly applied in multiple fields such as cultural tourism, medical care, and finance.

[0084] The entity relationship extraction model training method in the present invention includes:

[0085] Step S1, the teacher model and the student model are combined into an entity relationship extraction model with a knowledge distillation framework, the training data set is input, and the student model outputs the triple information of the text data; the teacher model and the student model input the training data at the same time, the teacher model is only used to train the student model with the learned knowledge, and the student model is used for prediction. Combined with the teacher model to extract relationship features in advance and the student model to extract subjects and objects in parallel, the system can obtain information from multiple angles and multiple levels. This diversified information acquisition method helps the system to understand the text content more comprehensively and enhances the ability to grasp the complex relationships between subjects, objects and relationships.

[0086] Step S2, determine whether the accuracy of the output triple information is the largest. If not, manually mark the text data corresponding to the triple information that does not meet the accuracy standard and merge it with the initial training data set to obtain an optimized training data set until the output accuracy of the model reaches the maximum. Use the optimized training data set as the input training data set of the entity relationship extraction model, and repeat steps S1 and S2 to train and update the teacher model and the student model.

[0087] Specifically, step S2 also includes an iterative step of the teacher model and the student model: the text data corresponding to the triple information that does not meet the accuracy standard is sent to the sample selection algorithm and combined with the active learning method to train the teacher model on the latest optimized training data set until the test accuracy of the teacher model reaches the maximum, and the knowledge distillation is used to synchronize the knowledge between the teacher model and the student model in the current iteration.

[0088] Among them, the sample selection algorithm is used to select the most representative or most informative samples from unlabeled data. The sample selection algorithm includes uncertainty sampling, maximum margin sampling, etc. The selected text data is sent to manual annotation through the sample selection algorithm to obtain their true labels. Active learning selects the most informative samples for labeling, which can effectively utilize limited labeling resources and thus improve model performance.

[0089] The training steps for the teacher model include: dividing the training data set into a training set and a test set, continuously training the model through the training set, and testing whether the effect of the current teacher model is better than the previous teacher model through the test set. If so, the current teacher model is used as the new teacher model to continue the iterative training of the teacher model until the test accuracy reaches the maximum, and using knowledge distillation to synchronize the knowledge between the teacher model and the student model in the current iteration.

[0090] Through cross-training between the teacher model and the student model, the student model is optimized with the goal of minimizing the loss function. The steps include:

[0091] S3.1: Calculate the cross entropy of the Soft-target obtained by the teacher model at temperature T and the Soft-prediction obtained by the student model at the same temperature as the first loss:

[0092] in, Refers to the value of the softmax output of the teacher model on the i-th category under the condition that the temperature is equal to T, Refers to the value of the softmax output of the student model on the i-th category under the condition that the temperature is equal to T;

[0093] S3.2: Calculate the cross entropy between the Hard-target and the actual value obtained by the student model at a temperature of 1 and softmax as the second loss:

[0094] Among them, c i Refers to the true label value on the i-th category, c i ∈{0,1}, where 1 represents a positive label and 0 represents a negative label, Refers to the value of the softmax output of the student model on the i-th category when the temperature is equal to 1;

[0095] S3.3: Combine the first loss and the second loss, the total loss function: L = αL soft +βL hard , optimize the student model parameters with the total loss function L as the minimum target, where α and β are both hyperparameters. During the training process, the student model back-propagates through the new loss function and tries to gradually adjust its own prediction ability by simulating the prediction results of the teacher model to improve the generalization ability of the model. Finally, the performance of the large model is achieved on the lightweight model.

[0096] Furthermore, step S1 also includes the step of expanding the training data set, and the specific method includes:

[0097] S1.1, divide the input training data set into training set and test set;

[0098] S1.2, input the training set of the training data set into the model for training, and test the model on the test set of the training data set, and calculate the test accuracy of the training data set;

[0099] S1.3, determine whether the test accuracy of the current training data set is greater than the test accuracy of the previous training data set, if so, proceed to step S1.4, if not, proceed to step S1.5;

[0100] S1.4, take the current teacher model as the new model and perform knowledge distillation of the student model simultaneously, and determine whether the uncertain data is exhausted. If so, stop training; if not, send the training set to the sample selection algorithm and perform manual annotation before integrating it into the current training data set to obtain an optimized training data set. Return the optimized training data set to step S1.1 to continue iterative training until the data is exhausted;

[0101] S1.5, determine whether the uncertain data is exhausted. If so, stop training. If not, send the training set to the sample selection algorithm and perform manual annotation before merging it into the current training data set to obtain an optimized training data set. Return the optimized training data set to step S1.1 to continue iterative training. Iterative training continuously feeds back to the teacher model to refine the parameter values ​​until the performance of the teacher model and the student model reaches the maximum.

[0102] Among them, in the above steps, after the training set is input into the model, if the accuracy of the triple information output by the student model is lower than that of the previous model, then these text data below the threshold are mixed with the previous round of training sets to re-iterate the teacher model; at the same time, during the teacher model training, some or all of these text data below the threshold can be manually annotated (here mainly in the form of relationship annotation), and the text data after manual annotation is mixed with the previous round of training sets to obtain a larger data set for iterative training; in the iterative process, the training set data is mainly sent to the teacher model for iterative training. After each iteration, the teacher model tests the corresponding effect (such as accuracy, F1 value, etc.) through the test set, and the effect of the teacher model is evaluated by testing the F1 value output by the current teacher model, and after the teacher model iteration, a new student model will be synchronously updated for outputting triples. Here, the iterative training of the teacher model is divided into multiple cases for explanation:

[0103] Case 1: After the current teacher model is tested on the test set after iteration, the effect of the teacher model is better than that of the previous generation. Then the current version of the teacher model is used. The teacher model is subjected to knowledge distillation to obtain an optimized student model, which is output by the student model and the training set is used as the input data set to continue training the model. If the output of this version of the student model still has triple data that does not meet the accuracy standards, then at this time it is necessary to continue training the model using steps S1 and S2, that is, these new triple data that do not meet the standards are continued to be manually labeled and mixed into the previous training set to form a new training set for training until the student model no longer outputs triple signals that do not meet the standards and the output accuracy of the student model reaches the maximum.

[0104] Case 2: If the current teacher model after iteration is tested on the test set and the effect of the teacher model is worse than that of the previous generation, then it is necessary to determine whether there is still substandard triple information. If not, these substandard text data have been exhausted, then stop training, and use the best teacher model of the previous version as the model to be used. If there is still triple information that does not meet the output standards, then these substandard data are mixed into the previous round of training sets, and continue training with manual labeling until the student model no longer outputs substandard triple information.

[0105] It should be noted that the selection process of setting pseudo labels through the student model is as follows (i.e., the text data corresponding to the triples that do not meet the accuracy standards are used as new labeled text data and trained with the training set):

[0106]

[0107] Among them, Pnew is the probability score of the sample that needs to be relabeled predicted by the teacher-student model, P old refers to the probability score of the original labels of these samples, γ is the boundary threshold, (d(u new )≤d(u old )-γ means that the new non-standard triplet information and the classification boundary of the previous round of training set are in a low-density area, and None refers to the label without category. If the output probability of the text data is lower than 0.1, the labeled text data will not be mixed with the training set as a pseudo-label, but the text data with relationship annotations will be mainly generated by manual annotation; if the output probability value score of the text data in the current version of the model is greater than the output probability value score of the previous version of the model, and the new non-standard triplet information and the classification boundary of the previous round of training set are in a low-density area, the newly labeled text data will be mixed with the training set as a pseudo-label. It is also possible to manually annotate part or all of the text data below the threshold during the teacher model training (here mainly relational annotation is used). ), mix the previous round of training set of the text data after manual annotation to obtain a larger data set for iterative training; and when the output probability value score of the text data in the current version of the model is less than the output probability value score of the previous version of the model, the last marked text data is mixed with the training set as a pseudo-label, and some or all of the text data below the threshold can be manually annotated during the teacher model training (here mainly the relationship annotation method is adopted), and the previous round of training set of the text data after manual annotation is mixed to obtain a larger data set for iterative training.

[0108] Through this process, new training sets can be continuously formed, which can be continuously fed back to the teacher model to refine the parameter values, and this process continues until the performance of the teacher model and the student model are saturated. The active learning method is used to expand the corpus, improve the model effect, and build a high-quality vertical field triple data set. It is oriented to most vertical fields and has good generalization capabilities. It can be flexibly applied in multiple fields such as cultural tourism, medical care, and finance.

[0109] Verification: This paper uses the introduction information of tourist attractions in Inner Mongolia Autonomous Region collected from the Internet to construct an unlabeled corpus. The initial unlabeled data includes tourist attractions in 13 leagues and cities in Inner Mongolia Autonomous Region. 12 entity types and 11 relationship types are designed. The first batch of 600 sentence data is annotated as a training set, and 200 sentences are used as a verification set.

[0110] The basic verification data information used in the present invention is shown in Table 1.

[0111] Table 1 Statistics of Inner Mongolia Autonomous Region Tourist Attractions Dataset (IMTAD)

[0112]

[0113] The baseline models in the experiment are: CasRel, one of the new tag-based methods, which models relations as a function of mapping topics to objects. PRGC, which decomposes the relation triple extraction task into three subtasks, relation judgment, entity extraction and subject-object alignment, greatly reduces the judgment of relation redundancy. The evaluation index used in the experiment of the present invention is mainly the F1 value. In order to evaluate the advantages and disadvantages of different algorithms, the concept of F1 value is proposed on the basis of Precision and Recall to evaluate Precision (accuracy) and Recall (recall rate) as a whole. The definition of F1 is as follows: F1 value = accuracy * recall rate * 2 / (accuracy + recall rate).

[0114] First, the model proposed in this patent application is evaluated and compared on different baseline models using the initial training data. For the sake of clarity and convenience, the entity relationship joint extraction model proposed in this patent application is referred to as the ActiveLRel model in Table 2. The main experimental results are shown in Table 2. It can be seen that the ActiveLRel model proposed in this patent application is significantly higher than the other baseline models in all indicators in the Inner Mongolia tourist attractions dataset.

[0115] Table 2 Evaluation of initial training phase for different baselines

[0116]

[0117] In addition, after the initial stage, the model F1 value reached 86.97. The model predicted unlabeled corpus, and the model prediction results were connected to LabelStudio for visualization. Manual intervention and modification were used to ensure the correctness of the predicted labeling results. The model prediction results reduced the workload of manual labeling, and finally obtained 1,780 labeled data in the field of Inner Mongolia tourist attractions.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for training an entity relationship extraction model, characterized in that: The entity relationship extraction model training method includes: Step S1, combining the teacher model and the student model into an entity relationship extraction model with a knowledge distillation framework, inputting a training data set, and having the student model output triple information of text data; Step S2: determine whether the accuracy of the output triplet information reaches the threshold. If not, filter out the data that needs to be manually labeled, and then merge it with the initial training data set to obtain the optimized training data set to continue inputting the model to train the model until the accuracy reaches the threshold and the model training is completed.

2. The entity relationship extraction model training method according to claim 1, characterized in that: It also includes the iterative steps of the teacher model and the student model: the teacher and student models are trained on the latest optimized training data set until the test accuracy reaches the maximum, and knowledge distillation is used to synchronize the knowledge between the teacher model and the student model in the current iteration.

3. The entity relationship extraction model training method according to claim 2 is characterized in that: Through cross-training between the teacher model and the student model, the student model is optimized with the goal of minimizing the loss function. The steps include: S3.1: Calculate the cross entropy of the Soft-target obtained by the teacher model at temperature T and the Soft-prediction obtained by the student model at the same temperature as the first loss: in, Refers to the value of the softmax output of the teacher model on the i-th category under the condition that the temperature is equal to T, Refers to the value of the softmax output of the student model on the i-th category under the condition that the temperature is equal to T; S3.2: Calculate the cross entropy between the Hard-target and the actual value obtained by the student model at a temperature of 1 and softmax as the second loss: Among them, c i Refers to the true label value on the i-th category, c i ∈{0,1}, where 1 represents a positive label and 0 represents a negative label, Refers to the value of the softmax output of the student model on the i-th category when the temperature is equal to 1; S3.3: Combine the first loss and the second loss, the total loss function: L = αL soft +βL hard , the student model parameters are optimized with the total loss function L as the minimum objective, where α and β are both hyperparameters.

4. The entity relationship extraction model training method according to claim 3 is characterized in that: Step S1 also includes the step of expanding the training data set, and the specific method includes: S1.1, divide the input training data set into training set and test set; S1.2, input the training set of the training data set into the model for training, and test the model on the test set of the training data set, and calculate the test accuracy of the training data set; S1.3, determine whether the test accuracy of the current training data set is greater than the test accuracy of the previous training data set, if so, proceed to step S1.4, if not, proceed to step S1.5; S1.4, take the current teacher model as the new model and perform knowledge distillation of the student model simultaneously, and determine whether the uncertain data is exhausted. If so, stop training; if not, send the training set to the sample selection algorithm and perform manual annotation before integrating it into the current training data set to obtain an optimized training data set. Return the optimized training data set to step S1.1 to continue iterative training until the data is exhausted; S1.5, determine whether the uncertain data is exhausted. If so, stop training. If not, send the training set to the sample selection algorithm and perform manual annotation before merging it into the current training data set to obtain an optimized training data set. Return the optimized training data set to step S1.1 to continue iterative training. Iterative training continuously feeds back to the teacher model to refine the parameter values ​​until the performance of the teacher model and the student model reaches the maximum.

5. A method for joint extraction of entity relations, characterized in that: The entity relationship joint extraction method is an entity relationship extraction model with a knowledge distillation framework obtained by combining a teacher model and a student model based on the entity relationship extraction model training method described in any one of claims 1 to 4.

6. The entity relationship joint extraction method according to claim 5, characterized in that: In the student model, the input mapping relationship of the entity is extracted, and in the teacher model, the entity is mapped through the input of the relationship. The teacher model and the student model are combined into an entity relationship extraction model with a knowledge distillation framework.

7. The entity relationship joint extraction method according to claim 6, characterized in that: The teacher model is a pre-trained BERT model, and the method for the teacher model to obtain triple information includes: Step S4.1: Data preprocessing: segment the text data, convert the word sequence into a vector to obtain the text feature h of the input text, and perform subword alignment to obtain h avg , in the set of vectorized representations of the sentence h; h = BERT(s), h avg =Avgpool(h), where Avgpool is an average pooling operation to align the embedding transformation length with the original sentence length; Step S4.2: Use the label classifier to perform multi-label binary classification on the sentence text features and relation information, and obtain a subset P of potential relations that may exist in the sentence based on the input privileged relation information. rel :P rel =σ(W r h avg +b r ), where σ represents the sigmoid function, and represent the weight and bias parameters when calculating the potential relationship subset, reflecting the existence probability of different relationships; Step S4.3: Perform two sequence labeling operations to extract the corresponding subject and object respectively, thereby extracting the complete triple information: in and denote the probability distribution that the i-th tag is the subject or object of the j-th relation, u j is the jth relation representation that can be embedded in the matrix U, is the encoding representation of the ith token after subword alignment, W sub 、b sub and W obj It is the weight and bias parameter when calculating the relationship between subject and object.

8. The entity relationship joint extraction method according to claim 7, characterized in that: The method for obtaining triple information of the student model includes: Step S5.1: Combine GloVe embedding X g and a trainable position embedding X p , use a convolutional encoder consisting of L stacked blocks to encode the text: X = [X g ;X p ];H=Block(···(Block(X))); where [;] represents a connection operation, and each Block contains two expansion rates p i The expanded convolution, a gated unit and a residual connection are used; each block is padded to ensure that the output dimension matches the input dimension: Y a =DilatedConv a (X); Y b =DilatedConv b (X); Among them, DilatedConv represents the dilated convolution module, represents element-by-element multiplication, Y i Refers to the output of the i-th Block and the input of the (i+1)-th Block. The sentence represents H and the output Y of the last Block. L equivalence; Step S5.2: Two different self-attention modules generate the subject auxiliary feature H through H calculation h and object auxiliary feature H t ; Step S5.3: The two lines S→O and O→S are carried out in parallel. The sentences are connected with the corresponding auxiliary features and sent to the feedforward network to extract the subject information S and the object information O respectively. The subject information S is then used to guide the extraction of the object, and the object information O is used to guide the extraction of the subject.

9. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the entity relationship extraction model training method according to any one of claims 1 to 4 or the entity relationship joint extraction method according to any one of claims 5 to 8.

Images