Entity extraction method, training method of first entity extraction model and related device

Abstract

The present application discloses an entity extraction method, a training method of a first entity extraction model and a related device, wherein the entity extraction method comprises: obtaining a text to be extracted; inputting the text to be extracted into the trained first entity extraction model to obtain entities and entity types in the text to be extracted; wherein a noise data set is used when training the first entity extraction model; and the noise data set is preliminarily passed through a second entity extraction model to obtain a set of missing annotation segments, a set of incorrectly annotated segments, a set of correctly annotated segments and a set of meaningless annotated segments, and the first entity extraction model is trained using the set of incorrectly annotated segments, the set of correctly annotated segments and the set of meaningless annotated segments, while the set of missing annotation segments is prohibited from being added to the training process. This not only improves the generalization performance of the model for noisy data, but also allows a highly generalizable entity extraction method to be trained without human involvement.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an entity extraction method, a training method for a first entity extraction model, and related devices. Background Art

[0002] Artificial intelligence is a science that studies how to simulate human cognitive abilities through machines. Deep learning is the main means of realizing artificial intelligence at this stage. Deep learning methods usually learn regularities or judgment rules from known data and establish prediction models. In practical applications, deep learning algorithms mostly adopt a supervised learning model, that is, they need to be trained on a large amount of labeled data. And these labeled training data must be manually labeled one by one, because the quality of the labeled data also has a great impact on the effect of the training model. However, the process of manual labeling is a subjective job after all, and even trained labelers will have a certain error rate. For example, in sentiment analysis labeling, the same sentence will have different emotions in the eyes of different people, and the labeling results will be inconsistent; for another example, for tasks such as entity extraction, relationship extraction, and event extraction that require professional labeling knowledge and have relatively complex labeling tasks, labelers are prone to missing labels and wrong labels.

[0003] Usually, in order to avoid noise in the training data, the same data is assigned to multiple people for labeling, and the results of the multiple labels are finally summarized, and the results labeled by most people are taken as the true labeling results. However, this will bring about a large labeling cost problem. Therefore, studying how to build a high-accuracy deep learning model on noisy manually labeled data is a very practical direction. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an entity extraction method, a training method for a first entity extraction model and related devices, which can improve the generalization performance of the model for noisy data without human intervention.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide an entity extraction method, comprising: obtaining a text to be extracted; inputting the text to be extracted into a trained first entity extraction model to obtain entities in the text to be extracted and the types of the entities; wherein, a noise data set is used when training the first entity extraction model; and the noise data set is preliminarily passed through a second entity extraction model to obtain a set of missing annotation segments, a set of incorrectly annotated segments, a set of correctly annotated segments, and a set of meaningless annotated segments, and the first entity extraction model is trained using the set of incorrectly annotated segments, the set of correctly annotated segments, and the set of meaningless annotated segments, while the set of missing annotation segments is prohibited from being added to the training process; wherein, the set of missing annotation segments includes entity segments that are manually omitted from the noise data set but correctly predicted by the second entity extraction model, the set of incorrectly annotated segments includes entity segments whose manually annotated types are different from the corresponding predicted types, the set of correctly annotated segments includes entity segments whose manually annotated types are the same as the corresponding predicted types, and the set of meaningless annotated segments includes entity segments that are not manually annotated and do not belong to the set of missing annotation segments.

[0006] Among them, the process of training the first entity extraction model includes: using the second entity extraction model to obtain a set of problem segments consisting of all entity segments whose first prediction results are inconsistent with the manual annotation results in the noise data set, and a set of correct segments consisting of entity segments whose first prediction results are consistent with the manual annotation results; wherein the first prediction result includes a predicted entity and a predicted type, and the manual annotation result includes a manually annotated entity and a manually annotated type; classifying the set of problem segments to obtain a set of missed annotation segments, a set of incorrectly annotated segments, and a set of meaningless annotated segments; using the set of incorrectly annotated segments, the set of correctly annotated segments, and the set of meaningless annotated segments to train the first entity extraction model, and prohibiting the use of the set of missed annotation segments to train the first entity extraction model.

[0007] Among them, the step of using the second entity extraction model to obtain a set of problem segments consisting of all entity segments whose prediction results are inconsistent with the manual annotation results in the noise data set includes: dividing the noise data set into multiple sub-datasets; using one of the sub-datasets as test data and the remaining sub-datasets as training data; setting multiple different random seeds to obtain multiple second entity extraction models respectively; for each second entity extraction model, using the training data to train the second entity extraction model; using the trained second entity extraction model to predict the test data to obtain a corresponding first prediction result, and obtaining candidate entity segments inconsistent with the manual annotation results according to the first prediction result; counting all the candidate entity segments obtained by all the second entity extraction models, and in response to the accumulated number of times the same candidate entity segment exceeds a first threshold, determining the candidate entity segment as a problem segment, and adding the problem segment to the set of problem segments; judging whether all the sub-datasets are used as the test data in turn; if not, using another sub-dataset as the test data and the remaining sub-datasets as the training data, and returning to the step of setting multiple different random seeds to obtain multiple second entity extraction models respectively.

[0008] Among them, the second entity extraction model includes a first pre-trained model and a first fully connected layer, and the output of the first pre-trained model is the input of the first fully connected layer; the step of using the training data to train the second entity extraction model includes: performing word segmentation processing on the text data in the training data and inputting it into the first pre-trained model to obtain a first semantic vector for each word in the text data; performing negative sampling on all the first semantic vectors to obtain multiple first combined fragments; wherein the first combined fragments include manually annotated entity fragments and non-manually annotated entity fragments; in response to the width of the first combined fragment being less than or equal to a second threshold, obtaining a second semantic vector of the first combined fragment; inputting the second semantic vector into the first fully connected layer to obtain a first predicted classification logit vector; and adjusting the parameters in the second entity extraction model based on the prediction results of the manually annotated entity fragments and part of the non-manually annotated entity fragments.

[0009] Among them, before the step of performing word segmentation processing on the training text in the training data and then inputting it into the first pre-training model to obtain the first semantic vector of each word in the training text, it includes: randomly initializing a first width-limited vector matrix; wherein the first width-limited vector matrix contains multiple width feature vectors; the step of obtaining the second semantic vector of the first combined fragment in response to the width of the first combined fragment being less than or equal to a second threshold value includes: obtaining the first semantic vector of the starting word and the first semantic vector of the ending word in the first combined fragment; concatenating the first semantic vector of the starting word, the first semantic vector of the ending word and the corresponding width feature vector to obtain the second semantic vector; when adjusting the parameters in the second entity extraction model based on the prediction results of the manually annotated entity fragments and part of the non-manually annotated entity fragments, the parameters in the first width-limited vector matrix are adjusted at the same time.

[0010] Among them, the first entity extraction model includes a second pre-trained model and a second fully connected layer, and the output of the second pre-trained model is the input of the second fully connected layer; the step of using the incorrectly labeled segment set, the correctly labeled segment set and the meaningless labeled segment set to train the first entity extraction model, and prohibiting the step of using the missing labeled segment set to train the first entity extraction model, includes: performing word segmentation processing on the text data in the noise data set and inputting it into the second pre-trained model to obtain a third semantic vector of each word in the text data; performing negative sampling and combination on all the third semantic vectors to obtain a plurality of second combined segments; in response to the width of the second combined segment being less than or equal to a second threshold, obtaining a fourth semantic vector of the second combined segment; inputting the fourth semantic vector into the second fully connected layer to obtain a second predicted classification logit vector; in response to the span of the second combined segment in the correctly labeled segment set, the meaningless labeled segment set and the incorrectly labeled segment set, obtaining a corresponding first loss according to the set type to which at least part of the second combined segment belongs, and adjusting the parameters in the first entity extraction model based on the first loss.

[0011] Among them, the step of obtaining the corresponding first loss according to the set type to which at least part of the second combined fragments belong includes: in response to the span of the second combined fragment being in the correctly labeled fragment set or the meaningless labeled fragment set, obtaining the first cross entropy loss between the second predicted classification logit vector corresponding to the second combined fragment and the multi-classification label vector of the manually labeled type, and obtaining the first KL divergence loss between the second predicted classification logit vector corresponding to the second combined fragment and the first predicted classification logit vector obtained by the second entity extraction model, and using the first cross entropy loss and the first KL divergence loss as the first loss; in response to the span of the second combined fragment being in the incorrectly labeled fragment set, obtaining the second KL divergence loss between the second predicted classification logit vector corresponding to the second combined fragment and the first predicted classification logit vector obtained by the second entity extraction model, and using the second KL divergence loss as the first loss.

[0012] Among them, the step of obtaining the corresponding first loss according to the set type to which at least part of the second combined fragments belong in response to the span of the second combined fragment in the correctly labeled fragment set, the meaningless labeled fragment set and the incorrectly labeled fragment set includes: obtaining the first loss of all the second combined fragments whose span is in the correctly labeled fragment set, the first loss of all the second combined fragments whose span is in the incorrectly labeled fragment set, and the first loss of some of the second combined fragments whose span is in the meaningless labeled fragment set.

[0013] To solve the above technical problems, another technical solution adopted in the present application is: providing a training method for a first entity extraction model, comprising: obtaining a noise data set; using a second entity extraction model to obtain a set of missed annotation fragments, a set of incorrectly annotated fragments, a set of correctly annotated fragments and a set of meaningless annotated fragments in the noise data set; wherein the missed annotation fragment set includes entity fragments that are manually missed in the noise data set but are correctly predicted by the second entity extraction model, the set of incorrectly annotated fragments includes entity fragments whose manually annotated types are different from the corresponding predicted types, the set of correctly annotated fragments includes entity fragments whose manually annotated types are the same as the corresponding predicted types, and the set of meaningless annotated fragments includes entity fragments that are not manually annotated and do not belong to the missed annotation fragment set; using the set of incorrectly annotated fragments, the set of correctly annotated fragments and the set of meaningless annotated fragments to train the first entity extraction model, and prohibiting the missed annotation fragment set from training the first entity extraction model.

[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide an entity extraction device, comprising: an acquisition module, used to acquire a text to be extracted; an entity module, connected to the acquisition module, used to input the text to be extracted into a trained first entity extraction model to obtain entities in the text to be extracted and the types of the entities; a first training module, connected to the entity module and the acquisition module, used to train the first entity extraction model using a noise data set; and the noise data set is pre-processed by a second entity extraction model to obtain a set of missing annotation segments, and the set of missing annotation segments is prohibited from being added to the training process when training the first entity extraction model; wherein the set of missing annotation segments includes entity segments that are manually omitted from the noise data set but correctly predicted by the second entity extraction model, the set of incorrectly annotated segments includes entity segments whose manually annotated types are different from the corresponding predicted types, the set of correctly annotated segments includes entity segments whose manually annotated types are the same as the corresponding predicted types, and the set of meaningless annotated segments includes entity segments that are not manually annotated and do not belong to the set of missing annotation segments.

[0015] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the entity extraction method mentioned in any of the above embodiments and the training method of the first entity extraction model.

[0016] To solve the above technical problems, another technical solution adopted in the present application is: providing a storage device storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the entity extraction method mentioned in any of the above embodiments and the training method of the first entity extraction model.

[0017] Different from the prior art, the beneficial effect of the present application is that the entity extraction method provided by the present application includes: inputting the acquired text to be extracted into the trained first entity extraction model to obtain the entities and entity types in the text to be extracted; wherein, the first entity extraction model is trained using a noise data set, and the noise data set is pre-trained by the second entity extraction model to obtain a set of missing labeled segments, a set of incorrectly labeled segments, a set of correctly labeled segments, and a set of meaningless labeled segments, and the first entity extraction model is trained using the set of incorrectly labeled segments, the set of correctly labeled segments, and the set of meaningless labeled segments, while the set of missing labeled segments is prohibited from being added to the training process. Through this design method, not only the unlabeled noise in the training data can be modeled, but also the incorrectly labeled noise in the training data can be modeled, thereby improving the generalization performance of the model for noisy data, and a highly generalized entity extraction method can be trained without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0019] Figure 1 It is a flowchart of an implementation method of the entity extraction method of the present application;

[0020] Figure 2 is a flowchart of the process of training the first entity extraction model;

[0021] Figure 3 yes Figure 2 A schematic flow chart of an implementation method of step S10;

[0022] Figure 4 yes Figure 3 A schematic flow chart of an implementation method of step S103;

[0023] Figure 5 It is a structural diagram of the second entity extraction model with width restriction;

[0024] Figure 6 yes Figure 4 A schematic diagram of a flow chart of an implementation method of step S22;

[0025] Figure 7 yes Figure 2 A schematic flow chart of an implementation method of step S12;

[0026] Figure 8 It is a structural diagram of the first entity extraction model for noise data;

[0027] Fig. 9 It is a flowchart of the training method of the first entity extraction model of the present application;

[0028] Fig.10 It is a schematic diagram of the framework of an implementation method of the entity extraction device of the present application;

[0029] Fig.11 It is a schematic diagram of a framework of an implementation method of an electronic device of the present application;

[0030] Fig.12 It is a structural diagram of an implementation method of a storage device of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In a noisy dataset, there are usually cases where some entities are not labeled, the entity types of the labeled entities in the data are incorrectly labeled, and the entity boundaries are incorrectly labeled. Currently, there are several methods to solve the problem of noisy datasets:

[0033] (1) In the industry, in order to better solve the labeling problem, the method of model error detection and manual correction is generally used to obtain all the problematic data in the full data set that need to be manually re-labeled, and this part of the data is resubmitted to the labelers for inspection. Most of the data found through model error correction is data that is difficult to fit the model due to labeling problems. After obtaining the re-labeled data, the previous labels are replaced and the model is retrained. However, this method requires continuous manual participation to modify or supplement the labeling results in the noise data, directly turning the noise data into non-noise data from the source, and cannot improve the model's modeling ability for noise data. It only corrects at the data level, which does not have the universality of the method and requires a lot of manpower, and cannot be promoted.

[0034] (2) In academia, researchers have proposed using the NegSampling method to model incompletely labeled entity extraction data. The method mainly extracts entities by arranging fragments. That is, for a text containing N characters, there are theoretically N(N+1) / 2 fragment arrangements as candidate entities. However, this approach will generate a large number of negative samples when the text is too long. This method can predict each candidate entity fragment separately, breaking the limitation of CRF for global prediction, and then negatively sample all non-entity fragments. Through negative sampling, the probability of unlabeled entities participating in the model training process is reduced. However, this method only considers the situation where entities in the noise data are not labeled, and does not consider the situation where entities in the noise data are incorrectly labeled. Entities with incorrect labels will still participate in the model training process, which will have a negative impact on the model training effect.

[0035] (3) A multi-stage training strategy is adopted to solve the entity extraction problem of noisy data, that is, a small amount of correct data and a large amount of noisy data are used to jointly train the model in stages. First, in the first step, only unlabeled text data is used for pre-training of the masked language model (MLM), so that the BERT model completes the domain data adaptation and becomes In-DomainBERT; secondly, a small amount of correct data is selected from all the manually annotated data, and the BERT-CRF model is established. Supervised training is performed on a small amount of correct data to obtain a basic model; then the BERT-CRF model is used to predict on the remaining noisy data, and the predicted entities are used to complete the unlabeled part of the noisy data, and each completed prediction result is given a confidence; then, the BERT-CRF model is continued to be trained on the completed data combined with the entity confidence; finally, the final BERT-CRF model is fine-tuned on a small amount of correct data. However, this multi-stage training strategy requires manual selection of a part of accurate annotated data from the noisy data, and also requires manual participation in the process of model effect optimization. In addition, in this multi-stage training strategy, it is necessary to train the BERT-CRF model based on accurately labeled data, and then use the model to perform inference and prediction on noisy data, and use the predicted entities to complete the unlabeled parts of the noisy data. Therefore, this method does not consider the situation when there are labeling errors in the data, and the impact of entity labeling errors on the model is worse than the situation where the entities are not labeled.

[0036] The following is a detailed introduction to the entity extraction method provided by this application.

[0037] See also Figure 1 , Figure 1 It is a flowchart of an implementation method of the entity extraction method of the present application. The above entity extraction method specifically includes:

[0038] S1: Get the text to be extracted.

[0039] Specifically, the entity extraction method in the present application is aimed at text data, and the text to be extracted can be a sentence or a paragraph.

[0040] S2: Input the text to be extracted into the trained first entity extraction model to obtain the entities and entity types in the text to be extracted.

[0041] In an application scenario, a noise data set is used when training the first entity extraction model. First, cross-validation is performed on the training data to select problematic spans, and then all entity spans that may have problems in the noise data set are classified into unlabeled spans and wrongly labeled spans, as well as correctly labeled spans and meaningless spans. Finally, a first entity extraction model is established to model the differences between different types of entity spans, and the cross entropy loss and KL divergence loss are calculated respectively to obtain the final first entity extraction model based on span classification. Specifically, in this embodiment, the noise data set is pre-processed by the second entity extraction model to obtain a set of unlabeled spans, a set of wrongly labeled spans, a set of correctly labeled spans, and a set of meaningless labeled spans, and the set of wrongly labeled spans, the set of correctly labeled spans, and the set of meaningless labeled spans are used to train the first entity extraction model, while the set of unlabeled spans is prohibited from being added to the training process. Specifically, the set of unlabeled spans includes entity spans that are manually unlabeled in the noise data set but are correctly predicted by the second entity extraction model. The set of incorrectly labeled segments includes entity segments whose manually labeled types are different from the corresponding predicted types, the set of correctly labeled segments includes entity segments whose manually labeled types are the same as the corresponding predicted types, and the set of meaningless labeled segments includes entity segments that are not manually labeled and do not belong to the set of missed labeled segments.

[0042] Specifically, in this embodiment, the first entity extraction model is established based on entity segment classification and negative sampling (NegSampling), but it will introduce a width limit for the entity segment, reducing the number of candidate entity segments generated in the training phase of the first entity extraction model. Figure 2 , Figure 2 It is a flowchart of the process of training the first entity extraction model. The process of training the first entity extraction model includes:

[0043] S10: Using the second entity extraction model, obtain a problem segment set consisting of all entity segments whose first prediction results are inconsistent with the manual annotation results in the noise data set, and a correct segment set consisting of entity segments whose first prediction results are consistent with the manual annotation results.

[0044] Specifically, the first prediction result includes the predicted entity and the predicted type, and the manual annotation result includes the manually annotated entity and the manually annotated type. For example, if the content of a predicted text is "My name is Xiao Ming", the corresponding entity segments can be "I", "My name is", "My name is Xiao", "Xiao Ming", etc., and the manual annotation result of the predicted text is "name". The entity type corresponding to one of the entity segments "Xiao Ming" is "name", which is consistent with the manual annotation result of the predicted text, indicating that the entity segment is a correct segment in the correct segment set; while other entity segments, such as "My name", the manual annotation result is also the manual annotation result of the predicted text, which is "name", but its entity type is not "name", then the entity type of the entity segment is inconsistent with the manual annotation result of the predicted text, indicating that the segment is a problem segment in the problem segment set.

[0045] Specifically, in this embodiment, please refer to Figure 3 , Figure 3 yes Figure 2 Schematic diagram of a flow chart of an implementation method of step S10 in FIG. Specifically, step S10 includes:

[0046] S100: Divide the noise data set into multiple sub-data sets.

[0047] Specifically, in this embodiment, the noise data set is disrupted and then divided into N sub-data sets according to a certain ratio. Optionally, N is generally set to 5, and the value can be set according to actual needs.

[0048] S101: Use one of the sub-datasets as test data, and use the remaining sub-datasets as training data.

[0049] Specifically, in this embodiment, N-1 sub-datasets are used as training data and 1 sub-dataset is used as test data, that is, each sub-dataset is used as test data in turn to obtain entity fragments in each sub-dataset whose test results are inconsistent with the manual annotation results.

[0050] S102: Setting a plurality of different random seeds to respectively obtain a plurality of second entity extraction models.

[0051] Specifically, in this embodiment, K different random seeds are set, and K second entity extraction models are obtained by training on N-1 sub-datasets, that is, K second entity extraction models are obtained using K groups of data with different initialization parameters. Optionally, it is generally set to K=3, and the above values ​​can be set according to actual needs. For example, there are 4 sub-datasets a, b, c, and d, and K is set to 3. A is used as test data, and b, c, and d are used as training data. Then b is used as test data, and a, c, and d are used as training data. Finally, c is used as test data, and a, b, and d are used as training data. Finally, 3 second entity extraction models are obtained.

[0052] S103: For each second entity extraction model, train the second entity extraction model using the training data.

[0053] Specifically, in this embodiment, for each second entity extraction model obtained in the above step S102, the second entity extraction model is trained using the training data in the N-1 subset data sets to obtain a trained second entity extraction model.

[0054] Please also read Figure 4 and Figure 5 , Figure 4 yes Figure 3 A schematic flow chart of an implementation method of step S103 in FIG. Figure 5 1 is a schematic diagram of the structure of the second entity extraction model with width restriction. Specifically, the second entity extraction model includes a first pre-trained model 100 and a first fully connected layer 102, and the output of the first pre-trained model 100 is the input of the first fully connected layer 102. The first pre-trained model 100 can be a BERT model, etc., and the first fully connected layer 102 can be an MLP, etc. Step S103 specifically includes:

[0055] S20: Perform word segmentation processing on the text data in the training data and input it into the first pre-training model to obtain a first semantic vector for each word in the text data.

[0056] Specifically, in this embodiment, the text data in the training data is segmented, for example, Figure 5 As shown, a text data in the training data is divided into x0, x1, x i 、x n etc., and replace x0, x1, x i 、x n The words are input into the first pre-training model 100 to obtain the first semantic vectors h0, h1, h i 、h netc. In addition, meaningless words after word segmentation, such as the word "de" (的 in Chinese), will be input into the first pre-training model 100 together.

[0057] Specifically, in this embodiment, an entity width limit is formulated, that is, for a segment, a maximum width is set. When the width of the segment exceeds the threshold, it is defaulted not to be an entity segment. And the width of the segment is also added as a feature to the model training process. Specifically, before step S30, it includes: randomly initializing the first width limit vector matrix W, and it will be continuously updated during the training process. Specifically, the first width limit vector matrix W contains width feature vectors d of multiple entity segments i , for example, Figure 5 d0, d1 shown in i-1 , d w . This way can not only greatly reduce the number of candidate entity segments generated by permutation and combination, which can not only improve the model efficiency, but also provide additional feature information to assist in improving the model effect.

[0058] S21: Perform negative sampling on all first semantic vectors to obtain multiple first combined segments.

[0059] Specifically, in this embodiment, negative sampling NegSampling is performed on all first semantic vectors to obtain multiple first combined segments. Generally speaking, the negative sampling method is to randomly select two first semantic vectors for combination, which will not be elaborated here. Specifically, the first combined segments include manually labeled entity segments and non-manually labeled entity segments. This way abandons the commonly used CRF layer in the entity extraction task, uses the idea of segment classification used in the negative sampling paper, permutes and combines the words in the input text data, and predicts the entity types of each candidate entity segment.

[0060] S22: In response to the width of the first combined segment being less than or equal to the second threshold, obtain the second semantic vector of the first combined segment.

[0061] Specifically, in this embodiment, when combining all possible first combined segments, if the width of the first combined segment exceeds the width limit threshold, it is directly discarded. When the width of the second combined segment does not exceed the width limit threshold, please refer to Figure 5 and Figure 6 , Figure 6 is Figure 4 a schematic flowchart of an implementation manner of step S22 in

[0062] S220: Obtain the first semantic vector of the starting word and the first semantic vector of the ending word in the first combined segment.

[0063] Specifically, for example, when the start index (i.e., the start word) of the first combined fragment in the data is 1 and the end index (i.e., the end word) is i, then the first semantic vector of the start word is h1 and the first semantic vector of the end word is h2. i .

[0064] S221: Concatenate the first semantic vector of the start word, the first semantic vector of the end word, and the corresponding width feature vector to obtain a second semantic vector.

[0065] Specifically, the first semantic vector h1 of the starting word and the first semantic vector h2 of the ending word are i And the corresponding width feature vector d i-1 Concatenate, expressed as (concat(h1, h i ,h i *h1,d i-1 )), and finally obtain the second semantic vector s of the first combined fragment 1,i .

[0066] S23: Input the second semantic vector into the first fully connected layer to obtain a first predicted classification logit vector.

[0067] Specifically, the second semantic vector s obtained in step S221 is 1,i Input to the first fully connected layer 102 to obtain the first predicted classification logit vector o 1,i .

[0068] S24: Adjusting parameters in the second entity extraction model based on the prediction results of the manually annotated entity segments and some of the non-manually annotated entity segments.

[0069] Specifically, the parameters in the second entity extraction model are adjusted based on the prediction results of the manually annotated entity segments and some non-manually annotated entity segments in the first combined segment, wherein the number of non-manually annotated entity segments used for training is a preset probability of the total number of training data in negative sampling. In this embodiment, the wrongly annotated segments in the training data are not considered for the time being.

[0070] In addition, in this embodiment, when adjusting the parameters in the second entity extraction model based on the prediction results of the manually annotated entity segments and some non-manually annotated entity segments in step S34, the parameters in the first width limit vector matrix W are also adjusted.

[0071] S104: Use the trained second entity extraction model to predict the test data to obtain a corresponding first prediction result, and obtain candidate entity fragments that are inconsistent with the manual annotation result according to the first prediction result.

[0072] Specifically, the trained second entity extraction model is used to perform inference prediction on the corresponding one test data to obtain the corresponding first prediction result, and the entity segments in the one test data that are inconsistent with the manual annotation results are obtained according to the first test result, that is, the candidate entity segments that may have problems. In this embodiment, the above steps are performed on all trained second entity extraction models.

[0073] S105: Count all candidate entity segments obtained by all second entity extraction models, and in response to the accumulated number of times the same candidate entity segment exceeds a first threshold, determine the candidate entity segment as a problem segment, and add the problem segment to a problem segment set.

[0074] Specifically, in this embodiment, statistics are taken of all candidate entity fragments that may have problems obtained by all second entity extraction models. When the cumulative number of the same candidate entity fragment exceeds a first threshold, that is, when the number of record appearances of the same candidate entity fragment exceeds K / 2, the candidate entity fragment is determined to be a problem fragment, and the problem fragment is added to the problem fragment set.

[0075] S106: Determine whether all sub-data sets are used as test data in turn.

[0076] Specifically, in this embodiment, in order to ensure that each sub-data set is used as test data, the integrity of the second entity extraction model after training is guaranteed.

[0077] S107: If yes, proceed to step S11.

[0078] Specifically, if all sub-datasets are used as test data for training, the process proceeds to the step of classifying the problem segment set to obtain a set of missingly labeled segments, a set of incorrectly labeled segments, and a set of meaninglessly labeled segments.

[0079] S108: If not, use another sub-dataset as test data, and use the remaining sub-datasets as training data, and return to step S102.

[0080] Specifically, if one of the sub-datasets has not been trained as test data, another sub-dataset is used as test data, and the remaining sub-datasets are used as training data, and the process returns to the step of setting multiple different random seeds to obtain multiple second entity extraction models respectively.

[0081] Through this design approach, the cross-validation strategy is used to model and predict the training data separately, and the problematic prediction results in the training data are selected and classified. In addition, the width limit matrix and negative sampling strategy are introduced in the modeling process. This can greatly reduce the number of candidate entities that are permuted and combined, improve model efficiency, and provide additional feature information to help improve model effects.

[0082] S11: Classify the problem segment set to obtain a set of missing annotation segments, a set of incorrectly annotated segments, and a set of meaningless annotated segments.

[0083] Specifically, after obtaining all entity fragments that may have problems in the training data, we start to classify these problematic fragments. First, we select entity fragments that are not manually labeled but predicted. If these entity fragments are added to the training process, the final model may tend to predict meaningless labeled fragments when predicting the fragments, or reduce the confidence of the model in the fragment prediction. Therefore, these fragments need to be discarded when training the final model. These fragments are recorded as missing labeled fragments, and then we select entity fragments whose manually labeled entity results are inconsistent with the entity results predicted by the model. Since problems are inevitable when manually labeling data, on the whole, the proportion of problems is relatively small, that is, most entities can be correctly labeled. Therefore, when the labeling result is inconsistent with the predicted result, the multi-classification label vector (one-hot vector) of the manually labeled label is not used as the training target, and the logit vector of the probability of predicting the entity fragment by the second entity extraction model is used as the training target of the final model. These fragments are recorded as incorrectly labeled fragments. For example, the manual labeling result of an entity fragment "called xx organization" is "person name", but it should actually be "organization name". Assuming that there are only these two types of entities, the one-hot vector of the fragment is [1,0]. The logit vector of the predicted probability p of the second entity extraction model for the entity fragment is [0.4,0.6].

[0084] like Figure 5 As shown in the figure, during model training, because there may be unlabeled noise fragments in the training data, a negative sampling strategy is adopted. When a fragment is the result of manual labeling, that is, a correctly labeled fragment, it is directly added to the training process. If a fragment is not the result of manual labeling, that is, a meaningless labeled fragment, the manually labeled entity fragment and the non-manually labeled entity fragment are added to the negative sampling training process with a predicted probability p. Finally, the cross entropy Loss is calculated with the manually labeled label, that is, Figure 5 Of course, in other embodiments, it can also be in the second semantic vector s 1,iThe part of the meaningless annotation segment that extracts the prediction probability p before entering the first fully connected layer MLP is not limited in this application.

[0085] S12: Train the first entity extraction model using the incorrect annotation segment set, the correct annotation segment set, and the meaningless annotation segment set, and prohibit training the first entity extraction model using the missed annotation segment set.

[0086] Specifically, after selecting the missed annotation segment set and the incorrect annotation segment set in the training data, these two segment sets are the noise data in the training data. In addition, the training data can also construct a new correct annotation segment set, which represents the entity segments manually annotated and not in the incorrect annotation segment set, and all segment sets that are not annotated as entities and not in the missed annotation segment set, denoted as meaningless annotation segments. These two segment sets are the relatively correct data parts in the training data. After constructing these 4 segment sets for each piece of training data, a first entity extraction model for noise data can be established. Please refer to Figure 7 and Figure 8 , Figure 7 is Figure 2 a schematic flowchart of an implementation manner of step S12 in Figure 8 a schematic structural diagram of the first entity extraction model for noise data. Specifically, in this embodiment, the first entity extraction model includes a second pre-training model 200 and a second fully connected layer 202, and the output of the second pre-training model 200 is the input of the second fully connected layer 202. The second pre-training model 200 can be a BERT model or the like, and the second fully connected layer 202 can be an MLP or the like. Specifically, step S12 includes:

[0087] S120: Tokenize the text data in the noise data set and input it into the second pre-training model to obtain the third semantic vector of each word in the text data.

[0088] Specifically, tokenize the text data in the noise data set. For example, as Fig.10 shown, divide a text data in the noise data set into words such as x0, x1, x i , x n and input words such as x0, x1, x i , x n into the second pre-training model 200 to obtain the third semantic vectors h0, h1, h i , h n and so on. In addition, meaningless words after tokenization, such as the word "de" etc., will be input into the second pre-training model 200 together.

[0089] Specifically, in this embodiment, similar to the method of step S20, an entity width restriction is formulated, that is, for a segment, a maximum width is set, and when the width of the segment exceeds the threshold, it is not considered as an entity segment by default. And the width of the segment is also added as a feature in the model training process. Specifically, a width restriction vector matrix W is randomly initialized and continuously updated during the training process. Specifically, the width restriction vector matrix W contains the width feature vectors d of multiple entity segments. i ,For example, Figure 8 The d0, d1, and d i-1 d w This method can not only greatly reduce the number of candidate entity fragments that are arranged and combined, and improve the efficiency of the model, but also provide additional feature information to help improve the model effect.

[0090] S121: Perform negative sampling combination on all third semantic vectors to obtain a plurality of second combined segments.

[0091] Specifically, in this embodiment, similar to the method of step S21, all third semantic vectors are negatively sampled NegSampling to obtain multiple second combined fragments. Generally speaking, the negative sampling method is to arbitrarily select two third semantic vectors for combination, which will not be repeated here. This method abandons the CRF layer commonly used in entity extraction tasks, uses the idea of ​​fragment classification used in negative sampling papers, arranges and combines the words in the input text data, and predicts the entity type of each candidate entity fragment.

[0092] S122: In response to the width of the second combined fragment being less than or equal to the second threshold, obtaining a fourth semantic vector of the second combined fragment.

[0093] Specifically, in this embodiment, when all possible second combined segments are combined, if the width of the second combined segment exceeds the width limit threshold, it is directly discarded. Similar to the approach of steps S220-S221, when the width of the second combined segment does not exceed the width limit threshold, the third semantic vector h1 of the starting word 1 and the third semantic vector h2 of the ending word i in the second combined segment are obtained. i , and then the third semantic vector h1 of the starting word and the third semantic vector h i And the corresponding width feature vector d i-1 Concatenate, expressed as (concat(h1, h i ,h i *h1,d i-1 )), and finally obtain the fourth semantic vector s of the second combined fragment 1,i .

[0094] S123: Input the fourth semantic vector into the second fully connected layer to obtain a second predicted classification logit vector.

[0095] Specifically, the fourth semantic vector s 1,i After the second fully connected layer 202, the second predicted classification logit vector o of the segment is classified into various types of entities. 1,i , specifically the second prediction classification logit vector o for correctly labeled segments, meaningless labeled segments, incorrectly labeled segments, and missed labeled segments 1,i .

[0096] This takes into account the situation where there are labeling errors in the labeled data. For fragments where the model prediction results are inconsistent with the manually labeled results in length or type in cross-validation, the predicted logit vectors of multiple second entity extraction models on the fragment are used as the learning objectives for training the first entity extraction model for noisy data, thereby using the cross-validation strategy to correct the incorrectly labeled data.

[0097] S124: In response to the span of the second combined fragment being in the correctly labeled fragment set, the meaningless labeled fragment set, and the incorrectly labeled fragment set, obtain a corresponding first loss according to the set type to which at least part of the second combined fragment belongs, and adjust parameters in the first entity extraction model based on the first loss.

[0098] In general, different from the construction process of the second entity extraction model, four sets of fragments are constructed, such as Figure 8 As shown in the figure, when the span of a segment is in the set of correctly labeled segments and the set of meaningless labeled segments, the cross entropy loss between the second predicted classification logit vector and the multi-classification label vector Taged One-hot Vector formed by the manually labeled label will be calculated, and the KL divergence loss between the second predicted classification logit vector and the logit vector of the segment predicted by the second entity extraction model will be calculated; when the span of a segment is in the set of incorrectly labeled segments, only the KL divergence loss between the second predicted classification logit vector and the logit vector of the segment predicted by the second entity extraction model will be calculated. At the same time, in order to maintain the balance of positive and negative samples in the calculation process, some segments in the meaningless segment set will also be introduced by negative sampling; when the span of a segment is in the set of missing labeled segments, the segment is prohibited from being added to the training process of the first entity extraction model, which is represented by Φ in the figure. In this way, a highly generalized entity extraction method can be trained without human participation, and a complete set of data processing and modeling solutions has been formed.

[0099] Specifically, in this embodiment, Figure 8As shown, the step of obtaining the corresponding first loss according to the set type to which at least part of the second combined fragments belong in step S124 includes: A. when the span of the second combined fragment is in the correctly labeled fragment set or the meaningless labeled fragment set, obtaining the first cross entropy loss between the second predicted classification logit vector corresponding to the second combined fragment and the multi-classification label vector Taged One-hot Vector of the manually labeled type, and obtaining the first KL divergence loss between the second predicted classification logit vector corresponding to the second combined fragment and the first predicted classification logit vector (Modellogit Vector) obtained by the second entity extraction model, and taking the first cross entropy loss and the first KL divergence loss as the first loss; B. when the span of the second combined fragment is in the incorrectly labeled fragment set, obtaining the second KL divergence loss between the second predicted classification logit vector corresponding to the second combined fragment and the first predicted classification logit vector obtained by the second entity extraction model, and taking the second KL divergence loss as the first loss.

[0100] Specifically, in this embodiment, in response to the span of the second combined segment in the correct labeled segment set, the meaningless labeled segment set and the wrong labeled segment set, the step of obtaining the corresponding first loss according to the set type to which at least part of the second combined segment belongs in step A and step B includes: obtaining the first loss of all second combined segments with spans in the correct labeled segment set, the first loss of all second combined segments with spans in the wrong labeled segment set, and the first loss of some second combined segments with spans in the meaningless labeled segment set. In this way, the text segments predicted by the model but not manually labeled are not added to the model training process. In this way, the influence of unlabeled noise in the data on model training is reduced, and there is no need for manual participation in modifying the original training data labeling results.

[0101] By performing cross-validation on the training data, multiple second entity extraction models are obtained, and then the model is used to make predictions on the training data to find text fragments that do not match the manual labels. Then, the first entity extraction model for noisy data is trained. The text fragments that are predicted by the model but not manually labeled can be excluded from the model training process. In this way, the impact of unlabeled noise in the data on model training is reduced, and there is no need for manual intervention to modify the original training data labeling results.

[0102] See also Fig. 9 , Fig. 9 : is a flowchart of the training method of the first entity extraction model of the present application. Specifically, the training method of the first entity extraction model includes:

[0103] S30: Obtain a noise data set.

[0104] Specifically, the above-mentioned noise data set includes a set of missing annotation segments, a set of incorrectly annotated segments, a set of correctly annotated segments, and a set of meaningless annotated segments.

[0105] S31: Using the second entity extraction model, a set of missed-annotated segments, a set of incorrectly annotated segments, a set of correctly annotated segments, and a set of meaningless annotated segments are obtained in the noise data set.

[0106] Specifically, the missed annotation segment set includes entity segments that are manually omitted from the noise data set but are correctly predicted by the second entity extraction model; the incorrectly annotated segment set includes entity segments whose manually annotated types are different from the corresponding predicted types; the correctly annotated segment set includes entity segments whose manually annotated types are the same as the corresponding predicted types; and the meaningless annotated segment set includes entity segments that are not manually annotated and do not belong to the missed annotation segment set.

[0107] S32: Using the incorrectly labeled segment set, the correctly labeled segment set and the meaningless labeled segment set to train the first entity extraction model, while prohibiting the missing labeled segment set from training the first entity extraction model.

[0108] Specifically, the execution process of S32 is similar to the execution process of step S12, which will not be repeated here.

[0109] Through this design approach, the impact of unlabeled noise in the data on model training can be reduced, and a highly generalized entity extraction method can be trained without human intervention to modify the original training data annotation results, forming a complete set of data processing and modeling solutions.

[0110] See also Fig.10 , Fig.10 Schematic diagram of a framework of an embodiment of the entity extraction device of the present application. Specifically, the entity extraction device includes:

[0111] The acquisition module 10 is used to acquire the text to be extracted.

[0112] The entity module 12 is connected to the acquisition module 10 and is used to input the text to be extracted into the trained first entity extraction model to obtain the entities in the text to be extracted and the types of the entities.

[0113] The first training module 14 is connected to the entity module 12 and the acquisition module 10, and is used to train the first entity extraction model using the noise data set; and the noise data set is pre-processed by the second entity extraction model to obtain a set of missing annotation segments, a set of incorrectly annotated segments, a set of correctly annotated segments, and a set of meaningless annotated segments, and the first entity extraction model is trained using the set of incorrectly annotated segments, the set of correctly annotated segments, and the set of meaningless annotated segments, while the missing annotation segment set is prohibited from being added to the training process; wherein the missing annotation segment set includes entity segments that are manually omitted from the noise data set but correctly predicted by the second entity extraction model, the set of incorrectly annotated segments includes entity segments whose manually annotated types are different from the corresponding predicted types, the set of correctly annotated segments includes entity segments whose manually annotated types are the same as the corresponding predicted types, and the set of meaningless annotated segments includes entity segments that are not manually annotated and do not belong to the missing annotation segment set. Of course, in other embodiments, the first training module 14 can also be connected only to the acquisition module 10, and the first entity extraction model is trained using the noise data set before acquiring the text to be extracted.

[0114] In one embodiment, the first training module 14 includes an acquisition module, a classification module and a first sub-training module that are sequentially connected to each other. The acquisition module is used to use the second entity extraction model to obtain a problem segment set consisting of all entity segments whose first prediction results are inconsistent with the manual annotation results in the noise data set, and a correct segment set consisting of entity segments whose first prediction results are consistent with the manual annotation results; wherein the first prediction result includes a predicted entity and a predicted type, and the manual annotation result includes a manually annotated entity and a manually annotated type. The classification module is used to classify the problem segment set to obtain a missing annotation segment set, an incorrectly annotated segment set and a meaningless annotated segment set. The first sub-training module is used to train the first entity extraction model using the incorrectly annotated segment set, the correctly annotated segment set and the meaningless annotated segment set, and it is forbidden to train the first entity extraction model using the missing annotation segment set.

[0115] Optionally, the acquisition module specifically includes a segmentation module, a data module, a second entity extraction model module, a second training module, candidate entity segments, a statistical module and a first judgment module which are sequentially connected to each other. Specifically, the segmentation module is used to segment the noise data set into multiple sub-datasets; the data module is used to use one of the sub-datasets as test data and the remaining sub-datasets as training data; the second entity extraction model module is used to set multiple different random seeds to obtain multiple second entity extraction models respectively; the second training module is used to train the second entity extraction model using the training data for each second entity extraction model; the candidate entity fragment is used to predict the test data using the trained second entity extraction model to obtain the corresponding first prediction result, and obtain the candidate entity fragment inconsistent with the manual annotation result according to the first prediction result; the statistical module is used to count all the candidate entity fragments obtained by all the second entity extraction models, and in response to the cumulative number of times the same candidate entity fragment exceeds the first threshold, the candidate entity fragment is determined as a problem fragment, and the problem fragment is added to the problem fragment set; the first judgment module is used to determine whether all sub-datasets are used as test data in turn; if not, another sub-dataset is used as test data, and the remaining sub-datasets are used as training data, and return to the step of setting multiple different random seeds to obtain multiple second entity extraction models respectively.

[0116] In another embodiment, the second entity extraction model includes a first pre-trained model and a first fully connected layer, and the output of the first pre-trained model is the input of the first fully connected layer. Optionally, the second training module includes a first word segmentation module, a first negative sampling module, a second judgment module, a first prediction classification logit vector module and a first adjustment module that are sequentially connected to each other. Specifically, the first word segmentation module is used to perform word segmentation processing on the text data in the training data and then input it into the first pre-training model to obtain the first semantic vector of each word in the text data; the first negative sampling module is used to perform negative sampling on all first semantic vectors to obtain multiple first combined fragments; wherein the first combined fragment includes manually annotated entity fragments and non-manually annotated entity fragments; the second judgment module is used to obtain the second semantic vector of the first combined fragment in response to the width of the first combined fragment being less than or equal to the second threshold; the first prediction classification logit vector module is used to input the second semantic vector into the first fully connected layer to obtain the first prediction classification logit vector; the first adjustment module is used to adjust the parameters in the second entity extraction model based on the prediction results of the manually annotated entity fragments and some non-manually annotated entity fragments.

[0117] In another embodiment, the entity extraction device also includes a width feature module, which is connected to one end of the first word segmentation module and is used to randomly initialize the first width limit vector matrix; wherein the first width limit vector matrix contains multiple width feature vectors. In this embodiment, the above-mentioned second judgment module includes a first semantic vector module and a second semantic vector module that are connected to each other, wherein the first semantic vector module is used to obtain the first semantic vector of the starting word and the first semantic vector of the ending word in the first combined fragment, and the second semantic vector module is used to splice the first semantic vector of the starting word, the first semantic vector of the ending word and the corresponding width feature vector to obtain the second semantic vector. In addition, in this embodiment, when adjusting the parameters in the second entity extraction model based on the prediction results of the manually annotated entity fragments and some non-manually annotated entity fragments, the parameters in the first width limit vector matrix are adjusted at the same time.

[0118] In another embodiment, the first entity extraction model includes a second pre-training model and a second fully connected layer, and the output of the second pre-training model is the input of the second fully connected layer. The above-mentioned first sub-training module includes a second word segmentation module, a second negative sampling module, a third judgment module, a second prediction classification logit vector module and a second adjustment module which are sequentially connected to each other. Specifically, in this embodiment, the second word segmentation module is used to input the text data in the noise data set into the second pre-training model after word segmentation processing to obtain the third semantic vector of each word in the text data; the second negative sampling module is used to negatively sample and combine all third semantic vectors to obtain multiple second combined segments; the third judgment module is used to obtain the fourth semantic vector of the second combined segment in response to the width of the second combined segment being less than or equal to the second threshold; the second prediction classification logit vector module is used to input the fourth semantic vector into the second fully connected layer to obtain the second prediction classification logit vector; the second adjustment module is used to obtain the corresponding first loss according to the set type to which at least part of the second combined segment belongs in response to the span of the second combined segment in the correct labeled segment set, the meaningless labeled segment set and the wrong labeled segment set, and adjust the parameters in the first entity extraction model based on the first loss.

[0119] Specifically, in the present embodiment, the second adjustment module also includes a first loss module, which is used to obtain, in response to the span of the second combined segment being in the correctly labeled segment set or the meaningless labeled segment set, a first cross entropy loss between the second predicted classification logit vector corresponding to the second combined segment and the multi-classification label vector of the manually labeled type, and a first KL divergence loss between the second predicted classification logit vector corresponding to the second combined segment and the first predicted classification logit vector obtained by the second entity extraction model, and use the first cross entropy loss and the first KL divergence loss as the first loss; in addition, in the present embodiment, the first loss is also used to obtain, in response to the span of the second combined segment being in the incorrectly labeled segment set, a second KL divergence loss between the second predicted classification logit vector corresponding to the second combined segment and the first predicted classification logit vector obtained by the second entity extraction model, and use the second KL divergence loss as the first loss.

[0120] In yet another embodiment, the first loss module is further used to obtain the first loss of all second combined segments that span in a correctly labeled segment set, the first loss of all second combined segments that span in an incorrectly labeled segment set, and the first loss of some second combined segments that span in a meaningless labeled segment set.

[0121] See also Fig.11 , Fig.11 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device includes a memory 20 and a processor 22 coupled to each other. Specifically, in the present embodiment, the memory 20 stores program instructions, and the processor 22 is used to execute the program instructions to implement the entity extraction method mentioned in any of the above embodiments and the training method of the first entity extraction model. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 22 can also be called a CPU (Central Processing Unit). The processor 22 may be an integrated circuit chip with signal processing capabilities. The processor 22 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 22 may be implemented by multiple integrated circuit chips.

[0122] See also Fig.12 , Fig.12 It is a structural diagram of an implementation of a storage device of the present application. The storage device 30 stores program instructions 300 that can be executed by a processor. The program instructions 300 are used to implement the entity extraction method mentioned in any of the above embodiments and the training method of the first entity extraction model.

[0123] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0127] In summary, different from the prior art, the entity extraction method provided by the present application includes: inputting the acquired text to be extracted into the trained first entity extraction model to obtain the entities and entity types in the text to be extracted; wherein, the first entity extraction model is trained using a noise data set, and the noise data set is pre-trained by the second entity extraction model to obtain a set of missing labeled segments, a set of incorrectly labeled segments, a set of correctly labeled segments, and a set of meaningless labeled segments, and the first entity extraction model is trained using the set of incorrectly labeled segments, the set of correctly labeled segments, and the set of meaningless labeled segments, while the missing labeled segments are prohibited from being added to the training process. Through this design method, not only the unlabeled noise in the training data can be modeled, but also the incorrectly labeled noise in the training data can be modeled, thereby improving the generalization performance of the model for noisy data, and a highly generalized entity extraction method can be trained without human intervention.

[0128] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An entity extraction method, characterized in that: include: Get the text to be extracted; The text to be extracted is input into the trained first entity extraction model to obtain entities in the text to be extracted and the types of the entities; wherein, a noise data set is used when training the first entity extraction model; and the noise data set is preliminarily passed through the second entity extraction model to obtain a set of missing annotation segments, a set of incorrectly annotated segments, a set of correctly annotated segments, and a set of meaningless annotated segments, and the first entity extraction model is trained using the set of incorrectly annotated segments, the set of correctly annotated segments, and the set of meaningless annotated segments, while the set of missing annotation segments is prohibited from being added to the training process; wherein, the set of missing annotation segments includes entity segments that are manually omitted from the noise data set but correctly predicted by the second entity extraction model, the set of incorrectly annotated segments includes entity segments whose manually annotated types are different from the corresponding predicted types, the set of correctly annotated segments includes entity segments whose manually annotated types are the same as the corresponding predicted types, and the set of meaningless annotated segments includes entity segments that are not manually annotated and do not belong to the set of missing annotation segments; The process of training the first entity extraction model includes: The second entity extraction model is used to obtain a problem segment set consisting of entity segments whose first prediction results are inconsistent with the manually labeled results in the noise data set, and a correct segment set consisting of entity segments whose first prediction results are consistent with the manually labeled results; wherein the first prediction result includes a predicted entity and a predicted type, and the manually labeled result includes a manually labeled entity and a manually labeled type; Classifying the problem segment set to obtain the missing-annotation segment set, the wrongly-annotated segment set, and the meaningless-annotated segment set; The first entity extraction model is trained using the set of incorrectly labeled segments, the set of correctly labeled segments, and the set of meaningless labeled segments, and it is prohibited to train the first entity extraction model using the set of omitted labeled segments.

2. The entity extraction method according to claim 1, characterized in that: The step of using the second entity extraction model to obtain a problem segment set consisting of all entity segments in the noise data set whose prediction results are inconsistent with manual annotation results includes: Splitting the noise data set into a plurality of sub-data sets; Using one of the sub-datasets as test data and the remaining sub-datasets as training data; Setting a plurality of different random seeds to obtain a plurality of second entity extraction models respectively; For each of the second entity extraction models, training the second entity extraction model using the training data; Using the trained second entity extraction model to predict the test data to obtain a corresponding first prediction result, and obtaining a candidate entity segment that is inconsistent with a manual annotation result according to the first prediction result; Counting all the candidate entity segments obtained by all the second entity extraction models, and in response to the accumulated number of times the same candidate entity segment exceeds a first threshold, determining the candidate entity segment as a problem segment, and adding the problem segment to the problem segment set; Determining whether all the sub-data sets are used as the test data in turn; If not, another of the sub-datasets is used as the test data, and the remaining sub-datasets are used as training data, and the process returns to the step of setting a plurality of different random seeds to respectively obtain a plurality of second entity extraction models.

3. The entity extraction method according to claim 2, characterized in that: The second entity extraction model includes a first pre-trained model and a first fully connected layer, and the output of the first pre-trained model is the input of the first fully connected layer; the step of training the second entity extraction model using the training data includes: Performing word segmentation processing on the text data in the training data and then inputting the result into the first pre-training model to obtain a first semantic vector for each word in the text data; Performing negative sampling on all the first semantic vectors to obtain a plurality of first combined segments; wherein the first combined segments include manually annotated entity segments and non-manually annotated entity segments; In response to the width of the first combined fragment being less than or equal to a second threshold, obtaining a second semantic vector of the first combined fragment; Inputting the second semantic vector into the first fully connected layer to obtain a first predicted classification logit vector; Parameters in the second entity extraction model are adjusted based on the prediction results of the manually annotated entity segments and part of the non-manually annotated entity segments.

4. The entity extraction method according to claim 3, characterized in that: Before the step of performing word segmentation on the training text in the training data and then inputting the training text into the first pre-training model to obtain the first semantic vector of each word in the training text, the method includes: randomly initializing a first width-limited vector matrix; wherein the first width-limited vector matrix includes a plurality of width feature vectors; In response to the width of the first combined segment being less than or equal to a second threshold, the step of obtaining a second semantic vector of the first combined segment includes: obtaining a first semantic vector of a starting word and a first semantic vector of an ending word in the first combined segment; concatenating the first semantic vector of the starting word, the first semantic vector of the ending word, and a corresponding width feature vector to obtain the second semantic vector; When adjusting the parameters in the second entity extraction model based on the prediction results of the manually annotated entity segments and part of the non-manually annotated entity segments, the parameters in the first width limit vector matrix are also adjusted.

5. The entity extraction method according to claim 1, characterized in that: The first entity extraction model includes a second pre-trained model and a second fully connected layer, and the output of the second pre-trained model is the input of the second fully connected layer; the step of using the set of incorrectly labeled segments, the set of correctly labeled segments, and the set of meaningless labeled segments to train the first entity extraction model, and prohibiting the step of using the set of missing labeled segments to train the first entity extraction model, includes: The text data in the noise data set is processed by word segmentation and then input into the second pre-training model to obtain a third semantic vector of each word in the text data; Performing negative sampling combination on all the third semantic vectors to obtain a plurality of second combined segments; In response to the width of the second combined fragment being less than or equal to a second threshold, obtaining a fourth semantic vector of the second combined fragment; Inputting the fourth semantic vector into the second fully connected layer to obtain a second predicted classification logit vector; In response to the span of the second combined fragment in the correctly labeled fragment set, the meaningless labeled fragment set and the incorrectly labeled fragment set, a corresponding first loss is obtained according to the set type to which at least part of the second combined fragment belongs, and parameters in the first entity extraction model are adjusted based on the first loss.

6. The entity extraction method according to claim 5, characterized in that: The step of obtaining the corresponding first loss according to the set type to which at least part of the second combined fragments belong comprises: In response to the span of the second combined segment being in the correctly labeled segment set or the meaningless labeled segment set, obtaining a first cross entropy loss between a second predicted classification logit vector corresponding to the second combined segment and a multi-classification label vector of a manually labeled type, and obtaining a first KL divergence loss between the second predicted classification logit vector corresponding to the second combined segment and a first predicted classification logit vector obtained by the second entity extraction model, and using the first cross entropy loss and the first KL divergence loss as the first loss; In response to the span of the second combined fragment being in the set of incorrectly labeled fragments, a second KL divergence loss between a second predicted classification logit vector corresponding to the second combined fragment and a first predicted classification logit vector obtained by the second entity extraction model is obtained, and the second KL divergence loss is used as the first loss.

7. The entity extraction method according to claim 6, characterized in that: The step of obtaining corresponding first losses according to the set type to which at least part of the second combined segments belong in response to the span of the second combined segment being in the correctly labeled segment set, the meaningless labeled segment set, and the incorrectly labeled segment set, comprises: The first losses of all the second combined segments in the correctly labeled segment set, the first losses of all the second combined segments in the incorrectly labeled segment set, and the first losses of some of the second combined segments in the meaningless labeled segment set are obtained.

8. A training method for a first entity extraction model, characterized in that: include: Get a noisy dataset; The second entity extraction model is used to obtain a set of missing annotation segments, a set of incorrectly annotated segments, a set of correctly annotated segments, and a set of meaningless annotated segments in the noise data set; wherein the set of missing annotation segments includes entity segments that are manually omitted from the noise data set but are correctly predicted by the second entity extraction model, the set of incorrectly annotated segments includes entity segments whose manually annotated types are different from the corresponding predicted types, the set of correctly annotated segments includes entity segments whose manually annotated types are the same as the corresponding predicted types, and the set of meaningless annotated segments includes entity segments that are not manually annotated and do not belong to the set of missing annotation segments; Using the set of incorrectly labeled segments, the set of correctly labeled segments, and the set of meaningless labeled segments to train the first entity extraction model, while prohibiting the set of omitted labeled segments from training the first entity extraction model; The process of training the first entity extraction model includes: The second entity extraction model is used to obtain a problem segment set consisting of entity segments whose first prediction results are inconsistent with the manually labeled results in the noise data set, and a correct segment set consisting of entity segments whose first prediction results are consistent with the manually labeled results; wherein the first prediction result includes a predicted entity and a predicted type, and the manually labeled result includes a manually labeled entity and a manually labeled type; Classifying the problem segment set to obtain the missing-annotation segment set, the wrongly-annotated segment set, and the meaningless-annotated segment set; The first entity extraction model is trained using the set of incorrectly labeled segments, the set of correctly labeled segments, and the set of meaningless labeled segments, and it is prohibited to train the first entity extraction model using the set of omitted labeled segments.

9. An entity extraction device, characterized in that: include: An acquisition module is used to obtain the text to be extracted; An entity module, connected to the acquisition module, for inputting the text to be extracted into the trained first entity extraction model to obtain entities in the text to be extracted and the types of the entities; A first training module is connected to the entity module and the acquisition module, and is used to train the first entity extraction model using a noise data set; and the noise data set is pre-processed by the second entity extraction model to obtain a set of missing annotation segments, and the missing annotation segment set is prohibited from being added to the training process when training the first entity extraction model; wherein the missing annotation segment set includes entity segments that are manually omitted from the noise data set but correctly predicted by the second entity extraction model, the wrongly annotated segment set includes entity segments whose manually annotated types are different from the corresponding predicted types, the correctly annotated segment set includes entity segments whose manually annotated types are the same as the corresponding predicted types, and the meaningless annotated segment set includes entity segments that are not manually annotated and do not belong to the missing annotation segment set; wherein the first training module includes an acquisition module, a classification module and a first sub-training module that are sequentially connected to each other. Module; wherein the acquisition module is used to use the second entity extraction model to obtain a problem segment set consisting of all entity segments whose first prediction results are inconsistent with the manual annotation results in the noise data set, and a correct segment set consisting of entity segments whose first prediction results are consistent with the manual annotation results; wherein the first prediction result includes a predicted entity and a predicted type, and the manual annotation result includes a manually annotated entity and a manually annotated type; the classification model is used to classify the problem segment set to obtain the missing annotation segment set, the incorrectly annotated segment set and the meaningless annotated segment set; the first sub-training module is used to use the incorrectly annotated segment set, the correctly annotated segment set and the meaningless annotated segment set to train the first entity extraction model, and it is prohibited to use the missing annotation segment set to train the first entity extraction model.

10. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the entity extraction method described in any one of claims 1 to 7, and the training method of the first entity extraction model described in claim 8.

11. A storage device, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the entity extraction method described in any one of claims 1 to 7 and the training method of the first entity extraction model described in claim 8.

Images