A method and system for evaluating and controlling the quality of named entity recognition data annotation

Abstract

The present invention relates to a method and system for evaluating and controlling the quality of named entity recognition data annotation, wherein the method comprises: dividing data into N sub-training sets; using the N sub-training sets to train N first named entity recognition models respectively; using the N first named entity recognition models to predict a given annotation corpus to obtain N results; voting for each entity prediction result in the N results; determining the error type according to the voting result and the annotator's annotation result, and providing a suspected error prompt. The present invention can intelligently analyze the real-time annotation results of the annotator during the annotation process, guide the annotator to perform reliable annotation, minimize the problems of inconsistent annotation rules and missing annotations before and after the annotator, increase the controllability of the annotation process, control the annotation quality of the annotator during the annotation process, provide immediate error feedback, and reduce the waste of the annotator's time.

Description

Technical Field

[0001] The present invention relates to the technical field of named entity recognition (NER), and in particular to a method and system for evaluating and controlling the annotation quality of named entity recognition data. Background Art

[0002] In recent years, with the continuous development and application of basic natural language processing technology and knowledge graph technology, the demand for text annotation has also increased significantly. For many supervised tasks, high-quality data is the basis for the model to achieve good performance.

[0003] As we all know, data labeling requires a lot of manpower and financial resources, and different labelers have different labeling results under the same labeling rules. Poor data quality not only causes time and money losses to the business side, but also has a negative impact on model performance. For labor-intensive work such as data labeling, the industry currently generally uses a crowdsourcing-based solution of "labeling platform + model pre-labeling + manual labeling" to maximize labeling efficiency and use unified quality assessment indicators to evaluate the quality of labeled data.

[0004] There are a lot of errors in the manually annotated data generated using general annotation solutions, which will undoubtedly affect the model effect. Summary of the invention

[0005] In response to the above technical problems, the present invention provides a method and system for evaluating and controlling the quality of named entity recognition data annotation.

[0006] The technical solution of the present invention to solve the above technical problems is as follows:

[0007] According to a first aspect of an embodiment of the present invention, a method for evaluating and controlling the quality of named entity recognition data annotation is provided, comprising:

[0008] Divide the data into N training sets;

[0009] Using the N sub-training sets to train N first named entity recognition models respectively;

[0010] Use N of the first named entity recognition models to predict a given annotated corpus to obtain N results;

[0011] Voting for each entity prediction result among the N results;

[0012] The error type is determined based on the voting results and the annotator's annotation results, and a suspected error prompt is given.

[0013] Based on the above technical solution, the present invention can also be improved as follows.

[0014] Furthermore, the data is divided into N training sets, specifically including:

[0015] The N-molecule training set is obtained through N-fold crossover.

[0016] Further, the error type is determined according to the voting result and the annotator's annotation result, specifically including:

[0017] If the annotator's annotation result is empty, but the model's voting result is not empty, the error type is determined to be suspected missing label; and / or

[0018] If the annotator's annotated location information completely matches the voting location information, but the labels do not completely match, the error type is determined to be a suspected category error; and / or

[0019] If it is not one of the two suspected errors mentioned above, and there is a difference between the annotation result of the annotator and the voting result of the model, the error type is judged as suspected other errors.

[0020] Furthermore, it also includes:

[0021] After all the annotators have completed the annotation, they will first vote on the annotation results of the entire corpus as a whole, and use the first voting result of the entire corpus as the quasi-answer;

[0022] Taking N folds of the entire corpus to train N second named entity recognition models;

[0023] Use N of the second named entity recognition models to predict the entire corpus and vote to obtain a reference answer;

[0024] By comparing the quasi-answer with the reference answer, a data quality indicator including the source of error is generated.

[0025] According to a second aspect of an embodiment of the present invention, a named entity recognition data annotation quality assessment and control system is provided, comprising:

[0026] A partitioning module is used to divide the data into N training sets;

[0027] A first training module, used to train N first named entity recognition models using the N sub-training sets respectively;

[0028] A first prediction and voting module, configured to use the N first named entity recognition models to predict a given annotated corpus, obtain N results, and vote for each entity prediction result in the N results;

[0029] The error prompt module is used to determine the error type based on the voting results and the annotation results of the annotator, and to provide suspected error prompts.

[0030] Furthermore, the partitioning module is specifically used to obtain N-molecule training sets through N-fold crossover.

[0031] Further, the error prompt module specifically includes:

[0032] The first determination unit is used to determine the error type as suspected missing label if the labeling result of the annotator is empty and the model voting result is not empty;

[0033] The second determination unit is used to determine the error type as a suspected category error if the annotator's annotation position information is completely consistent with the voting position information, but the labels are not completely consistent;

[0034] The third determination unit is used to determine the error type as suspected other errors if it is not the above two suspected errors and there is a difference between the annotator's annotation result and the model voting result.

[0035] Furthermore, it also includes:

[0036] The voting module is used to vote on the annotation results of the entire corpus as a whole after all annotators have completed the annotation. The first voting result of the entire corpus is used as the quasi-answer;

[0037] A second training module is used to train N second named entity recognition models by taking N folds of the entire corpus;

[0038] A second prediction and voting module, used to use the N second named entity recognition models to predict the entire corpus and vote to obtain a reference answer;

[0039] The comparison module is used to generate a data quality indicator including the source of error by comparing the quasi-answer with the reference answer.

[0040] According to a third aspect of an embodiment of the present invention, a terminal device is provided, including:

[0041] Processor; and

[0042] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0043] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method as described above.

[0044] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0045] It can intelligently analyze the real-time annotation results of the annotators during the annotation process, guide the annotators to make reliable annotations, minimize the problems of inconsistent annotation rules and missed annotations before and after, increase the controllability of the annotation process, control the annotation quality of the annotators during the annotation process, provide instant error feedback, and reduce the waste of annotators' time.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a method for evaluating and controlling the quality of named entity recognition data annotation provided by an embodiment of the present invention;

[0048] Figure 2 A flowchart of another method for evaluating and controlling the quality of named entity recognition data annotation provided by an embodiment of the present invention;

[0049] Figure 3 A visual overview interface of annotation quality. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0051] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0052] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0053] Figure 1 A flowchart of a method for evaluating and controlling the quality of named entity recognition data annotation provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0054] 110. Divide the data into N training sets;

[0055] 120. Use the N sub-training sets to train N first named entity recognition models respectively;

[0056] 130. Use the N first named entity recognition models to predict a given annotated corpus to obtain N results;

[0057] 140. Voting for each entity prediction result among the N results;

[0058] 150. Determine the error type based on the voting results and the annotator's annotation results, and provide a suspected error prompt.

[0059] Specifically, N sub-datasets (sub-training sets) can be obtained through an N-fold crossover method, and N BERT-CRF models can be trained using these N sub-training sets respectively. These N models are used to predict the given annotated corpus, and a vote is performed for each entity prediction result among the N results. Prompts for suspected errors are given based on the voting results and the annotator's annotation results, and prompts for suspected annotation errors are added to the annotators, so as to guide the annotators to quickly become familiar with the annotation rules and develop good annotation habits.

[0060] Among them, the annotation error types are divided into three types: "suspected missing label", "suspected category error" and "suspected other errors", which are explained as follows:

[0061] Suspected missing label judgment criteria: the annotator's annotation result is empty, but the model's voting result is not empty. Taking the BIO annotation mode as an example, if the annotator's annotation result is "O", but the model's voting result is not "O".

[0062] Suspected category error: The annotator’s annotated location information is completely consistent with the voting location information, but the labels are not completely consistent. Taking the BIO annotation mode as an example, consider the sentence "Shangri-La is great", the annotator’s annotation result may be "B-BRAND I-BRAND I-BRAND I-BRAND OOO", while the model’s voting result is "B-LOC I-LOC I-LOC I-LOCO OO". At this time, it is necessary to remind that the segment "Shangri-La" should be "LOC" for location rather than "BRAND" for brand.

[0063] Suspected other errors: If it is not one of the two suspected errors mentioned above, and there is a difference between the annotator's annotation results and the model voting results, it will be prompted as suspected other errors.

[0064] The embodiment of the present invention provides a method for evaluating and controlling the quality of named entity recognition data annotation, which can intelligently analyze the real-time annotation results of the annotator during the annotation process, guide the annotator to perform reliable annotation, minimize the problems of inconsistent annotation rules and missed annotations before and after the annotator, increase the controllability of the annotation process, control the annotation quality of the annotator during the annotation process, provide immediate error feedback, and reduce the waste of the annotator's time.

[0065] Optionally, in this embodiment, the method further includes:

[0066] 160. After all the annotators have completed the annotation, they will first vote on the annotation results of the entire corpus as a whole, and the first voting result of the entire corpus will be used as the quasi-answer;

[0067] 170. Taking N folds of the entire corpus to train N second named entity recognition models;

[0068] 180. Use the N second named entity recognition models to predict the entire corpus and vote to obtain a reference answer;

[0069] 190. Generate a data quality indicator including the source of error by comparing the quasi-answer with the reference answer.

[0070] Specifically, after the initial annotation is completed (at the individual / team level), N sub-datasets (sub-training sets) are obtained through N-fold crossover, and N models are trained. These N models are used to predict the entire data set to obtain N prediction results, and these N prediction results are voted on to obtain reference answers. The reference answers are compared with the actual annotation results to obtain the annotation data quality assessment indicators including the error sources and distribution.

[0071] In the embodiment of the present invention, by using the error source and distribution as reference indicators of data quality, it is convenient for administrators and data annotators to quickly locate annotation problems and put forward corresponding evaluation suggestions.

[0072] In general, the present invention mainly includes the following parts: instant reminder of annotation errors for entity annotators, overall annotation quality analysis and error source analysis in the administrator interface. The overall process of the technical solution can be referred to Figure 2 , Figure 2 The main content shows the whole process of adding annotation quality analysis device to the existing named entity annotation platform.

[0073] The present invention mainly focuses on the immediate prompt of data annotation errors and the analysis of error sources. The technical solution design steps of the present invention are mainly as follows:

[0074] 1. Immediate feedback device for annotation errors: During the annotation process, when the annotator's annotation sentences reach the threshold K preset by the administrator, the training of the immediate feedback device for annotation errors is started. The training steps are mainly divided into making N-fold cross-validation data (the scale of the corpus is K, and N in the N-fold cross-validation is a pre-set parameter, the default is 5), training N named entity recognition models, saving N models, loading N models, starting the N model voting program, and using the voting program to generate model annotation results for unlabeled entries.

[0075] 2. Instant feedback and modification of errors: After the training of the device for instant feedback on annotation errors is started, every time the annotator completes the annotation of a corpus, the annotation result of the annotator is compared with the annotation result of the model generated by the voting program, and the type of annotation error is prompted for the fragments with inconsistent labels.

[0076] 3. Administrator's annotation quality analysis control device and error sources: When the administrator releases a task, the progress of the annotator's annotation task will be sent to the administrator in real time for the administrator to query the annotator's annotation progress. After the annotator has annotated a preset K sentences (after the annotator has triggered the annotation error instant feedback device), each time the annotator annotates a sentence, the suspected annotation errors and categories will be sent to the administrator for annotation quality control. If the suspected annotation errors of a certain annotator are significantly higher than those of other annotators, the administrator can provide guidance and corrections to the annotator in the annotation process.

[0077] 4. Annotation quality overview: After all annotators have completed the annotation, they will first vote on the annotation results of the entire corpus as a whole. The first voting result of the entire corpus is used as the quasi-answer, and N models are trained on the entire corpus N times. The N models are used again to predict the entire corpus, and the reference answer is obtained by another vote. According to the judgment criteria in step 2, the quasi-answer and the reference answer are compared to generate data quality indicators that include the source of errors. Among them, the "quasi-answer" corresponds to the "annotator's annotation result" in step 2, and the "reference answer" corresponds to the "model voting result" in step 2. The administrator backend displays the following Figure 3 The visual overview interface shown.

[0078] The embodiment of the present invention provides a named entity recognition data annotation quality assessment and control system, including:

[0079] A partitioning module is used to divide the data into N training sets;

[0080] A first training module, used to train N first named entity recognition models using the N sub-training sets respectively;

[0081] A first prediction and voting module, configured to use the N first named entity recognition models to predict a given annotated corpus, obtain N results, and vote for each entity prediction result in the N results;

[0082] The error prompt module is used to determine the error type based on the voting results and the annotation results of the annotator, and to provide suspected error prompts.

[0083] Optionally, in this embodiment, the partitioning module is specifically used to obtain N-molecule training sets by means of N-fold crossover.

[0084] Optionally, in this embodiment, the error prompt module specifically includes:

[0085] The first determination unit is used to determine the error type as suspected missing label if the labeling result of the annotator is empty and the model voting result is not empty;

[0086] The second determination unit is used to determine the error type as a suspected category error if the annotator's annotation position information is completely consistent with the voting position information, but the labels are not completely consistent;

[0087] The third determination unit is used to determine the error type as suspected other errors if it is not the above two suspected errors and there is a difference between the annotator's annotation result and the model voting result.

[0088] Optionally, in this embodiment, the system further includes:

[0089] The voting module is used to vote on the annotation results of the entire corpus as a whole after all annotators have completed the annotation. The first voting result of the entire corpus is used as the quasi-answer;

[0090] A second training module is used to train N second named entity recognition models by taking N folds of the entire corpus;

[0091] A second prediction and voting module, used to use the N second named entity recognition models to predict the entire corpus and vote to obtain a reference answer;

[0092] The comparison module is used to generate a data quality indicator including the source of error by comparing the quasi-answer with the reference answer.

[0093] An embodiment of the present invention provides a terminal device, including:

[0094] Processor; and

[0095] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0096] An embodiment of the present invention provides a non-transitory machine-readable storage medium on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0097] The reader should understand that in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the modules and units in the above-described system embodiment can refer to the corresponding processes in the aforementioned method embodiment and will not be repeated here.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0103] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for evaluating and controlling the quality of named entity recognition data annotation, characterized in that: include: Divide the data into N training sets; Using the N sub-training sets to train N first named entity recognition models respectively; Use N of the first named entity recognition models to predict a given annotated corpus to obtain N results; Voting for each entity prediction result among the N results; The error type is determined based on the voting results and the annotator's annotation results, and suspected error prompts are given, including: If the annotator's annotation result is empty, but the model's voting result is not empty, the error type is determined to be suspected missing label; and / or If the annotator's annotated location information completely matches the voting location information, but the labels do not completely match, the error type is determined to be a suspected category error; and / or If it is not one of the two suspected errors mentioned above, and there is a difference between the annotation result of the annotator and the voting result of the model, the error type is judged as suspected other errors.

2. The method according to claim 1, characterized in that The data is divided into N training sets, specifically including: The N-molecule training set is obtained through N-fold crossover.

3. The method according to any one of claims 1 or 2, characterized in that: Also includes: After all the annotators have completed the annotation, they will first vote on the annotation results of the entire corpus as a whole, and use the first voting result of the entire corpus as the quasi-answer; Taking N folds of the entire corpus to train N second named entity recognition models; Use N of the second named entity recognition models to predict the entire corpus and vote to obtain a reference answer; By comparing the quasi-answer with the reference answer, a data quality indicator including the source of error is generated.

4. A named entity recognition data annotation quality assessment and control system, characterized in that: include: A partitioning module is used to divide the data into N training sets; A first training module, used to train N first named entity recognition models using the N sub-training sets respectively; A first prediction and voting module, configured to use the N first named entity recognition models to predict a given annotated corpus, obtain N results, and vote for each entity prediction result in the N results; The error prompt module is used to determine the error type according to the voting results and the annotator's annotation results, and to give a suspected error prompt. The error prompt module specifically includes: The first determination unit is used to determine the error type as suspected missing label if the labeling result of the annotator is empty and the model voting result is not empty; The second determination unit is used to determine the error type as a suspected category error if the annotator's annotation position information is completely consistent with the voting position information, but the labels are not completely consistent; The third determination unit is used to determine the error type as suspected other errors if it is not the above two suspected errors and there is a difference between the annotator's annotation result and the model voting result.

5. The system according to claim 4, characterized in that The partitioning module is specifically used to obtain N-element training sets through N-fold crossover.

6. The system according to any one of claims 4 or 5, characterized in that: Also includes: The voting module is used to vote on the annotation results of the entire corpus as a whole after all annotators have completed the annotation. The first voting result of the entire corpus is used as the quasi-answer; A second training module is used to train N second named entity recognition models by taking N folds of the entire corpus; A second prediction and voting module, used to use the N second named entity recognition models to predict the entire corpus and vote to obtain a reference answer; The comparison module is used to generate a data quality indicator including the source of error by comparing the quasi-answer with the reference answer.

7. A terminal device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 3.

8. A non-transitory machine-readable storage medium having executable code stored thereon, characterized in that: When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 3.

Images