A supervised method for semi-automatic entity labeling

Abstract

A semi-automatic entity labeling supervision method and system, the method comprising the following steps: step 1, labeling a first set number of small samples; step 2, training a labeling model; if the quality of the labeling model meets the requirements, executing step 4, if the quality of the labeling model needs to be improved, executing step 3; step 3, iteratively executing automatic labeling, labeling quality supervision, proofreading, updating the training set and the training model until the number of iterations reaches a maximum number of iterations; if the F1 value of the iterative result labeling model is not less than a first threshold, executing step 4; if the F1 value of the iterative result labeling model is still less than the first threshold, adjusting the label and labeling specifications and returning to step 1; step 4, labeling the remaining unlabeled samples and supervising the labeling, including quality supervision of the labeling model and supervision of the labelers; step 5, analyzing whether the manual and labeled labeling results are consistent, and if not, sorting and analyzing the categories to which they belong.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and more specifically, relates to a semi-automatic entity labeling supervision method. Background Art

[0002] The marketing entity labeling model based on deep learning requires a large number of manually labeled samples. Directly labeling documents not only makes it impossible to visually see the labeling results, but also makes it impossible to count the number of labeled entities. Documents also need to be parsed during the model training process.

[0003] Prior art document 1 (CN111144099A) discloses a quality inspection method, apparatus, and device for entity tagging based on part of speech, which performs part of speech analysis on entity tagging data, analyzes the part of speech features of the entity tagging data, inserts the part of speech features into the data to be entity-tagged, and performs quality inspection on the data to be entity-tagged after the part of speech features are inserted. However, the shortcoming of prior art document 1 is that entity part of speech tagging is only applicable to general fields. For samples in the power field, a large amount of professional knowledge needs to be learned, that is, the part of speech of the power field entities needs to be labeled, and there are cases where the part of speech is the same but the labeled entity is inconsistent with the standard entity. Summary of the Invention

[0004] In order to solve the deficiencies in the prior art, the present invention aims to provide a semi-automatic entity labeling supervision method.

[0005] The present invention adopts the following technical solution. A first aspect of the present invention provides a semi-automatic entity labeling supervision method, comprising the following steps:

[0006] Step 1: At the beginning of the entity labeling work, a first set number of small samples are manually labeled;

[0007] Step 2: Use the annotation results of step 1 as the training set to train the annotation model; monitor the quality of the annotation model. If the quality of the annotation model meets the requirements, proceed to step 4; if the quality of the annotation model needs to be improved, proceed to step 3.

[0008] Step 3: Iterate the automatic labeling, labeling quality supervision, proofreading, updating the training set and training model until the maximum number of iterations is reached; if the F1 value of the labeling model of the iterative result is not less than the first threshold, then execute step 4; if the F1 value of the labeling model of the iterative result is still less than the first threshold, adjust the label and labeling specifications and return to step 1;

[0009] Step 4: Label the remaining unlabeled samples and supervise the labeling, including quality supervision of the labeling model and supervision of the labelers;

[0010] Step 5: Analyze whether the manual and annotated results are consistent. If not, sort out and analyze the categories to which they belong.

[0011] Preferably, in step 1, the number of small samples is 1000 documents.

[0012] Preferably, in step 2, the annotation results of step 1 are used as the training set to train the annotation model; after the training, the quality of the annotation model is supervised; if the F1 value of the annotation model is not less than the first threshold, the text can be automatically annotated directly and step 4 is executed; if the F1 value is less than the first threshold and greater than the second threshold, step 3 is executed; if the F1 value is not greater than the second threshold, it is preliminarily judged that the number of small samples in step 1 is small, and it is necessary to increase the samples and retrain, and return to step 1 to increase the number of annotated samples.

[0013] Preferably, in step 2, the first threshold is 80% and the second threshold is 60%.

[0014] Preferably, step 3 specifically includes:

[0015] Step 3.1, using the annotation model to automatically annotate a second set number of samples to form an annotation result;

[0016] Step 3.2: Perform annotation quality supervision and proofreading on the annotation results of step 3.1 to obtain proofreading results;

[0017] Step 3.3: Determine whether the maximum number of iterations has been reached. If not, proceed to step 3.4. If it has been reached, terminate the iteration and proceed to step 3.6.

[0018] Step 3.4: Update the training set by combining the small sample annotation results in step 1 and the proofreading results obtained in step 3.2 as the new training set;

[0019] Step 3.5: Use the updated training set to train the annotation model and return to step 3.1 to start a new iteration;

[0020] In step 3.6, the F1 value of the current annotation model is compared with the first threshold. If the F1 value of the current annotation model is not less than the first threshold, proceed to step 4. If the F1 value of the current annotation model is still less than the first threshold, adjust the label and specification and return to step 1.

[0021] Preferably, in step 3.2, proofreading means correcting, in each iteration, labeling errors in the second set number of samples automatically labeled using the labeling model in the current iteration to form a labeling result.

[0022] Preferably, in step 4, supervision of the annotation model algorithm includes: monitoring the model by the F1 value of the training test set or manually proofreading the automatically annotated text.

[0023] Preferably, in step 5, the inconsistency between manual and annotated labeling results is analyzed, and the categories to which the analysis belongs are sorted out, including: model learning errors, labeling errors by the labelers, and problems not in compliance with the labeling specifications.

[0024] The second aspect of the present invention provides a semi-automatic entity annotation supervision system, which runs a semi-automatic entity annotation supervision method, including: a manual annotation module, a annotation model module, a supervision module, a parameter setting module, a proofreading module and an output module; wherein,

[0025] Manual labeling module, used to manually label samples;

[0026] Annotation model module, used to automatically annotate samples;

[0027] Supervision module, used to supervise the quality of the annotation model and the annotation personnel;

[0028] Parameter setting module, used to set training parameters;

[0029] Proofreading module, used to proofread the annotation results;

[0030] Output module, used to output supervision results.

[0031] Preferably, the semi-automatic entity labeling supervision system is a subsystem of the power marketing system.

[0032] The beneficial effect of the present invention is that, compared with the existing technology, it can effectively supervise the annotation quality of the annotation personnel, solve the problem that the sampling detection method is difficult to accurately detect entities with annotation errors, improve the supervision efficiency of the supervisory management personnel, and reduce management costs.

[0033] In particular, compared with the existing technology in the general field represented by Comparative Document 1, the present invention can verify and supervise entity labeling through small-scale sample learning in vertical fields, especially the power grid field. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the overall process of the semi-automatic labeling technology for sensitive samples in the marketing field based on human-machine collaboration in the present invention;

[0035] Figure 2 A flowchart of a semi-automatic entity labeling supervision method provided by the present invention;

[0036] Figure 3 A schematic diagram of the bidirectional supervision provided by the present invention;

[0037] Figure 4 This is an example diagram of the present invention that is not in the marking standard. DETAILED DESCRIPTION

[0038] The present application will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present application.

[0039] like Figure 1 、 2 As shown, the present invention provides a semi-automatic entity labeling supervision method, comprising the following steps:

[0040] Step 1: When the entity labeling work begins, a first set number of small samples are labeled.

[0041] A preferred but non-limiting implementation is that the first set number of small samples is about 1,000 documents. Too small a number will affect the model training effect, and too large a number will require a lot of time and energy from managers.

[0042] A preferred, but non-limiting, implementation involves both annotators and management personnel jointly participating in the annotation of a set number of small samples. Regardless of the quality of annotators' annotations, their performance will not be included in the performance evaluation system. Management personnel should not only participate in the annotation process but also review the quality of the annotations, collect any issues raised by the annotators, and discuss them collectively after the small sample is completed. Annotators should also be actively encouraged to raise new issues with their annotations, which will be incorporated into the performance evaluation system and facilitate their participation in the development of annotation standards.

[0043] Step 2: Use the annotation results of step 1 as the training set to train the annotation model; after the training, supervise the quality of the annotation model; specifically, if the F1 value of the annotation model is not less than the first threshold, it means that the initial effect of the annotation model is relatively good, and the text can be automatically annotated directly, and execute step 4; if the F1 value is less than the first threshold and greater than the second threshold, it means that the model effect is average, but it can participate in automatic annotation, and execute step 3; if the F1 value is not greater than the second threshold, it can be preliminarily judged that the number of samples is small, and it is necessary to increase the samples for retraining, and return to step 1 to increase the number of annotated samples.

[0044] In a preferred but non-limiting embodiment, the first threshold is 80% and the second threshold is 60%.

[0045] The F1 value is expressed as follows:

[0046]

[0047] Where:

[0048] p represents the accuracy of the annotation model,

[0049] r represents the recall rate of the annotation model;

[0050] The values ​​of both the accuracy p and the recall r are between 0 and 1. The closer the value is to 1, the higher the accuracy or recall. The accuracy and recall may sometimes conflict with each other. In this case, we need to consider their weighted harmonic mean, that is, the F score. The most commonly used one is the F1 value. When the F1 value is high, it means that the experimental method is more effective.

[0051] Step 3, iteratively perform automatic labeling, labeling quality supervision, proofreading, updating the training set and training model until the number of iterations reaches the maximum number of iterations; if the F1 value of the labeling model of the iterative result is not less than the first threshold, then execute step 4; if the F1 value of the labeling model of the iterative result is still less than the first threshold, adjust the label and labeling specifications and return to step 1. In a further preferred embodiment of the present invention, step 3 specifically includes:

[0052] Step 3.1, using the annotation model to automatically annotate a second set number of samples to form an annotation result;

[0053] Step 3.2: Implement annotation quality supervision and proofreading. It is worth noting that proofreading refers to correcting the annotation errors in the second set number of samples automatically annotated by the annotation model in each iteration to form the annotation results.

[0054] Step 3.3: Determine whether the maximum number of iterations has been reached. If not, proceed to step 3.4. If it has been reached, terminate the iteration and proceed to step 3.6.

[0055] Step 3.4: Update the training set by combining the small sample annotation results in step 1 and the proofreading results obtained in step 3.2 as the new training set;

[0056] Step 3.5: Use the updated training set to train the annotation model and return to step 3.1 to start a new iteration;

[0057] In step 3.6, compare the F1 value of the current annotation model with the first threshold. If the F1 value of the current annotation model is not less than the first threshold, it means that the current annotation model is relatively good, and the text can be automatically annotated directly, and step 4 can be continued. If the F1 value of the current annotation model is still less than the first threshold, it means that there are problems with the task entity label definition and / or annotation specifications, there is a problem of scattered features of the same label, or there are problems with similar features of different labels, or the entity features are complex and the model cannot learn the features of a certain label. It is necessary to adjust the entity labels and annotation specifications in time. After adjusting the labels and specifications, return to step 1.

[0058] like Figure 3 As shown, a preferred but non-limiting implementation method is to use manual proofreading, that is, the annotator can see the machine annotation results, correct the annotation errors in time, upload the machine annotation results to the annotation system, and the annotator manually modify the annotation results.

[0059] Another preferred but non-restrictive implementation is to use a mutual supervision mechanism for proofreading, that is, the annotators do not know the machine annotation results in advance. The annotators re-annotate the samples annotated by the machine, and the management personnel judge the annotation quality by comparing the differences between the annotation results of the machine and the annotators. This method can not only test the quality of automatic annotation, but also test the annotation quality of the annotators, playing a role of two-way supervision.

[0060] In a preferred but non-limiting embodiment, the second set number of samples is 10,000 samples.

[0061] In addition, for problems where the proportion of labeled samples is small, samples of the label can be extracted separately for training, and manual correction can be made by comprehensively referring to the labeling results of different models.

[0062] Step 4: Label the remaining unlabeled samples and supervise the labeling, including quality supervision of the labeling model and supervision of the labelers. Specifically, it includes:

[0063] A preferred but non-restrictive implementation of the supervision of the labeling model algorithm is to perform supervision based on the training results, that is, the F1 value of the training test set.

[0064] Another preferred but non-limiting embodiment is to supervise the model by manually proofreading the automatically annotated text. The results of manual proofreading are mainly categorized into four types: complete consistency between the manual and machine annotations, manual annotations but machine omissions, machine annotations but manual omissions, and inconsistencies between the manual and machine annotations. The latter three are considered annotation errors and require analysis.

[0065] Regarding the quality supervision of annotation personnel, a preferred but non-restrictive implementation method is that the quality supervision of annotation personnel mainly includes methods such as full detection, sampling detection and model-assisted detection.

[0066] Full-scale testing can comprehensively identify problems, but since the annotation quality of each labeler needs to be tested, the workload is the sum of the workload of all labelers. When the number of samples is large, managers do not have the time and energy to supervise.

[0067] Sampling testing generally randomly selects a portion of the labeled samples for proofreading. This method can reduce the workload of managers, but most problems cannot be discovered by managers.

[0068] In a further preferred embodiment, model-assisted detection can be used to monitor both the learning effectiveness of the model and the quality of the annotations by the annotators. Inconsistencies between the annotations by the annotators and the machine model can be analyzed by management and categorized into three categories: model learning errors, annotator errors, and non-compliance with annotation standards.

[0069] Regarding the supervision of the number of markings made by the marking personnel, the quantity supervision can summarize the marking situation of the marking personnel every day. The marking situation of the marking personnel every day can be used as the basis for performance appraisal, and performance rewards can be implemented for the marking personnel with higher marking efficiency. If there is no marking on a certain day, the marking personnel need to be asked the reason, and punishment measures will be implemented for those who fail to participate in marking without reason.

[0070] Step 5: Analyze whether the manual and annotated results are consistent. If not, analyze the categories: 1. Model learning errors, 2. Annotator errors, and 3. Non-standard labeling issues. Model recognition issues arising during the supervision process are handled by algorithm engineers. Annotator issues are promptly fed back to the annotators and included in the assessment system. Issues outside the labeling standards require revision and improvement of the labeling standards.

[0071] For model learning errors, we can leave it to algorithm engineers to solve.

[0072] For labeling errors made by labelers, for example, labeling "Room 702, No. 56" as "Room 702", with one "Room" missing, this can be recorded in the performance appraisal and promptly fed back to the labeler for rectification.

[0073] like Figure 4 As shown, because the "PMS_Jielong Line" entity features differ from common address features, some instances were manually annotated despite not being machine-annotated. This is not covered by the annotation specifications, so they need to be revised and improved promptly. Annotators are also encouraged to raise issues with annotations, and performance rewards will be given as appropriate for those who raise issues not covered by the annotation specifications.

[0074] Figure 4This is a screenshot of address entity annotation in the power sector. The annotation consists of two parts: the sentence to be annotated and the annotation label. The upper part contains the entity label defined based on the business or scenario, while the lower part requires manual entity labeling through mouse scrolling and selection. The annotation interface provides specific annotation instructions and shortcut keys for labeling, facilitating efficient annotation. The current model can only recognize general address entities, but annotators' annotations of power sector entities often result in inconsistencies. Management personnel need to refine annotation standards and retrain the model.

[0075] For example Figure 4 In the example, the sentence to be labeled is: "3) The lines where this household is located are PMS_Jielong Line 122 and PMS_Huangjinba Line 146." Use the mouse to select PMS_Jielong Line, then click the "Block Address Class - Electricity Related" label or press Ctrl+j to label the "PMS_Jielong Line" text as "Block Address Class - Electricity Related". Similarly, use the same operation to label "122", "PMS_Huangjinba Line 1", "146", etc. with different entity labels based on human understanding.

[0076] Embodiment 2 of the present invention provides a semi-automatic entity annotation supervision system, which runs the semi-automatic entity annotation supervision method, including: a manual annotation module, a annotation model module, a supervision module, a parameter setting module, a proofreading module and an output module; wherein,

[0077] Manual labeling module, used to manually label samples;

[0078] Annotation model module, used to automatically annotate samples;

[0079] Supervision module, used to supervise the quality of the annotation model and the annotation personnel;

[0080] Parameter setting module, used to set training parameters;

[0081] Proofreading module, used to proofread the annotation results;

[0082] Output module, used to output supervision results.

[0083] Preferably, the semi-automatic entity labeling supervision system is a subsystem of the power marketing system.

[0084] The beneficial effect of the present invention is that, compared with the existing technology, it can effectively supervise the annotation quality of the annotation personnel, solve the problem that the sampling detection method is difficult to accurately detect entities with annotation errors, improve the supervision efficiency of the supervisory management personnel, and reduce management costs.

[0085] In particular, compared with the existing technology in the general field represented by Comparative Document 1, the present invention can verify and supervise entity labeling through small-scale sample learning in vertical fields, especially the power grid field.

[0086] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation plans of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, and is not a limitation on the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A semi-automatic entity labeling supervision method for power marketing system, characterized by: The following steps are involved: Step 1: The annotation is divided into two parts: the sentences to be annotated and the annotation labels. When the entity annotation work starts, the first set number of small samples are manually annotated; Step 2: Use the annotation results of step 1 as the training set to train the annotation model. After the training, monitor the quality of the annotation model. If the F1 value of the annotation model is not less than the first threshold, the text is automatically annotated directly and step 4 is executed. If the F1 value is less than the first threshold and greater than the second threshold, proceed to step 3; if the F1 value is not greater than the second threshold, it is preliminarily determined that the number of small samples in step 1 is small and it is necessary to increase the samples and retrain, and return to step 1 to increase the number of labeled samples; Step 3: Iterate the automatic labeling, labeling quality supervision, proofreading, updating the training set, and training the model until the number of iterations reaches the maximum number of iterations, including: Step 3.1, using the annotation model to automatically annotate a second set number of samples to form an annotation result; Step 3.2: Perform annotation quality supervision and proofreading on the annotation results of step 3.1 to obtain proofreading results; Step 3.3: Determine whether the maximum number of iterations has been reached. If not, proceed to step 3.

4. If it has been reached, terminate the iteration and proceed to step 3.

6. Step 3.4: Update the training set by combining the small sample annotation results in step 1 and the proofreading results obtained in step 3.2 as the new training set; Step 3.5: Use the updated training set to train the annotation model and return to step 3.1 to start a new iteration; Step 3.6: Compare the F1 value of the current annotation model with the first threshold. If the F1 value of the current annotation model is not less than the first threshold, continue to step 4. If the F1 value of the current annotation model is still less than the first threshold, adjust the label and specification and return to step 1. Step 4: Label the remaining unlabeled samples and supervise the labeling, including quality supervision of the labeling model and supervision of the labelers. Supervision of the labeling model algorithm includes: F1 value of the training test set or manual proofreading of the automatically labeled text to supervise the model. Step 5: Analyze whether the manual and annotated labeling results are consistent. If not, sort out and analyze the categories to which they belong. Analyze whether the manual and annotated labeling results are inconsistent, and sort out and analyze the categories to which they belong, including: model learning errors, labeler errors, and problems that are not in accordance with the labeling specifications.

2. A semi-automatic entity labeling supervision method according to claim 1, characterized in that: In step 1, the number of small samples is 1000 documents.

3. The semi-automatic entity labeling supervision method according to claim 1, characterized in that: In step 2, the first threshold is 80% and the second threshold is 60%.

4. The semi-automatic entity labeling supervision method according to claim 1, characterized in that: In step 3.2, proofreading refers to correcting the labeling errors in the second set number of samples automatically labeled by the labeling model in this iteration in each iteration to form a labeling result.

5. A semi-automatic entity labeling supervision system, running a semi-automatic entity labeling supervision method according to any one of claims 1 to 4, comprising: Manual annotation module, annotation model module, supervision module, parameter setting module, proofreading module and output module, characterized by: Manual labeling module, used to manually label samples; Annotation model module, used to automatically annotate samples; Supervision module, used to supervise the quality of the annotation model and the annotation personnel; Parameter setting module, used to set training parameters; Proofreading module, used to proofread the annotation results; Output module, used to output supervision results.

6. A semi-automatic entity labeling supervision system according to claim 5, characterized in that: The semi-automatic entity labeling supervision system is a subsystem of the power marketing system.

Images