Annotation device and method

Abstract

To provide technology for realizing efficient labeling regarding a machine learning model for executing a task constituted of a plurality of subtasks.SOLUTION: An annotation device 1 for realizing labeling of machine learning comprises: a machine learning module 103 with a teacher having a plurality of subtask modules 104 for performing learning to each of a plurality of subtasks constituting a task of the machine learning by using data with a label to estimate an estimation label to data without a label; a subtask mutual influence degree calculation module 107 which calculates an influence degree showing degrees of influence to other subtask modules 104 by learning by the subtask modules 104; and a label correction module 110 which corrects the estimation label according to a confirmation result to the estimation label presented according to the total priority score based on the influence degree.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for annotation, i.e., labeling, in machine learning. [Background technology]

[0002] Currently, in supervised machine learning, so-called annotators create (label) training data using a user interface. For this reason, manually labeling all data using a user interface by an annotator requires a huge amount of work. For this reason, a framework called active learning is sometimes used to reduce this work.

[0003] In active learning, an annotator first manually labels a portion of the target data. Next, the labeled data is used as training data to train a machine learning model. The trained machine learning model also estimates labels for unlabeled data and presents them to the annotator via a user interface. The annotator then checks the presented results and makes any corrections.

[0004] The machine learning model is then retrained using the confirmed labeled data, and the process of inferring data from unlabeled data, followed by confirmation and correction by annotators, is repeated. This reduces the amount of work required compared to labeling everything manually.

[0005] Active learning is also being considered from the perspective of what kind of data should be presented to the annotator when presenting results. For example, one method for this purpose is to quantify the confidence level of the estimated results and prioritize the presentation of results with low confidence levels, thereby increasing the rate of accuracy improvement during retraining of the machine learning model. Specifically, one method uses the entropy of the estimated probability as the confidence level. Another method is to prioritize the presentation of data with a large number of similar data among unlabeled data. In this case, the efficiency of retraining can be improved by calculating the similarity between data and presenting data with high similarity first.

[0006] As described above, technologies for reducing labeling man-hours by combining a machine learning model and user feedback have been proposed in, for example, Patent Document 1 and Patent Document 2. Patent Document 1 describes an information processing device that prevents a decrease in the accuracy of machine learning due to incorrect tagging. Patent Document 2 describes extracting synonyms / related words with good efficiency and accuracy. Patent Documents 1 and 2 are described below.

[0007] Some machine learning tasks involve learning and inference for multiple subtasks, and the desired output is obtained by combining the inference results of each subtask. For example, in the task of event extraction in natural language processing, named entities are extracted from text using named entity extraction, and then relationships between pairs of named entities in the text are inferred using relationship extraction. Furthermore, in machine learning models that solve tasks consisting of multiple subtasks, the models for each subtask may share parts. For example, in a neural network that simultaneously infers named entity extraction and relationship extraction, several layers from the input are shared, with only the final layer being split between named entity extraction and relationship extraction. This model configuration allows learning in one subtask to contribute to learning and inference in other subtasks, mutually improving accuracy.

[0008] In annotation systems that employ an active learning framework, the above-mentioned Patent Documents 1 and 2 have been proposed as methods for improving re-learning efficiency for a single task by determining the presentation priority according to a quantitative priority score of the estimation results for each piece of data. First, Patent Document 1 assumes that multiple labels (tags) can be attached to one piece of data, and determines whether the estimation result is abnormal based on the co-occurrence probability between tags, and notifies the user.

[0009] Furthermore, in Patent Document 2, structured data candidates relating to relationships are generated for document data, and validity is determined according to the likelihood of the generated structured data candidates. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Publication No. 2018-185601 [Patent Document 2] Japanese Patent Publication No. 2020-095452 Summary of the Invention [Problem to be solved by the invention]

[0011] As described above, Patent Documents 1 and 2 describe processes for making judgments using values ​​related to the co-occurrence probability and likelihood of labels in a single task. However, in a machine learning model in which multiple subtasks share part of the model, it has been difficult to determine presentation priority using the same method as in the case of a single task. Here, when determining presentation priority by simply adding up the priority scores for each subtask, it is possible that only one subtask has not been properly trained, resulting in low accuracy. In this case, it is not possible to focus on data that will improve the accuracy of that subtask. This makes efficient re-learning difficult. Furthermore, since the mutual contribution of the learning of one subtask to other subtasks is not taken into consideration, there is a possibility that sufficient improvement in accuracy during re-learning cannot be expected.

[0012] Therefore, an object of the present invention is to provide a technology that realizes efficient labeling for a machine learning model that executes a task consisting of multiple subtasks. [Means for solving the problem]

[0013] In machine learning labeling, a priority for confirmation of an inferred result, e.g., an inferred label, is determined based on an influence level indicating the degree of influence of learning of a subtask included in multiple subtasks on a subtask module that learns other subtasks. Note that this priority includes a priority for presentation to an annotator.

[0014] More specifically, the present invention provides an annotation device for realizing machine learning labeling, the annotation device including: a supervised machine learning module having a plurality of subtask modules that learn using labeled data and estimate estimated labels for unlabeled data for each of a plurality of subtasks constituting the machine learning task; a subtask mutual influence calculation module that calculates an influence indicating the degree of influence of learning in the subtask modules on other subtask modules; and a label correction module that corrects the estimated labels in accordance with a confirmation result for the estimated labels presented in accordance with an overall priority score based on the influence. The present invention also includes an annotation method using this annotation device. [Effects of the Invention]

[0015] According to the present invention, efficient labeling can be achieved for machine learning using multiple subtasks. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating a functional configuration of an annotation device according to an embodiment of the present invention. [Figure 2] FIG. 2 illustrates unlabeled data in one embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating labeled data in one embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating the processing of a supervised machine learning module in one embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating subtask accuracy information in one embodiment of the present invention. [Figure 6] FIG. 10 illustrates an example output of a subtask priority score calculation module in one embodiment of the present invention. [Figure 7] FIG. 10 illustrates an example output of a subtask priority score calculation module in one embodiment of the present invention. [Figure 8] FIG. 1 is an overall processing flow diagram according to an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing priority scores for one piece of data in one embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing a total priority score for each piece of data in one embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing a confirmation screen for all data to be confirmed in this embodiment. [Figure 12] FIG. 10 is a diagram showing a details confirmation screen in the present embodiment. [Figure 13] 1 is a configuration diagram of an annotation system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] An embodiment of the present invention will be described below. In this embodiment, an event extraction task from a document will be described as an example. That is, the event extraction task is used as a task in the supervised machine learning module 103 described below. Here, the supervised machine learning module 103 has three subtask modules. First, the first subtask module extracts named entities from the document. Furthermore, the second subtask module extracts relationships from the document. Furthermore, the third subtask module performs name matching in the document. That is, in this embodiment, the subtasks executed are named entity extraction, relationship extraction, and name matching.

[0018] To execute these subtasks, unlabeled data 101 and labeled data 102 are used. The unlabeled data 101 and labeled data 102 are failure response history data. In this embodiment, a supervised machine learning module 103 extracts event information on the actions taken from the data by combining the part name and the action name.

[0019] In this embodiment, in order to execute these event extraction tasks and their subtasks, estimated labels for the unlabeled data 101 are estimated. Then, a total priority score indicating the priority in checking these estimated labels is calculated. As a result, the annotator can check and correct the estimated labels in a more appropriate order. Details of this will be explained below with reference to the drawings.

[0020] Fig. 1 is a diagram showing the functional configuration of an annotation device 1 in this embodiment. In Fig. 1, the annotation device 1 has a supervised machine learning module 103, a subtask accuracy calculation module 106, a subtask mutual influence calculation module 107, a subtask priority score calculation module 108, an overall priority score calculation module 109, a label correction module 110, and an overall priority score recalculation module 111. Note that each of these components (modules) and a subtask module 104 (described later) are components that execute processing, and can also be expressed as a number of modules. In this embodiment, unlabeled data 101 and labeled data 102 are used.

[0021] First, the unlabeled data 101 is data to which no labels have been assigned. In this embodiment, the unlabeled data 101 is text data describing a failure response history in natural language. However, in the present invention, the target data may be in the form of image data, table data, time-series data, or the like other than text data. Details of the unlabeled data 101 will be described later with reference to FIG. 2.

[0022] The labeled data 102 is data to which labels of each subtask and task of the supervised machine learning module are assigned, the details of which will be described later with reference to FIG.

[0023] Furthermore, supervised machine learning module 103 performs learning, verification, and estimation for each subtask module 104. That is, labeled data 102 is used as input to perform learning and verification for each subtask module 104. Furthermore, supervised machine learning module 103 uses unlabeled data 101 as input to perform estimation for each subtask module 104. Note that this estimation refers to estimating an estimated label, which is a label for unlabeled data 101.

[0024] For this purpose, supervised machine learning module 103 has multiple subtask modules 104 and shared model 105. Here, shared model 105 performs common processing within subtask module 104. Note that shared model 105 may be realized as a processing entity, or may be realized as information indicating a model.

[0025] Here, each subtask module 104 may perform not only machine learning but also preprocessing of input data and postprocessing of the estimation results. Details of the supervised machine learning module 103, subtask module 104, and shared model 105 will be described later with reference to FIG. 4.

[0026] Furthermore, subtask accuracy calculation module 106 calculates the estimation accuracy of the estimated labels estimated by supervised machine learning module 103. To this end, subtask accuracy calculation module 106 calculates the estimation accuracy of the estimated labels in the subtasks of each subtask module 104 from the estimated labels, which are the estimation results output from supervised machine learning module 103. Details of this will be described later with reference to FIG. 5.

[0027] Furthermore, subtask mutual influence calculation module 107 calculates an influence indicating the degree of influence of learning about a subtask on other subtask modules 104. To do this, subtask mutual influence calculation module 107 uses estimated labels output from supervised machine learning module 103. This will be described in detail later with reference to FIG. 6.

[0028] Furthermore, the subtask priority score calculation module 108 calculates a subtask priority score related to the confirmation order for each data unit of data handled by each subtask module 104. Here, a data unit refers to each unlabeled data 101 itself or each element of the unlabeled data 101. Any calculation method may be used as long as it outputs a scalar value for the data or data elements. For example, in the case of a multi-class classification problem, the entropy of the output probability is calculated as the subtask priority score, so that the priority score increases in descending order of the confidence of the estimation. Alternatively, a similarity measure such as cosine similarity between the target data and other unlabeled data is calculated as the subtask priority score, so that data similar to other data will have a higher priority score.

[0029] The overall priority score calculation module 109 also calculates an overall priority score for each subtask for each unlabeled data 101 using at least one of the estimation accuracy, influence, and subtask priority score. More preferably, the overall priority score is calculated based on at least the estimation accuracy. Alternatively, influence may be used. For example, the overall priority score can be calculated by multiplying each subtask priority score by the weight of each subtask, which is calculated as a function of the estimation accuracy and influence. The overall priority score indicates the priority in confirming the estimated label, and the estimated label is presented, for example, according to the overall priority score. The presentation may include presentation in an order according to the overall priority score or presentation in a format (size, layout) according to the overall priority score. Details of the calculation of the overall priority score will be described with reference to FIG. 8.

[0030] Furthermore, the label correction module 110 presents the estimated labels according to the overall priority score calculated by the overall priority score calculation module 109. As a result, the annotator can confirm the corresponding estimated labels. Then, when the label correction module 110 receives a correction instruction from the annotator according to the confirmation result, it corrects the estimated labels and adds them to the labeled data. For this purpose, the annotation device 1 is provided with a user interface (GUI) function, allowing the annotator to check and correct the labels. For example, the annotation device 1 may be provided with an input device and a display device itself, or may be connected to these.

[0031] As described above, in this embodiment, the annotator confirms and corrects the estimated labels, but the label correction module 110 may be configured to perform confirmation and correction in cooperation with another system. The GUI in this embodiment will be described later with reference to FIG.

[0032] Furthermore, the overall priority score recalculation module 111 recalculates the overall priority score in response to confirmation and correction of the estimated label described above. For example, the accuracy calculated by the subtask accuracy calculation module 106 is updated using the accuracy of the data already confirmed by the label correction module 110. Then, the influence calculated by the subtask mutual influence calculation module 107 and the overall priority score calculated by the overall priority score calculation module 109 are recalculated in response to the update results. The overall priority score may also be recalculated by manually adjusting parameters related to the importance of the accuracy and influence that appear in the calculation formula for the overall priority score using a GUI. The function of the overall priority score recalculation module 111 may be provided in the overall priority score calculation module 109. This concludes the explanation of FIG. 1.

[0033] Next, each piece of data handled in this embodiment will be described. Fig. 2 is a diagram showing unlabeled data 101 in this embodiment. The unlabeled data 101 in this embodiment is text data of a failure response history. Therefore, as shown in Fig. 2, the unlabeled data 101 is composed of a data ID 201 that uniquely determines each piece of data and corresponding data content 202.

[0034] Here, the data ID 201 identifies the corresponding unlabeled data 101. Here, the data ID 201 shown in Fig. 2 takes an integer value, but it may take any form, such as a numeric value or a character string, as long as it does not overlap with multiple data. Furthermore, the data ID 201 must not overlap with the data ID of labeled data 102, which will be described later.

[0035] 2 is composed of only one piece of data, it may be composed of multiple elements, such as an error code that has occurred, the contents of the inspection results, and the estimated cause.

[0036] Next, FIG. 3 is a diagram showing labeled data 102 in this embodiment. As shown in FIG. 3, labeled data 102 is composed of a data ID 301, data content 302, and labels 303, 304, and 305 corresponding to each subtask. The data ID 301 and data content 302 are the same items as those in unlabeled data 101. The labels 303, 304, and 305 indicate three subtasks: named entity extraction, relationship extraction, and name identification. Here, each label 303, 304, and 305 may have any structure as long as it corresponds to the data content and the label is clearly identifiable. In FIG. 3, named entity extraction label 303 stores, for each data content, the named entity extraction label ID (e.g., "3-0"), the corresponding character string, the position of the character string, and the named entity label.

[0037] Here, the relationship extraction label stores a pair of named entity extraction label IDs and the corresponding relationship labels. The name matching label 305 lists the named entity extraction label ID and the corresponding word after name matching. For example, for the data with data ID=3, the expression "part D" in data content 302 is a named entity for "part." There is a relationship called "part treatment relationship" between the two named entities, "part D" and "exchange." When name matching is performed, the expression "part D" in data content 302 is matched to the expression "AAA."

[0038] Next, the supervised machine learning module 103 including the shared model 105 will be described. FIG. 4 is a diagram illustrating the processing of the supervised machine learning module 103 in this embodiment. As described above, the subtasks in this embodiment are named entity extraction, relationship extraction, and name matching. To this end, the supervised machine learning module 103 is composed of a shared model 105, a named entity extraction model 401, a relationship extraction model 402, and a name matching model 403. Each of these is assumed to be in the form of a neural network, but any method can be used as long as it can learn the label of each subtask from the labeled data 102 and estimate the label of each subtask from the unlabeled data 101. Before each subtask is trained by the models 401, 402, and 403, the shared model 105 performs common processing, and the parameters of the shared model 105 are also changed through training. By training each subtask in parallel, the training of each subtask can contribute to improving the accuracy of the other subtasks. However, each of these models may be stored as template data. FIG. 4 will now be described.

[0039] First, text data 404 in the figure indicates that text data to be the target of the task is input. In this example, two pieces of text data are prepared: "Error..." and "Inspection...". These pieces of data are then divided into morphemes ("Error", "Part A", "to", "Replace", ".") using morphological analysis. For evaluation using training and validation data of the supervised machine learning module 103, data extracted from the labeled data 102 is used as this text data. For estimation by the supervised machine learning module 103, data extracted from the unlabeled data 101 is used.

[0040] Next, in vector representation 405, each morpheme of the text data is converted into a vector representation. An example of a conversion method at this time is word2vec. In addition, in shared model 105, feature extraction is performed on the vector representation 405 of each morpheme using a neural network or the like, and a feature vector 406 is output. An example of shared model 105 is a method such as BERT.

[0041] Furthermore, the named entity extraction model 401 receives a feature vector 406 as input and estimates the named entity label 407 of the target morpheme, i.e., whether it corresponds to a part name, a procedure name, or other. When a part name that spans multiple morphemes exists, estimation may be performed using a BIO notation or the like instead of a label name such as a part name.

[0042] Furthermore, for a pair of two morphemes in the same text data whose named entity label 407 is estimated to be a named entity such as a part name and a procedure name, the relationship extraction model 402 inputs a pair of feature vectors 406 and estimates the relationship label 408. That is, the relationship extraction model 402 estimates whether there is a part-procedure relationship or no relationship. Here, in addition to the two morphemes, the feature vector 406 of the morpheme between them, or a feature vector related to the context of the entire text that can be calculated within the shared model, may also be input.

[0043] Furthermore, the name identification model 403 estimates a name identification label 409 indicating whether or not the two morphemes should be identified for a pair of morphemes that are estimated to have the same named entity label 407 in the two text data.

[0044] During estimation by the supervised machine learning module 103, the data content 202 of the unlabeled data 101 is used as input data, and named entity labels 407, relationship labels 408, and name identification labels 409 are output. During learning, the data content 302 of the labeled data 102 is used as input, and learning is performed so that the output labels 407, 408, and 409 match the subtask labels 303, 304, and 305, respectively. During this learning, the parameters of the models 105, 401, 402, and 403 are changed using a method such as backpropagation, so that the labels 303, 304, and 305 can be output for the data content 302 of the labeled data. During evaluation using the labeled data 102, the degree to which the output labels 407, 408, and 409 match the labels 303, 304, and 305 in the labeled data 102 is quantitatively calculated.

[0045] Next, the processing details of the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107 will be explained. Note that in this embodiment, the influence is calculated from the difference in accuracy between when all subtasks are learned and when the subtasks are learned excluding those subtasks, but any calculation method may be used as long as it can output the influence quantitatively. Note that "during learning" refers to when learning occurs or at the time of learning.

[0046] Here, subtask accuracy information 204 is created by the processing of the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107. Fig. 5 is a diagram showing the subtask accuracy information 204 in this embodiment. Below, the processing of the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107 will be described with reference to Fig. 5.

[0047] First, the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107 divide the labeled data 102 into training data and validation data. Then, the subtask accuracy calculation module 106 performs training on the supervised machine learning module 103 using the training data from the labeled data 102, and evaluates (i.e., calculates) the estimation accuracy when estimation is performed using the validation data. The subtask accuracy calculation module 106 then displays this estimation accuracy (evaluation value) in the column 501 for all subtask training and the row 503 for each subtask accuracy in FIG. 5 . For example, the accuracy a_NER of named entity extraction during all subtask training is 0.85. This accuracy a is calculated by the subtask accuracy calculation module 106. The accuracies a_NER, a_RE, and a_CR of each subtask are used in the overall priority score calculation module 109, which will be described later. NER, RE, and CR represent named entity extraction, relationship extraction, and name matching, respectively. Then, the subtask accuracy calculation module 106 records the calculated items in the subtask accuracy information 204 .

[0048] Furthermore, for the three subtasks this time, the subtask accuracy calculation module 106 also calculates the estimation accuracy when only the remaining two subtasks are learned without learning each of them, in the same way as when all subtasks are learned. Then, when the subtask accuracy calculation module 106 performs the calculation, the results can be recorded in the column 502 for learning when each subtask is excluded and in the row 503 for the accuracy of each subtask.

[0049] Furthermore, the difference in accuracy of subtasks other than the excluded subtasks during learning with each subtask excluded and during all-subtask learning is a value that reflects the influence of the excluded subtask on other subtasks. In other words, this indicates the influence calculated by the subtask mutual influence calculation module 107. Here, the average value of the difference (accuracy difference) between column 502 during learning with each subtask excluded and column 501 during all-subtask learning can be recorded in influence 504 on other subtasks. In other words, the subtask mutual influence calculation module 107 calculates the average value of the difference in accuracy column 503 as the influence. The above calculation of influence c is realized by the subtask mutual influence calculation module 107 as described above. c_NER, c_RE, and c_CR are used in the overall priority score calculation module 109, which will be described later.

[0050] Next, we will explain the processing of the subtask priority score calculation module 108. Here, the output of the subtask priority score calculation module 108 is shown in FIGS. The processing of the subtask priority score calculation module 108 will be described below with reference to FIGS.

[0051] In this embodiment, the priority score for named entity extraction is output for each morpheme obtained by morphologically analyzing text data. The priority score for the target text data is then determined based on the maximum value of these scores. The priority score for relationship extraction is calculated for each combination of all named entities in the text data, and the priority score for the target text data is determined based on the maximum value of these scores. For name matching, the priority score for the target morpheme is the average of the priority scores output when determining whether a morpheme to be matched should be matched with other morphemes, and the name matching priority score for the target text data is the maximum value of these scores.

[0052] 6 shows an example of priority score information 205 for unlabeled data 101 with a data ID 201 of 1. Here, named entity extraction priority score 601 for one piece of data is the calculation result of the priority score for the named entity extraction subtask for one piece of data. Furthermore, relationship extraction priority score 602 indicates the value of the relationship priority score for a pair of entities inferred as named entities by named entity extraction. Name identification priority score 603 indicates the name identification priority score for entities inferred as named entities by named entity extraction.

[0053] FIG. 7 also shows the priority score of each subtask for each data. Here, priority scores 703, 704, and 705 for each subtask are shown for a data ID 701 and data content 702 similar to those of the unlabeled data 101. In this embodiment, the maximum value of the priority scores for each subtask of one data element shown in FIG. 6 is used as the priority score for each subtask of the data, but the overall average value or the average value of the top few scores may also be used. The output, i.e., information, shown in FIGS. 6 and 7 above may be managed together with the overall priority score as priority score information 205, which will be described below.

[0054] Next, the processing of the total priority score calculation module 109 will be described. The total priority score calculation module 109 calculates the priority score and the total priority score for one piece of data. Therefore, the processing of the total priority score calculation module 109 will be described below with reference to this. The priority score and the total priority score for one piece of data can be managed as priority score information 205.

[0055] FIG. 9 is a diagram showing the priority score for one piece of data in this embodiment. The overall priority score calculation module 109 integrates the priority scores of the results of each subtask within one piece of data to calculate the priority score for all subtasks. Specifically, the priority score for each element of the data in each subtask is multiplied by the weight calculated for each subtask to calculate the priority score for all subtasks. Note that FIG. 6 lists the target data 901 for each subtask, the target data for which subtask 902, and the weighted priority score 903.

[0056] 10 is a diagram showing the overall priority score for each data item in this embodiment. Fig. 10 shows an overall priority score 1006 obtained by weighting and adding up the priority scores 1003, 1004, and 1005 of the subtasks for the data ID 1001 and data content 1002 of the unlabeled data 101 obtained in Fig. 7. That is, the overall priority score calculation module 109 calculates the overall priority score 1006 by weighting and adding up the priority scores 1003, 1004, and 1005 of the subtasks for the data ID 1001 and data content 1002 of the unlabeled data 101. For this purpose, the following equation (1) is used.

[0057]

number

[0058] 9 and 10, the total priority score calculation module 109 can calculate the weight of each subtask used in the calculation as a function of the estimation accuracy of each subtask and the influence of each subtask on other subtasks. The calculation formula used for this calculation is shown in (Equation 2).

[0059]

number

[0060] In (Equation 2), by multiplying the result obtained by subtracting the estimated accuracy from 1 by the influence, the weight of a subtask with poor accuracy or a subtask with a large influence on other subtasks is increased. Also, α and β in (Equation 2) are parameters related to accuracy and influence, respectively, and can be adjusted in the user interface described below. Also, formulas other than (Equation 2) may be used. In other words, the weight calculation formula is a function of accuracy and influence, and any formula will do as long as α and β are parameters related to accuracy and influence, respectively.

[0061] Next, the processing flow in this embodiment will be described. Fig. 8 is a diagram showing the overall processing flow in this embodiment. First, in step S801, the supervised machine learning module 103 accepts input of labeled data 102. Then, in step S802, the supervised machine learning module 103 divides the accepted labeled data 102 into training data and validation data. Note that steps S802 to S808 may be performed using multiple patterns of division, and the accuracy and influence values ​​may be calculated by using average values ​​of the multiple patterns.

[0062] In step S803, the supervised machine learning module 103 uses the training data to learn about each subtask module 104. In other words, it learns about the subtask. In step S804, the supervised machine learning module 103 that learned in step S803 estimates validation data.

[0063] Furthermore, in parallel with steps S803 and S804, steps S805 and S806 are executed. First, in step S805, the supervised machine learning module 103 learns about the subtask modules 104 that make up itself, excluding one subtask module. In other words, learning is performed about the subtasks. In this embodiment, learning is performed about patterns excluding each of the three subtasks: named entity extraction, relationship extraction, and name matching. Furthermore, in step S806, the supervised machine learning module 103 estimates an estimated label according to the learning result in S805.

[0064] In step S807, the subtask accuracy calculation module 106 inputs the estimated labels, which are the estimation results for the validation data obtained in S804, and calculates the estimation accuracy of each subtask during all subtask learning. That is, row 503 of the subtask accuracy in column 501 during all subtask learning shown in FIG. 5 is calculated.

[0065] In step S808, the subtask mutual influence calculation module 107 calculates the influence of each subtask on other subtasks using the estimation results obtained in steps S804 and S806. This calculation method has already been explained using Fig. 5. The influence results are exemplified in influence on other subtasks 504 in Fig. 5.

[0066] In step S809, the supervised machine learning module 103 receives the unlabeled data 101 and estimates the corresponding labels. This estimation uses the model used during all-task learning. In step S810, the subtask priority score calculation module 108 calculates the priority scores for each data and each subtask based on the estimation results obtained in step S809. Examples of the calculation results are shown in FIGS. 6 and 7.

[0067] In step S811, the overall priority score calculation module 109 calculates an overall priority score based on the estimation accuracy obtained in step S807, the influence obtained in step S808, and the estimation result of each unlabeled data 101 obtained in S810. It is also desirable that the overall priority score calculation module 109 calculates a priority score for each element in each piece of data. Examples of the calculation results are shown in Figures 9 and 10.

[0068] In step S812, the label correction module 110 outputs the estimation results in order of the overall priority score. For example, the label correction module 110 outputs the estimation results to a display device, which then displays a confirmation screen including the results. As a result, the annotator can check the estimation results and input corrections as necessary. The label correction module 110 then corrects the estimation results, i.e., the estimated labels, in accordance with the input corrections. The specific contents of the confirmation screen will be described later with reference to FIGS. 11 and 12.

[0069] Next, in step S814, the label correction module 110 determines whether or not additional data to be confirmed is required. The determination of whether or not additional data is required may be made according to input from the annotator, or may be made based on criteria prepared in advance. If the result indicates that additional data is required (Yes), the process proceeds to step S813. If additional data is not required (No), the process proceeds to step S815.

[0070] In step S813, the overall priority score recalculation module 111 recalculates the overall priority score. In the recalculation, the estimation result for the confirmed data is compared with the confirmed and corrected data to calculate the estimation accuracy of each subtask. In response to this, the overall priority score recalculation module 111 updates the accuracy of each subtask using a weighted average weighted by the estimation accuracy calculated in step S807 and the number of target data items. Then, in step S808, the overall priority score recalculation module 111 also updates the influence of each subtask on other tasks according to the updated estimation accuracy of each subtask. In this way, the overall priority score recalculation module 111 recalculates the weight for the priority score of each subtask in step S811. As a result, the overall priority score can be recalculated to reflect the recalculated weight. Once the overall priority score has been recalculated, the process of step S812 is executed for data that has not yet been confirmed.

[0071] In step S815, the label correction module 110 determines whether re-learning is necessary. Regarding this determination, the label correction module 110 may make the determination based on criteria prepared in advance, or may make the determination based on input from the annotator. As a result, if re-learning is necessary (Yes), the process proceeds to step S816. If re-learning is not necessary (No), the process ends.

[0072] Furthermore, in step S816, supervised machine learning module 103 adds the confirmed data to labeled data 102. Then, in step S801, supervised machine learning module 103 performs re-learning using the added labeled data 102. Thereafter, the processes of steps S801 to S815 are repeated. This concludes the description of this flow diagram.

[0073] Next, the confirmation screen displayed on the display device in step S812 will be described. FIGS. 11 and 12 are diagrams showing confirmation screens in this embodiment. First, FIG. 11 is a diagram showing a confirmation screen 1100 for all data to be confirmed in this embodiment. In FIG. 11, the confirmation screen 1100 has display areas for data to be confirmed 1101, data ID 1102, data content 1103, and label 1104. The confirmation screen 1100 also displays designation tools such as an edit button 1105, a total priority score adjustment slide bar 1106, an add confirmation data button 1107, a relearn button 1108, and a complete button 1109.

[0074] First, the data to be confirmed and its estimation result are displayed in data to be confirmed 1101. At this time, the unlabeled data 101 and its estimation result are sorted in descending order of overall priority score 1006 calculated for each data by overall priority score calculation module 109 and overall priority score recalculation module 111, and a pre-specified number of data are displayed. Furthermore, information corresponding to data ID 201 and data content 202 in unlabeled data 101 is displayed in data ID 1102 and data content 1103.

[0075] Furthermore, label 1104 displays the inference result, i.e., the inferred label. At this time, the inference result may be displayed for each subtask, but in this example, the results of named entity extraction, relationship extraction, and name matching are combined to display the result of event extraction inferred from the data. Furthermore, edit button 1105 is a button used to check and edit the details of each piece of data. When the annotator selects edit button 1105, the screen transitions to a details confirmation screen 1200 for each piece of data. The details confirmation screen 1200 after the transition will be described later with reference to FIG. 12.

[0076] Furthermore, the overall priority score adjustment slide bar 1106 is used to make adjustments to the overall priority score. That is, the overall priority score of each data is recalculated according to the values ​​of the slide bars for the accuracy of the subtask and the degree of influence on other subtasks specified by the annotator. For this purpose, the overall priority score recalculation module 111 changes the values ​​of α and β in (Equation 1) according to the specified values, and recalculates the overall priority score of each data. After recalculation, the overall priority score recalculation module 111 rearranges the data again in order of overall priority score, and displays it in the data to be confirmed 1101 on the confirmation screen 1100.

[0077] Furthermore, when the annotator presses the confirmation data addition button 1107, recalculation is performed in step S813. That is, the total priority score recalculation module 111 recalculates the total priority score using the accuracy of the data to be confirmed 1101 up to that point, and the process returns to step S812 again.

[0078] Furthermore, when the annotator presses the Relearn button 1108, the process of step S816 is executed. That is, the supervised machine learning module 103 adds the confirmed data to the labeled data 102, and the process returns to step S801. Furthermore, when the annotator presses the Done button 1109, the process flow shown in FIG. 8 ends.

[0079] Next, a description will be given of the details confirmation screen 1200 that is displayed when the edit button 1105 is pressed, and the processing that is performed at that time. Fig. 12 is a diagram showing the details confirmation screen 1200 in this embodiment. In this example, the case where the data ID in Fig. 11 is 1 is shown. In Fig. 12, the details confirmation screen 1200 displays display areas for the data body 1201 and confirmation items 1202, as well as a YES button 1203 and a NO button 1204. The details confirmation screen 1200 will also be displayed on the display device described above.

[0080] First, the data content is displayed in the data body 1201. In this example, the confirmation items are highlighted, and it is also indicated which part of the text to focus on.

[0081] Furthermore, confirmation items 1202 display the content to be confirmed on this screen. This display is performed in the order of the weighted priority score 903 shown in FIG. 9. In FIG. 12, a display is shown for confirming named entity extraction for "Part A," which has the highest weighted priority score. For this display, it is desirable that the question format be determined in advance according to each subtask, such as named entity extraction, relationship extraction, and name matching. For example, the question format is determined as follows for named entity extraction: "Is 'Part A' a part name?"; for relationship extraction: "Is there a relationship between 'Part A' and 'Exchange'?"; and for name matching: "Can 'Part A' be rephrased as 'AAA'?"

[0082] In addition, the YES button 1203 and NO button 1204 accept the annotator's feedback on the detailed confirmation. When the YES button 1203 is pressed, the label correction module 110 adds the data to the confirmed data as is. When the NO button 1204 is pressed, the label correction module 110 may save only the information that the label is incorrect as labeled data, or may have the annotator correct the label. As described above, when each button is pressed, the label correction module 110 displays the confirmation item with the next highest priority score in the confirmation item 1202, in accordance with the weighted priority score 903 shown in FIG. 9. As a result, the annotator can confirm this content. Furthermore, when a certain number of confirmations have been completed for one confirmation data, the screen returns to the screen shown in FIG. 11. This concludes the description of the processing of the embodiment.

[0083] Here, the annotation device 1 of this embodiment can be configured as an annotation system realized by a cloud system. This configuration will be described below. Fig. 13 is a configuration diagram of the annotation system in this embodiment. In this embodiment, a corporate system made up of the annotation device 1, a terminal device 2, a corporate terminal device 3, and a corporate server 4 is connected via a network 5. In other words, the annotation system is configured as a cloud system.

[0084] First, the annotation device 1 can be realized by a computer such as a server. To this end, the annotation device 1 has a processing device 11, a communication device 12, a memory 13, and an auxiliary storage device 14, which are connected to each other via a communication path. Furthermore, the annotation device 1 in FIG. 13 is provided with a group of arithmetic devices 15 that performs calculations (such as voice synthesis) on documents for which an event extraction task has been executed, but this can be omitted. Furthermore, the group of arithmetic devices 15 may be realized by a dedicated circuit such as an FPGA (Field-Programmable Gate Array), or the processing device 11 described below may perform calculations on documents. Furthermore, calculations on documents may be performed by a device other than the annotation device 1, such as a terminal device 2 or a functional system.

[0085] First, the processing device 11 can be realized by a processor such as a CPU (Central Processing Unit), and executes processes such as calculations according to programs stored in an auxiliary storage device 14, which will be described later. The communication device 12 communicates with other devices via the network 5. The memory 13 stores the programs stored in the auxiliary storage device 14 and information used in the processing by the processing device 11.

[0086] The auxiliary storage device 14 can be realized by a so-called storage and stores the above-mentioned programs and information. The auxiliary storage device 14 may also be realized by various storage media such as an external HDD (Hard Disk Drive), SSD (Solid State Drive), or memory card. Furthermore, at least one of the information may be stored in a database system other than this device.

[0087] In the example shown in Figure 13, each module in Figure 1 is implemented as a computer program. To this end, the auxiliary storage device 14 stores a program consisting of a supervised machine learning module 103, a subtask accuracy calculation module 106, a subtask mutual influence calculation module 107, a subtask priority score calculation module 108, an overall priority score calculation module 109, a label correction module 110, and an overall priority score recalculation module 111. Therefore, the processing device 11 executes the processing of each module in accordance with the computer program. Note that each module may be implemented as an independent program, or a program may be configured by combining parts of the modules.

[0088] Furthermore, the auxiliary storage device 14 stores the unlabeled data 101, the labeled data 102, the subtask accuracy information 204, and the priority score information 205. These are as described above. This concludes the description of the annotation device 1.

[0089] Next, the terminal device 2 is a terminal device used by an annotator, and can be realized by a computer such as a PC, tablet, or smartphone. For this reason, it is desirable that the terminal device 2 has the above-mentioned input device and display device. In addition, the terminal device 2 may be provided with the functions of any of the above-mentioned modules.

[0090] The enterprise system is composed of enterprise terminal devices 3-1 and 3-2 (hereinafter simply referred to as enterprise terminal devices 3) and business servers 4-1 and 4-2 (hereinafter simply referred to as enterprise servers 4), and is used to carry out the business of the enterprise. In this embodiment, business is carried out using documents, which are the target data. This embodiment is configured so that the annotation device 1 can be accessed and used from the enterprise systems of multiple enterprises via a network 5. This network 5 can be realized by a wide area network such as the Internet. However, the annotation device 1 may also be constructed as a dedicated system for a specific enterprise.

[0091] Furthermore, the company terminal device 3 may be used by an employee of the company as an annotator. In other words, the company terminal device 3 may execute the same functions as the terminal device 2. In this way, the company terminal device 3 may function as an input / output device for the annotation device 1, or may execute a function for performing document-related tasks. Furthermore, the company terminal device 3 may execute both of these functions. This concludes the explanation of Fig. 13. The annotation device 1 of this embodiment may be configured as a so-called standalone type, or the function may be provided on the terminal device side, such as a PC.

[0092] This concludes the description of this embodiment, but the present invention is not limited to the above-described aspects. For example, the present invention can be applied to data other than documents. Furthermore, in this embodiment, processing is performed on a subtask-by-subtask basis, but it may also be performed on a combination of tasks or subtasks. [Explanation of symbols]

[0093] 101 Unlabeled Data 102 Labeled Data 103 Supervised Machine Learning Module 104 Subtask Module 105 Shared Models 106 Subtask accuracy calculation module 107 Subtask mutual influence calculation module 108 Subtask Priority Score Calculation Module 109 Overall Priority Score Calculation Module 110 Label Correction Module 111 Overall Priority Score Recalculation Module 401 Named Entity Recognition Model 402 Relationship Extraction Model 403 Name Matching Model 404 Text Data 405 Vector Representation 406 Feature Vector 407 Named Entity Labels 408 Relationship Labels 409 Name Estimation Label

Claims

1. In an annotation device that realizes machine learning labeling, a supervised machine learning module having a plurality of subtask modules that learn using labeled data and estimate estimated labels for unlabeled data for each of a plurality of subtasks that constitute the machine learning task; a subtask mutual influence calculation module that calculates an influence indicating the degree of influence of learning in the subtask module on other subtask modules; An annotation device having a label correction module that corrects the estimated label in accordance with a confirmation result of the estimated label presented in accordance with a total priority score based on the influence degree.

2. The annotation device according to claim 1, Further, a subtask accuracy calculation module is provided for calculating an estimation accuracy indicating the accuracy of the estimated label, The label correction module corrects the estimated label in accordance with a confirmation result of the estimated label presented according to an overall priority score based on the estimation accuracy and the influence.

3. The annotation device according to claim 2, further comprising a subtask priority score calculation module that calculates a subtask priority score relating to the order of confirmation for each data unit of the data handled by the plurality of subtask modules; The label correction module corrects the estimated label in accordance with a confirmation result of the estimated label presented according to an overall priority score based on the estimation accuracy, the influence, and the subtask priority score.

4. The annotation device according to claim 3, The annotation device further includes an overall priority score calculation module that calculates an overall priority score for the unlabeled data as the overall priority score using the estimation accuracy, the influence, and the subtask priority scores.

5. The annotation device according to claim 4, The annotation device further includes a re-overall priority score calculation module that recalculates the overall priority score in response to correction of the estimated label by the label correction module.

6. An annotation method for realizing machine learning labeling using an annotation device, A supervised machine learning module having a plurality of subtask modules for estimating estimated labels for unlabeled data is used to learn, using labeled data, each of a plurality of subtasks constituting the machine learning task; Calculating an influence level indicating the degree of influence of learning in the subtask module on other subtask modules; The annotation method modifies the estimated label according to a confirmation result of the estimated label presented according to the overall priority score based on the influence.

7. The annotation method according to claim 6, Furthermore, an estimation accuracy indicating the accuracy of the estimated label is calculated, An annotation method that modifies the estimated label according to a confirmation result for the estimated label presented according to an overall priority score based on the estimation accuracy and the influence.

8. The annotation method according to claim 7, Furthermore, a subtask priority score is calculated for each data unit of the data handled by the plurality of subtask modules, and the subtask priority score is calculated for each data unit of the data handled by the plurality of subtask modules. An annotation method that modifies the estimated label according to a confirmation result of the estimated label presented according to an overall priority score based on the estimation accuracy, the influence, and the subtask priority score.

9. The annotation method according to claim 8, Furthermore, the annotation method calculates an overall priority score for the unlabeled data using the estimation accuracy, the influence, and the subtask priority score as the overall priority score.

10. The annotation method according to claim 9, Furthermore, the annotation method recalculates the overall priority score in response to modifications of the estimated label.

Images