Program, learning device and method for retraining an explainable machine learning engine

Abstract

To retrain a machine learning engine where a labeled sample is trained as teacher data, by applying a label with high validity even to a label-less sample.SOLUTION: A learning device repeatedly executes the steps of: by using a first machine learning engine, performing training with teacher data where a label is applied to a sample of one or more target items and a state value, and then estimating a feature amount importance contributing to the label when the label is predicted from the one or more target items and the state value, and outputting the target items, the state value, and the label when the feature amount importance is equal to or greater than a first prescribed condition; and by using a second machine learning engine, estimating an appropriateness likelihood of the label with respect to the target items and the state value inputted from the first machine learning engine, resetting the state value to a pseudo value when the appropriateness likelihood is equal to or less than a second prescribed condition, outputting the target items, the pseudo value, and the label, and retraining the first machine learning engine.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a machine learning engine technology that trains a group of labeled samples as training data. In particular, it is suitable for reducing the burden of annotation work on annotators who create training data by assigning labels to samples. [Background technology]

[0002] Supervised learning uses training data in which samples are labeled in advance. Training data is generally created by annotators who recognize samples and manually assign labels they deem appropriate.

[0003] In contrast, there is a technique called "active learning" that sequentially selects samples that will contribute to improving classification accuracy and asks the annotator for their recommendations for efficient learning. "Active learning" refers to a learning strategy in which a machine learning engine for class classification is trained using a small number of labeled samples available, and then the engine "actively" selects from a group of unlabeled samples those samples that will most improve the engine's classification performance, and asks annotators to label only those samples. Since only specific samples need to be labeled, the overall work cost of the annotators can be reduced. The machine learning engine can then be retrained using the samples labeled by the annotators. This cycle is then repeated. The end of the cycle can be determined, for example, when the learning model of the machine learning engine achieves a predetermined accuracy, or when the resources available to the annotator (such as the number of allowable inquiries and monetary costs) are exhausted.

[0004] Active learning involves the technique of "uncertainty sampling" to actively select samples from a set of unlabeled samples in order to improve the classification accuracy of a machine learning engine. "Uncertainty sampling" refers to a technique for selecting the data with the lowest confidence when classifying a group of unlabeled samples into each class. Specifically, the likelihood p={p1, p2, …, pc} (c is the number of classes) when classifying a group of unlabeled samples into classes is output, and the sample with the lowest probability of maximizing the likelihood is selected (Least confident sampling). In addition to least confident sampling, uncertainty sampling also includes margin sampling, a technique for selecting a sample that minimizes the difference between the highest probability and the second highest probability, and entropy sampling, a technique for selecting a sample that maximizes the entropy of the distribution of p.

[0005] Conventionally, there is a technique for avoiding sampling more data of the same class than necessary for each learning step by clustering a group of unlabeled samples in advance and then sampling a predetermined number of samples from within the cluster (see, for example, Patent Document 1). For example, in the uncertainty sampling used in the active learning described above, sampling is based solely on the confidence level. As a result, there is a possibility that samples belonging to the same class will be selected biasedly for each learning step (i.e., only data from classes that are difficult to classify at that point in time will be selected). According to Patent Document 1, when requesting labeling from an annotator for each learning step, it is possible to resolve the inefficiency caused by requesting similar data.

[0006] There is also a technique for sampling by evaluating the degree of agreement between samples (see, for example, Patent Document 2). Specifically, this technique first creates multiple augmented training samples by perturbing (subtle changes) the features of each piece of data in a group of unlabeled samples. Then, the degree of disagreement between the classes resulting from classification of each augmented training sample by a classifier is evaluated, and samples with high disagreement are selected. This is based on the hypothesis that the model decisions must be consistent between groups of samples obtained by perturbing the same sample.

[0007] FIG. 1 is an explanatory diagram showing annotations for training data in the prior art.

[0008] FIG. 1 shows the work of an annotator annotating human behavior (label) based on the ON / OFF (state value) of each sensor (target item). As shown in Figure 1, a wide variety of sensors are placed in a room, for example. The ON / OFF state values ​​detected by each sensor are collected by the learning device 1. The following types of sensors are assumed to capture the spatial presence of a person: Kitchen_sensor Refrigerator_sensor Bedroom_sensor Air-con_sensor Door_sensor

[0009] Each sensor detects ON / OFF at a predetermined cycle (e.g., every second) and transmits the data to the learning device 1. For example, these time-series sensor data can be considered to represent the behavior of people staying in the room. By analyzing the patterns of these sensor data, it becomes possible to recognize people's behavior.

[0010] According to FIG. 1, the annotator visually recognizes the ON / OFF status of the following multiple sensors: Kitchen_sensor ON Refrigerator_sensor ON Bedroom_sensor ON Air-con_sensor ON Door_sensor OFF In this case, the annotator needs to recognize all the data from multiple sensors. For example, the annotator may focus on the ON status of the Kitchen_sensor and Refrigerator_sensor and assign the label "Cooking." The longer it takes the annotator to assign a label, the higher the work cost and the greater the workload. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Patent Publication No. 2021-047751 [Patent Document 2] Special Publication No. 2022-545476 [Non-patent literature]

[0012] [Non-Patent Document 1] Yoshitaka Kameya, "Explainable AI Technology: The Past and Future," [online], [Retrieved February 20, 2023], Internet<URL: https: / / www.jstage.jst.go.jp / article / essfr / 16 / 2 / 16_83 / _pdf> Summary of the Invention [Problem to be solved by the invention]

[0013] However, according to the aforementioned Figure 1, this does not reduce the workload of the annotator in assigning labels (behaviors) to samples (ON / OFF for each sensor). The greater the checking burden on the annotator, the greater the possibility of errors in labeling. Even in the case of Figure 1, it is necessary to visually confirm the ON / OFF status of five sensors. If the annotator assigns an incorrect label, the quality of the training data will deteriorate, and the classification accuracy of the learning model trained using that training data will also deteriorate. In particular, when only a few annotators can be employed (for example, when only the annotators can label data due to knowledge that only they have access to, or when the cost of employing them exceeds the budget), the workload increases, which can easily lead to a deterioration in the quality of the training data. Furthermore, methods such as majority voting and weighting the reliability of each annotator may not work.

[0014] Therefore, an object of the present invention is to provide a program, a learning device, and a method that can retrain a machine learning engine that has been trained using labeled samples as training data by assigning highly valid labels to unlabeled samples. [Means for solving the problem]

[0015] According to the present invention, a first machine learning engine is provided that is trained using teacher data in which labels are assigned to samples of one or more target items and state values, and that, when predicting a label from one or more target items and state values, estimates feature importance that contributed to the label, and outputs the target item, state value, and label when the feature importance is equal to or greater than a first predetermined condition; a second machine learning engine that uses a pre-trained language model to estimate the validity likelihood of the label for the target item and state value input from the first machine learning engine, and if the validity likelihood is equal to or less than a second predetermined condition, resets the state value to a pseudo value, outputs the target item, the pseudo value, and the label, and causes the first machine learning engine to retrain; The present invention is characterized by the fact that it makes a computer function by using the above-mentioned method.

[0016] According to another embodiment of the program of the present invention, The first machine learning engine is XAI (Explainable AI), Each target item and state value becomes a feature, Labels are a determining factor. Feature importance is an explanatory factor It is also preferable to make the computer function in this way.

[0017] According to another embodiment of the program of the present invention, The state value is binary, The second machine learning engine resets the state value to a pseudo value that is the inverse of the state value input from the first machine learning engine. It is also preferable to make the computer function in this way.

[0018] According to another embodiment of the program of the present invention, The state value is a number, The second machine learning engine resets the state value to a pseudo-value that is a statistic of the state values ​​of multiple other samples. It is also preferable to make the computer function in this way.

[0019] According to another embodiment of the program of the present invention, The statistical value of the state value is the mean, median, or mode. It is also preferable to make the computer function in this way.

[0020] According to another embodiment of the program of the present invention, the first predetermined condition in the first machine learning engine is a predetermined threshold of feature importance; The second predetermined condition in the second machine learning engine is a predetermined threshold of the likelihood of validity. It is also preferable to make the computer function in this way.

[0021] According to another embodiment of the program of the present invention, The second machine learning engine uses a language corpus and is based on a graph model in which target item words and label words are treated as nodes, and the relationships between nodes are linked by state values. It is also preferable to make the computer function in this way.

[0022] According to another embodiment of the program of the present invention, The second machine learning engine is based on a natural language model trained with natural language sentences with correct labels and natural language sentences with incorrect labels; natural language sentence conversion means for converting the target items, state values, and labels output from the first machine learning engine into natural language sentences and inputting them to the second machine learning engine; It is also preferable to make the computer function as described above.

[0023] According to another embodiment of the program of the present invention, The natural language sentences with correct labels are sentences from a language corpus, A natural language sentence with an incorrect label is one in which some words in the natural language sentence with the correct label have been replaced with other words. It is also preferable to make the computer function in this way.

[0024] According to another embodiment of the program of the present invention, a confirmation flag assigning means for assigning a confirmation flag to an annotator for the labeled sample whose state value has been reset to the pseudo value, which is input from the second machine learning engine; and further enhance its functionality. The confirmation flag assigning means outputs the target item, the state value or the pseudo value, and the label to which the confirmation flag has been assigned to the user interface, and prompts the annotator to answer whether the answer is correct or incorrect. It is also preferable to make the computer function in this way.

[0025] According to another embodiment of the program of the present invention, The first machine learning engine actively selects samples from a set of unlabeled samples for which labels should be predicted using uncertainty sampling. It is also preferable to make the computer function in this way.

[0026] According to the present invention, a first machine learning engine is provided that is trained using teacher data in which labels are assigned to samples of one or more target items and state values, and that, when predicting a label from one or more target items and state values, estimates feature importance that contributed to the label, and outputs the target item, state value, and label when the feature importance is equal to or greater than a first predetermined condition; a second machine learning engine that uses a pre-trained language model to estimate the validity likelihood of the label for the target item and state value input from the first machine learning engine, and if the validity likelihood is equal to or less than a second predetermined condition, resets the state value to a pseudo value, outputs the target item, the pseudo value, and the label, and causes the first machine learning engine to retrain; The present invention is characterized by having the following.

[0027] According to the present invention, a learning device comprises: a first step of using a first machine learning engine to train with teacher data in which labels are assigned to samples of one or more target items and state values, estimating feature importance that contributed to the label when predicting the label from one or more target items and state values, and outputting the target item, state value, and label when the feature importance is equal to or greater than a first predetermined condition; a second step of using a second machine learning engine to estimate the validity likelihood of the label for the target item and state value input from the first machine learning engine using a pre-trained language model, and if the validity likelihood is equal to or less than a second predetermined condition, resetting the state value to a pseudo value, outputting the target item, the pseudo value, and the label, and causing the first machine learning engine to retrain; The method is characterized in that the above steps are repeatedly executed. [Effects of the Invention]

[0028] According to the program, learning device, and method of the present invention, a machine learning engine that has been trained using labeled samples as training data can be retrained by assigning highly valid labels to unlabeled samples. [Brief explanation of the drawings]

[0029] [Figure 1] FIG. 1 is an explanatory diagram showing annotations for training data in the prior art. [Figure 2] FIG. 2 is a functional configuration diagram of a learning device according to the present invention. [Figure 3] FIG. 2 is an explanatory diagram showing interactions between machine learning engines in the present invention. [Figure 4] FIG. 1 is a first diagram illustrating a second machine learning engine based on a graph model. [Figure 5] FIG. 2 is a second diagram illustrating a second machine learning engine based on a language corpus. [Figure 6] FIG. 2 is an explanatory diagram showing an annotation in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings. Specifically, an example of an application in which indoor sensors are used to estimate the behavior of a resident is taken as an example.

[0031] FIG. 2 is a functional configuration diagram of the learning device according to the present invention.

[0032] 2, the learning device 1 includes a first machine learning engine 11, a second machine learning engine 12, a natural language-to-text conversion unit 13, and a confirmation flag assignment unit 14. These functional components are realized by executing a program that causes a computer installed in the learning device to function. The processing flow of these functional components can also be understood as a learning method for the device.

[0033] [First Machine Learning Engine 11] The first machine learning engine 11 is XAI (see, for example, Non-Patent Document 1). This can estimate feature importance (Permutation Importance) (sometimes called confidence), which is an index for evaluating "how much a certain feature contributes to improving the prediction accuracy of that label." Feature importance can be estimated by perturbing a subset of samples and observing the change in class classification prediction. This involves looking at a counterfactual situation for the perturbed subset. That is, if the perturbation significantly reduces the confidence of the class, then the subset can be said to be an important feature for that class. Examples of XAI include LIME (Local Interpretable Model-agnostic Explanations), SHAP (SHapley Additive Explanations), and Anchor.

[0034] According to FIG. 2, the first machine learning engine 11 is trained using teacher data in which labels (judgment factors) are assigned to one or more target items and state values ​​(samples). <Sample> Target item, state value (equivalent to feature value) <Decision Element> Label <Explanatory element> Feature importance The target item is the sensor identifier of each sensor, and the status value is the ON / OFF detected by each sensor. The status value may be a binary value or a numeric value. Furthermore, the label represents a person's behavior as a determining factor. Furthermore, feature importance represents the degree to which a feature contributes to a label (decision factor) when the label is predicted from one or more target items and state values ​​(samples). Specifically, the state values ​​of some target items are perturbed, and the change in the predicted label is observed. If the perturbation of the state values ​​of some target items significantly reduces the confidence of the label, the target item can be said to have a high feature importance contributing to the label.

[0035] The first machine learning engine 11 of the present invention can assign highly relevant labels to unlabeled samples via the second machine learning engine 12, and use them for retraining. Here, the first machine learning engine 11 may randomly select samples whose labels are to be predicted from a group of unlabeled samples. Alternatively, the first machine learning engine 11 may actively select samples by uncertainty sampling. This can improve the classification accuracy of the learning model in the first machine learning engine 11 itself.

[0036] [Second Machine Learning Engine 12] The second machine learning engine 12 is a general-purpose machine learning engine that uses a language model pre-trained with a large amount of general knowledge. It can input pairs of target items and state values ​​(samples) and labels (judgment factors) and estimate their validity likelihood. Specifically, the second machine learning engine 12 may be a general-purpose engine such as a random forest, a support vector machine, or a neural network. As will be described later, the second machine learning engine 12 may be based on a natural language model or a graph model, for example.

[0037] FIG. 3 is an explanatory diagram showing the interactions between machine learning engines in the present invention.

[0038] When the feature importance is equal to or greater than the “first predetermined condition,” the first machine learning engine 11 outputs the target item, the state value, and the label to the second machine learning engine 12. Here, the "first predetermined condition" may be, for example, a predetermined threshold value for feature importance. When the feature importance is equal to or greater than the predetermined threshold value, it can be determined that the target item and state value contribute to the label. Of course, the first machine learning engine 11 may output multiple pairs of target item, state value, and label, or may not output any pairs at all.

[0039] According to FIG. 3, the first machine learning engine 11 predicts the label of an unlabeled sample and estimates the feature importance as follows. < Sample >< Decision Element >< Explanation Element > Target item State value Label Feature importance Kitchen_sensor ON Cooking 0.9 Refrigerator_sensor ON Cooking 0.7 Bedroom_sensor ON Cooking 0.7 Air-con_sensor ON Cooking 0.2 Door_sensor OFF Cooking 0.1

[0040] Here, for example, ON detection of the Kitchen_sensor or Refrigerator_sensor is likely to occur when a person's behavior is Cooking. On the other hand, ON detection of the Bedroom_sensor is likely to occur when a person's behavior is Sleeping. However, according to Figure 3, even when the Bedroom_sensor is ON, it is assigned the label Cooking within the overall system. Note that ON detection of the Air-con_sensor depends on whether a person is present, and OFF detection of the Door_sensor depends on whether a person is moving inside or outside the room. In this way, the detection of each sensor differs depending on the person's behavior in the room.

[0041] 3, the predetermined condition is a predetermined threshold of 0.7, and only pairs of target items and state values ​​(samples) and labels (decision factors) whose feature importance is equal to or greater than the predetermined threshold of 0.7 are output to the second machine learning engine 12. In other words, only labeled samples with high feature importance are output. < Sample >< Decision Factors > Target item Status value Label Kitchen_sensor ON Cooking Refrigerator_sensor ON Cooking Bedroom_sensor ON Cooking

[0042] Next, the second machine learning engine 12 estimates the likelihood of validity of the label for the target item and state value input from the first machine learning engine 11. < Sample >< Decision Factors > Target item State value Label Validity likelihood Kitchen_sensor ON Cooking 0.9 Refrigerator_sensor ON Cooking 0.8 Bedroom_sensor ON Cooking 0.2

[0043] Then, the second machine learning engine 12 determines whether the validity likelihood is equal to or less than a "second predetermined condition." Here, the "second predetermined condition" may be a predetermined threshold value of the validity likelihood. According to Fig. 3, the predetermined threshold value is set to 0.3, and only pairs of target items and state values ​​(samples) and labels (judgment factors) whose validity likelihood is equal to or less than the predetermined threshold value of 0.3 are selected. < Sample >< Decision Factors > Target item State value Label Validity likelihood Bedroom_sensor ON Cooking 0.2

[0044] Here, the second machine learning engine 12 determines whether the likelihood of validity based on general knowledge is high for target items, state values, and labels that are estimated by the first machine learning engine 11 to have high feature importance. Here, if the likelihood of validity is high, there is no problem. On the other hand, a low likelihood of validity is problematic: the first machine learning engine 11 has been trained with a label that is estimated to have a low likelihood of validity based on general knowledge, even though the feature importance that contributed to the label when predicting the label from the target item and state value (sample) is high.

[0045] According to the present invention, in order to correct this bias, general knowledge is taken into account and if the likelihood of validity is less than or equal to a second predetermined condition, the state value is reset to a pseudo value.

[0046] The state value may be a binary value such as ON / OFF, or may be a numerical value within a predetermined range. <When the state value is two-valued> The second machine learning engine 12 resets the state value to a "pseudo value" that is the inverse value (or counterexample) of the state value input from the first machine learning engine. 3, the state value is expressed as two values, ON / OFF, and the validity likelihood for the state value ON of Bedroom_sensor and the label Cooking is expressed as 0.2. Here, the state value ON is reset to the pseudo value OFF. Target item Pseudo value Label Bedroom_sensor OFF Cooking

[0047] <If the status value is a number> The second machine learning engine 12 resets the state value to a "pseudo value" that is a statistical value of the state values ​​of multiple other samples. The statistical value of the state value may be, for example, the mean, median, or mode. In particular, for categorical features, it is preferable to use the mode. The second machine learning engine 12 acquires the statistical values ​​of the state values ​​for the same target item in the other samples from the training data.

[0048] Then, the second machine learning engine 12 outputs the target item and the set of the “pseudo value” and the label to the first machine learning engine 11.

[0049] 3, the first machine learning engine 11 executes retraining using the target item Bedroom_sensor, the pseudo value OFF, and the label "Cooking." This means that the first machine learning engine 11 is retrained so that the feature importance of the target item Bedroom_sensor, the state value ON, and the label "Cooking" is lowered. In other words, the XAI can be retrained based on general knowledge without the need to confirm feedback to the annotator, which is expected to improve generalization performance.

[0050] FIG. 4 is a first diagram illustrating a second machine learning engine based on a graph model.

[0051] <Graph model> The second machine learning engine 12 is based on a graph model using a language corpus, in which target item words and label words are treated as nodes, and the relationships between the nodes are linked by state values. The graph model may be created by linking sensors and behavioral states from a general language corpus, or may be created by linking sensors and behavioral states previously obtained from a large number of households. The links may express weights, which may represent the proportion of times that a given behavior label is assigned when the sensor is detected as ON. The weights may express the likelihood of validity between the sensor (target item and state value) and the behavior label.

[0052] FIG. 5 is a second diagram illustrating a second machine learning engine based on a language corpus.

[0053] <Natural Language Model> The second machine learning engine 12 is based on a natural language model trained with natural language sentences labeled as valid and invalid, i.e., natural language sentences are trained with a binary value of either valid or invalid.

[0054] Here, a natural language sentence with a valid label is a sentence from a language corpus, while a natural language sentence with an invalid label is a natural language sentence with some words replaced with other words. For example, consider a valid natural language sentence: "He puts sugar in his coffee." In this case, for example, by replacing "sugar" with "the building," we can create an invalid natural language sentence: "He puts the building in his coffee." The second machine learning engine 12 is then trained using the natural language sentence "He puts sugar in his coffee." that has been correctly labeled and the natural language sentence "He puts the building in his coffee." that has been incorrectly labeled. Here, the second machine learning engine 12 can be created by fine-tuning a trained language model such as BERT (Bidirectional Encoder Representations from Transformers). Alternatively, the second machine learning engine 12 can be created by training a general-purpose classification model such as a random forest, a support vector machine, or a neural network using Word2vec to average vectors of words contained in a sentence as features.

[0055] When the second machine learning engine 12 is based on a natural language model, a natural language text conversion unit 13 is further provided between the first machine learning engine 11 and the second machine learning engine 12 .

[0056] [Natural Language Text Conversion Section 13] The natural language sentence conversion unit 13 converts the target item, state value, and label output from the first machine learning engine 11 into a natural language sentence, and inputs it to the second machine learning engine 12. For example, if the target item is Kitchen_sensor, the state value is ON, and the label is Cooking, the sentence is converted into a natural language sentence such as "I think you were cooking, because the kitchen sensor responded." Also, for example, if the target item is Bedroom_sensor, the state value is ON, and the label is Cooking, the sentence is converted into a natural language sentence such as "I think you were cooking because the bedroom sensor responded." The second machine learning engine 12 receives the converted natural language sentence and determines whether it is valid or invalid. If it determines that the sentence is invalid, the second machine learning engine 12 resets the state values ​​of the target item, state value, and label to pseudo values ​​and returns them to the first machine learning engine 11.

[0057] [Confirmation flag assignment section 14] The confirmation flag assigning unit 14 assigns a confirmation flag to the annotator for the labeled sample, which has been input from the second machine learning engine 12 and whose state value has been reset to a pseudo value. The confirmation flag assigning unit 14 may assign confirmation flags to all labeled samples whose state values ​​have been reset to pseudo values, or may assign confirmation flags to only a predetermined percentage or a predetermined number of samples out of multiple samples. Of course, confirmation flags may be assigned randomly. According to the present invention, confirmation flags are assigned only to labeled samples determined by the second machine learning engine 12 to be highly valid. Therefore, confirmation may be performed by picking samples according to the workload of the annotator.

[0058] The confirmation flag assigning unit 14 then outputs the target item, state value or pseudo value, and label to which the confirmation flag has been assigned to the user interface, and prompts the annotator to answer whether the answer is correct or incorrect. This prompts the annotator to confirm the labeled samples that will become training data for retraining. According to the present invention, samples that are strongly biased by a user (which generally have a low likelihood of validity) are identified using general knowledge, and a confirmation flag is assigned to them, allowing annotators to intuitively recognize them.

[0059] FIG. 6 is an explanatory diagram showing annotations in the present invention.

[0060] According to FIG. 6, target items to which confirmation flags have been assigned and whose feature importance is equal to or greater than the first predetermined condition, as well as state values ​​or pseudo values ​​and labels, are output to the user interface. The first representation format intentionally displays the previous "state value" for the annotator to confirm. During the action "Cooking," the annotator recognizes that the Bedroom_sensor is ON and will not be detected, and answers "NO." On the other hand, the second representation format displays the reset “pseudo value” and allows the annotator to confirm it. When the action is “Cooking,” the Bedroom_sensor is recognized as being detected as OFF, and the answer is YES.

[0061] According to the conventional technique shown in Figure 1, all target items and state values ​​(features) are displayed, which places a high recognition burden on the annotator. In other words, frequent interaction with the annotator is required to create highly accurate training data. On the other hand, according to FIG. 6 of the present invention, the target items and state values ​​are limited and can be intuitively recognized by the confirmation flags, thereby reducing the cognitive burden on the annotator. The target items and state or pseudo values ​​and labels identified by the annotator are to be retrained by the first machine learning engine 11.

[0062] According to the present invention, the annotator only needs to confirm labeled samples that are determined to have a high likelihood of validity by the second machine learning engine 12. That is, since only labeled samples that are highly valid based on general knowledge are displayed, the annotator can basically answer as correct [YES]. Furthermore, since a confirmation flag is assigned to labeled samples that are low in validity based on general knowledge, the annotator can intuitively recognize them and determine whether they are correct [YES] or incorrect [NO].

[0063] As described above in detail, according to the program, learning device, and method of the present invention, a machine learning engine that has been trained using labeled samples as training data can be retrained by assigning highly valid labels to unlabeled samples.

[0064] According to the present invention, even if the first machine learning engine of the XAI is trained only on teacher data of a small number of labeled samples, it can be retrained with highly relevant labeled samples by obtaining feedback from the second machine learning engine of the general-purpose AI. Furthermore, when applied to active learning, only highly relevant labeled samples can be presented to the annotator, thereby reducing the workload. In other words, the labeling cost of the annotator in active learning can be significantly reduced.

[0065] The present invention can also be applied to, for example, human behavior estimation using smart homes and wearable devices. Such applications typically require a large number of labeled samples, and from the perspective of privacy and accuracy improvement, it is desirable for users to label the samples themselves. In contrast, the present invention significantly reduces the burden of labeling on the user side.

[0066] Furthermore, this will enable, for example, "machine learning engines to be retrained by assigning highly relevant labels to unlabeled samples," which will contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0067] With respect to the various embodiments of the present invention described above, various changes, modifications, and omissions that fall within the scope of the technical spirit and aspects of the present invention may be easily made by those skilled in the art. The above description is merely illustrative and is not intended to be limiting in any way. The present invention is limited only by the claims and their equivalents. [Explanation of symbols]

[0068] 1 Learning device 11 The No. 1 Machine Learning Engine 12. The Second Machine Learning Engine 13 Natural Language Text Conversion Unit 14 Confirmation flag assignment section

Claims

1. a first machine learning engine that is trained using teacher data in which labels are assigned to samples of one or more target items and state values, estimates feature importance that contributed to the label when predicting the label from one or more target items and state values, and outputs the target item, state value, and label when the feature importance is equal to or greater than a first predetermined condition; a second machine learning engine that uses a pre-trained language model to estimate the validity likelihood of the label for the target item and state value input from the first machine learning engine, and if the validity likelihood is equal to or less than a second predetermined condition, resets the state value to a pseudo value, outputs the target item, the pseudo value, and the label, and causes the first machine learning engine to retrain; A program that causes a computer to function.

2. The first machine learning engine is XAI (Explainable AI), Each target item and state value becomes a feature, Labels are a judgmental element, Feature importance is an explanatory factor 2. The program according to claim 1, which causes a computer to function as follows.

3. The state value is binary, The second machine learning engine resets the state value to a pseudo value that is the inverse of the state value input from the first machine learning engine.

3. The program according to claim 1, wherein the program causes a computer to function as described above.

4. The state value is a numeric value, The second machine learning engine resets the state value to a pseudo-value that is a statistic of the state values ​​of multiple other samples.

3. The program according to claim 1, wherein the program causes a computer to function as described above.

5. The statistical value of the state value is the mean, median, or mode.

5. The program according to claim 4, which causes a computer to function as follows.

6. the first predetermined condition in the first machine learning engine is a predetermined threshold value of feature importance; The second predetermined condition in the second machine learning engine is a predetermined threshold of the likelihood of validity.

3. The program according to claim 1, wherein the program causes a computer to function as described above.

7. The second machine learning engine uses a language corpus and is based on a graph model in which target item words and label words are treated as nodes, and the relationships between the nodes are linked by state values.

3. The program according to claim 1, further causing a computer to function as follows:

8. the second machine learning engine is based on a natural language model trained with natural language sentences labeled with correct labels and natural language sentences labeled with incorrect labels; natural language sentence conversion means for converting the target items, state values, and labels output from the first machine learning engine into natural language sentences and inputting them to the second machine learning engine; 3. The program according to claim 1, further causing a computer to function.

9. The natural language sentences with correct labels are sentences from a language corpus, A natural language sentence with an incorrect label is one in which some words in the natural language sentence with the correct label have been replaced with other words.

9. The program according to claim 8, further causing a computer to function as follows:

10. a confirmation flag assigning means for assigning a confirmation flag to an annotator for the labeled sample whose state value has been reset to the pseudo value, which is input from the second machine learning engine; and further enhance its functionality. The confirmation flag assigning means outputs the target item, the state value or the pseudo value, and the label to which the confirmation flag has been assigned to the user interface, and prompts the annotator to answer whether the answer is correct or incorrect.

3. The program according to claim 1, wherein the program causes a computer to function as described above.

11. The first machine learning engine actively selects samples for which labels should be predicted from a group of unlabeled samples by uncertainty sampling.

3. The program according to claim 1, further causing a computer to function as follows:

12. a first machine learning engine that is trained using teacher data in which labels are assigned to samples of one or more target items and state values, estimates feature importance that contributed to the label when predicting the label from one or more target items and state values, and outputs the target item, state value, and label when the feature importance is equal to or greater than a first predetermined condition; a second machine learning engine that uses a pre-trained language model to estimate the validity likelihood of the label for the target item and state value input from the first machine learning engine, and if the validity likelihood is equal to or less than a second predetermined condition, resets the state value to a pseudo value, outputs the target item, the pseudo value, and the label, and causes the first machine learning engine to retrain; A learning device comprising:

13. The learning device is a first step of using a first machine learning engine to train with teacher data in which labels are assigned to samples of one or more target items and state values, estimating feature importance that contributed to the label when predicting the label from one or more target items and state values, and outputting the target item, state value, and label when the feature importance is equal to or greater than a first predetermined condition; a second step of using a second machine learning engine to estimate the validity likelihood of the label for the target item and state value input from the first machine learning engine using a pre-trained language model, and if the validity likelihood is equal to or less than a second predetermined condition, resetting the state value to a pseudo value, outputting the target item, the pseudo value, and the label, and retraining the first machine learning engine; A learning method characterized by repeatedly executing the above.

Images