Program of active learning giving label to sample data, apparatus therefor, and method thereof

The active learning program efficiently accumulates labeled data for all classes by using a machine learning engine and knowledge conditions to select relevant samples, addressing inefficiencies in existing techniques and reducing user burden.

JP2024165052AActive Publication Date: 2024-11-28KDDI CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023080878
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-28
Estimated Expiration
2043-05-16

AI Technical Summary

Technical Problem

Existing active learning techniques face challenges in efficiently accumulating labeled sample data for all classes, especially in a cold start scenario where class distribution is unbalanced or classification tasks are difficult, leading to inefficient data selection and biased classification models.

Method used

An active learning program and device that utilizes a machine learning engine with a class classification model, knowledge condition storage, and empty class specification to quickly accumulate labeled data by searching for sample data with predetermined similarity to knowledge conditions, and transitioning from random sampling to uncertainty sampling through confirmation flags and user interaction.

Benefits of technology

Enables rapid accumulation of labeled data for all classes, reducing user workload and improving classification model performance by minimizing the number of required sample data selections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165052000001_ABST
    Figure 2024165052000001_ABST
Patent Text Reader

Abstract

To provide a program and so on which can give a label to sample data in order to rapidly accumulate labeled sample data for all classes of classification tasks.SOLUTION: A program according to the present invention has: a machine learning engine having a class classification model trained using sample data as an explanatory variable and a label as an objective variable; knowledge condition storing means for storing a knowledge condition of each feature amount of each label corresponding to a class; empty class specifying means for specifying an empty class if labeled sample data accumulating means has empty classes with less than a predetermined number of labeled sample data; sample data retrieval means for, by using the knowledge condition storing means, retrieving, from a non-labeled sample data group, non-labeled sample data having a predetermined similarity or more with the knowledge condition of a part of or the whole of feature amount j of the label corresponding to the empty class; and confirmation flag giving means for giving a confirmation flag as an active learning to the retrieved non-labeled sample data.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an active learning technique for assigning labels to sample data. [Background technology]

[0002] With the development of machine learning technologies such as deep learning, the performance of classification tasks, regression tasks, and decision-making tasks in AI (Artificial Intelligence) has improved dramatically. On the other hand, in order to improve the accuracy of classification tasks, a large amount of labeled sample data is required as training data for machine learning engines. The task of creating training data by labeling sample data is usually done manually, which poses a problem of high labeling costs.

[0003] In a cold start, when there is no or very little labeled sample data, arbitrary unlabeled sample data is selected and labels are assigned by querying the user (or annotator). Conventional methods for selecting sample data in a cold start include, for example, the "random sampling method" and the "diversity sampling method." In the random sampling method, unlabeled sample data is randomly selected from a group of unlabeled sample data, and the user is asked to assign a label to the sample data. In the diversity sampling method, unlabeled sample data is clustered using a cluster-based algorithm such as K-means, sample data is selected uniformly from each cluster, and the user is asked to assign a label. Here, the selected sample data can be made diverse.

[0004] There is also a technology that uses a pre-trained model (BERT (registered trademark)) trained as self-supervised learning (see, for example, Non-Patent Document 1). According to this technology, the loss value of a pre-trained model of unlabeled sample data is used to select sample data with different scales, and the user is prompted to assign a label. Here too, it is possible to provide diversity to the selected sample data.

[0005] In response to this, "active learning" technology, which allows users to minimize the amount of sample data to which they assign labels, has attracted attention. With active learning, the performance of the classification model of a machine learning engine can be efficiently improved even if the amount of sample data to which users assign labels is small. At the stage where a small amount of labeled sample data has been accumulated, there is a technology called "uncertainty sampling method" that uses output information of a classification model of a machine learning engine as active learning (see, for example, Non-Patent Document 2). According to this technology, a machine learning engine for class classification is trained in advance using a small amount of labeled sample data, and sample data with the lowest confidence of all classes is "actively" selected from a group of unlabeled samples.

[0006] This allows the user to label only the selected sample data, reducing the overall workload of the user. The labeled sample data is used as teacher data for retraining by the machine learning engine. In this way, the machine learning engine repeats the selection of sample data, labeling by the user, and retraining of the classification model as active learning. This is a simple and effective method of active learning, and can efficiently improve the performance of the classification model of the machine learning engine.

[0007] Furthermore, there is a technology for active learning that uses the output data of an explainable AI as an index for selecting new sample data (see, for example, Patent Document 1). According to this technology, the importance of each of a plurality of element data for the calculation of the prediction result indicated by the output data of the explainable AI is identified corresponding to each of a plurality of input data. Based on the multiple importances corresponding to the multiple element data included in each of the identified multiple input data, target input data to which correct answer data indicating the correct answer to a predetermined event is assigned is selected from the multiple input data. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2023-048357 A [Non-patent literature]

[0009] [Non-Patent Document 1] Michelle Yuan, et. al. “Cold-start Active Learning through Self-supervised Language Modeling”, [online], [Retrieved April 14, 2023], Internet<URL:https: / / aclanthology.org / 2020.emnlp-main.637 / > [Non-Patent Document 2] Anita Krishnakumar “Active Learning Literature Survey”, [online], [Retrieved April 14, 2023], Internet <URL:https: / / www.researchgate.net / publication / 228971426_Active_Learning_Literature_Survey> Summary of the Invention [Problem to be solved by the invention]

[0010] According to the above-mentioned Random Sampling method and the technology of Non-Patent Document 1, even at a stage where labels corresponding to some classes of a classification task have not yet been assigned, such as in a cold start, it is possible to select unlabeled sample data with randomness and diversity. However, when the class distribution is unbalanced (for example, the proportion of sample data of a certain class is 0.01%) or when the processing of the classification task is relatively difficult (for example, when sample data of a class with a low proportion is very similar to sample data of a class with a high proportion), it may be necessary to repeatedly select a large amount of sample data, like a blind treasure hunt.

[0011] In addition, according to the technology of Non-Patent Document 2, active learning is premised on starting from a state in which a certain amount of labeled sample data is accumulated and labeled sample data is given to all classes of the classification task. The machine learning engine trains the classification model with at least a small amount of labeled sample data for all classes. However, if there is a bias in the small amount of labeled sample data, the classification model of the machine learning engine will also be biased.

[0012] That is, in the cold start, techniques such as the Random sampling method can be applied, and in the state where labeled sample data for all classes of the classification task has been accumulated, techniques such as the Uncertainty sampling method can be applied. In contrast, if the uncertainty sampling method is applied to a situation where there is an empty class for a classification task for which no labeled sample data has been accumulated, the classification performance of the empty class will always be 0, and the performance of the task classification model will stagnate. For this reason, the inventors of the present application considered that, after a cold start, it is necessary to quickly accumulate labeled sample data for all classes in the classification task before switching from a random sampling method or the like to an uncertainty sampling method.

[0013] Therefore, an object of the present invention is to provide an active learning program, device, and method for assigning labels to sample data in order to quickly accumulate labeled sample data for all classes of a classification task. [Means for solving the problem]

[0014] According to the present invention, there is provided a program for causing a computer to function in such a manner that, for a plurality of sample data Xi={xi,1, xi,2, . . . , xi,N} each consisting of a vector of a plurality of feature quantities j (j=1 to N), a group of labeled sample data is used to assign labels to a group of unlabeled sample data, the program comprising: A labeled sample data storage means for storing the labeled sample data Xi; A machine learning engine having a class classification model that uses a labeled sample data accumulation means to train sample data Xi as explanatory variables and labels Yk (k = 1 to L) as objective variables; A knowledge condition storage means for storing a knowledge condition for each feature j for each label Yk corresponding to a class; an empty class identifying means for identifying an empty class in which the number of labeled sample data is less than a predetermined number in the labeled sample data storage means; a sample data search means for searching, from a group of unlabeled sample data, for unlabeled sample data that has a predetermined similarity or more with the knowledge conditions of a part or all of the feature quantities j of the label Yclass_target corresponding to the empty class, using a knowledge condition storage means; A confirmation flag assignment means for assigning a confirmation flag as active learning to the retrieved unlabeled sample data; The present invention is characterized in that it causes a computer to function.

[0015] According to another embodiment of the program of the present invention, The confirmation flag assigning means outputs the unlabeled sample data to which the confirmation flag has been assigned to a user interface, and allows a user to assign a label Yk to the unlabeled sample data; The labeled sample data storage means stores labeled sample data to which a label Yk has been assigned by a user. It is also preferable to function the computer in such a manner.

[0016] According to another embodiment of the program of the present invention, a knowledge condition updating means for outputting to a user interface items of knowledge conditions of a part of feature quantities j in a label Yclass_target corresponding to an empty class, and updating the knowledge condition storage means with the knowledge conditions input by a user; It is also preferable to make the computer function as follows.

[0017] According to another embodiment of the program of the present invention, In the knowledge condition storage means, a knowledge condition is stored for the label Yclass_target corresponding to an empty class, but in the labeled sample data storage means, there is an empty class in which the number of labeled sample data is less than a predetermined number. The sample data search means selects unlabeled sample data that is most similar to the knowledge condition of part or all of the feature quantity j of the label Yclass_target corresponding to the empty class. It is also preferable to function the computer in such a manner.

[0018] According to another embodiment of the program of the present invention, If the knowledge condition storage means does not store a knowledge condition for the label Yclass_target corresponding to the empty class, The sample data search means searches the labeled sample data storage means for unlabeled sample data having a feature quantity that is the least similar to the feature quantities of the plurality of labeled sample data. It is also preferable to function the computer in such a manner.

[0019] According to another embodiment of the program of the present invention, The method further comprises a progress management means for managing a progress stage of creating the labeled sample data, The progress management method is a first phase in which the total number of labeled sample data is less than a predetermined number; a second phase in which the total number of labeled sample data is equal to or greater than a predetermined number and there is an empty class in which the number of labeled sample data is less than a predetermined number; a third phase in which the total number of labeled sample data is equal to or greater than a predetermined number and there is no empty class in which the number of labeled sample data is less than a predetermined number; We manage In the second phase, the empty class identification means, the sample data search means, and the confirmation flag assignment means are executed. It is also preferable to function the computer in such a manner.

[0020] According to another embodiment of the program of the present invention, In the first phase, the user is asked to assign labels to unlabeled sample data as active learning using either random sampling or diversity sampling. In the third phase, we label the unlabeled sample data using an uncertainty sampling method. It is also preferable to function the computer in such a manner.

[0021] According to another embodiment of the program of the present invention, A feature vector conversion means for converting sensor data from a plurality of sensors into sample data Xi consisting of vectors of a plurality of feature amounts j; It is also preferable to make the computer function as follows.

[0022] According to another embodiment of the program of the present invention, The feature vector conversion means divides the time series events detected by the multiple sensors into multiple sensor data by a sliding window, and converts the sensor data into sample data Xi. It is also preferable to function the computer in such a manner.

[0023] According to another embodiment of the program of the present invention, The sensor is placed in the room. The labels correspond to the recognition of people's activities in the room. It is also preferable to function the computer in such a manner.

[0024] According to the present invention, in an active learning device that assigns labels to a group of unlabeled sample data using a group of labeled sample data for a plurality of sample data Xi={xi,1, xi,2, . . . , xi,N} consisting of vectors of a plurality of feature quantities j (j=1 to N), A labeled sample data storage means for storing the labeled sample data Xi; A machine learning engine having a class classification model that uses a labeled sample data accumulation means to train sample data Xi as explanatory variables and labels Yk (k = 1 to L) as objective variables; A knowledge condition storage means for storing a knowledge condition for each feature j for each label Yk corresponding to a class; an empty class identifying means for identifying an empty class in which the number of labeled sample data is less than a predetermined number in the labeled sample data storage means; a sample data search means for searching, from a group of unlabeled sample data, for unlabeled sample data that has a predetermined similarity or more with the knowledge conditions of a part or all of the feature quantities j of the label Yclass_target corresponding to the empty class, using a knowledge condition storage means; A confirmation flag assignment means for assigning a confirmation flag as active learning to the retrieved unlabeled sample data; The present invention is characterized by having the following.

[0025] According to the present invention, there is provided an active learning method for a device that assigns labels to a group of unlabeled sample data using a group of labeled sample data for a plurality of sample data Xi={xi,1, xi,2, . . . , xi,N}, each of which is a vector of a plurality of feature quantities j (j=1 to N), comprising: The device is a labeled sample data storage unit that stores labeled sample data Xi; A machine learning engine having a class classification model that uses a labeled sample data storage unit to train sample data Xi as explanatory variables and labels Yk (k = 1 to L) as objective variables; A knowledge condition storage unit that stores knowledge conditions for each feature j for each label Yk corresponding to a class; having a first step of identifying an empty class in which the number of labeled sample data is less than a predetermined number in the labeled sample data storage unit; A second step of searching for unlabeled sample data that has a predetermined similarity or more with the knowledge conditions of some or all of the feature quantities j of the label Yclass_target corresponding to the empty class from the group of unlabeled sample data using a knowledge condition storage unit; The third step is to flag the retrieved unlabeled sample data as a confirmation for active learning. The present invention is characterized by carrying out the following steps. Effect of the Invention

[0026] According to the active learning program, device, and method of the present invention, labels can be added to sample data in order to rapidly accumulate labeled sample data for all classes of a classification task. [Brief description of the drawings]

[0027] [Figure 1] FIG. 2 is a functional configuration diagram of an active learning device according to the present invention. [Diagram 2] FIG. 1 is an explanatory diagram illustrating a process of phase 1. [Diagram 3] FIG. 11 is an explanatory diagram showing the processing of phase 3. [Figure 4] FIG. 11 is an explanatory diagram showing the processing of phase 2 in the present invention. [Diagram 5] 3 is an example of a program code according to the present invention. [Figure 6] 1 is an example of a user's home in which multiple sensors are arranged. [Figure 7] FIG. 1 is an explanatory diagram showing a group of time-series events divided into sensor data. [Figure 8] FIG. 11 is an explanatory diagram illustrating the process of a feature vector conversion unit. [Figure 9] 3 is a table showing a knowledge condition storage unit in the present invention. [Figure 10] A user interface that queries the user about the behavioral state. [Figure 11] This is a user interface that queries the user for knowledge conditions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0029] FIG. 1 is a functional configuration diagram of an active learning device according to the present invention.

[0030] The active learning device 1 assigns labels to a group of unlabeled sample data for a plurality of sample data Xi each consisting of a vector of a plurality of feature amounts j, using a group of labeled sample data. Xi: Sample data {xi,1, xi,2, , xi,N} xi,j: The value of feature j of sample data xi i: index of sample data j: Feature index (=1 to N) N: Number of dimensions of the feature vector Yk: Labels corresponding to classes in a classification task k: label index (=1~L) L: The number of classes in the classification task Labeled sample data is a set of sample data Xi and label Yk.<Xi,Yk> The labeled sample data is removed from the unlabeled sample data storage unit 101 and stored in the labeled sample data storage unit 102.

[0031] 1, the active learning device 1 includes an unlabeled sample data storage unit 101, a labeled sample data storage unit 102, a random sampling unit 11, a knowledge condition storage unit 120, an empty class identification unit 121, a sample data search unit 122, a confirmation flag assignment unit 123, a knowledge condition update unit 124, a machine learning engine 130, an uncertainty sampling unit 13, a progress management unit 14, and a feature vector conversion unit 15. These functional components are realized by executing a program that causes a computer mounted in the device to function. The process flow of these functional components can also be understood as an active learning method of the device.

[0032] [Unlabeled sample data storage unit 101] The unlabeled sample data storage unit 101 stores a group of sample data Xi to which no label is assigned.

[0033] [Labeled sample data storage unit 102] The labeled sample data storage unit 102 stores a group of labeled sample data Xi. The labeled sample data storage unit 102 stores sample data to which a label has been assigned by a user and sample data to which a label estimated by the uncertainty sampling unit 13 has been assigned.

[0034] [Progress Management Department 14] The progress management unit 14 manages the progress stages of creating labeled sample data. First, the process of creating labeled sample data from unlabeled sample data can be considered to involve the following three phases. <Phase 1> The stage where the total number of labeled sample data is less than a predetermined number (N1) (for example, N1=10, which is a cold start) <Phase 2> The total number of labeled sample data is equal to or greater than a predetermined number (N1), and there is an empty class in which the number of labeled sample data is less than the predetermined number (for example, when the predetermined number is 1, there is no labeled sample data corresponding to the empty class). <Phase 3> The total number of labeled sample data is equal to or greater than a predetermined number (N1), and there is no empty class in which the number of labeled sample data is less than the predetermined number. According to active learning, labeled sample data is collected as the model progresses from phase 1 to phase 2 to phase 3.

[0035] <Phase 1> FIG. 2 is an explanatory diagram showing the processing of phase 1.

[0036] [Random Sampling Part 11] In Phase 1, the user is asked to assign labels to unlabeled sample data as active learning using either random sampling or diversity sampling.

[0037] According to FIG. 2, in phase 1, as a cold start, three unlabeled sample data are randomly selected from the unsampled data accumulation unit 101, and labels are assigned by the user. Unlabeled sample data (equivalent to label A) -> Label A sample data Unlabeled sample data (equivalent to label B) -> Label B sample data Unlabeled sample data (equivalent to label B) -> Label B sample data The labeled sample data storage unit 102 stores label A sample data corresponding to class a and label B sample data corresponding to class b, but does not store label C sample data corresponding to label c. According to Fig. 2, for the classification task of classes a, b, and c, the labeled sample data is classified as follows: 1 piece of labeled sample data is classified into class a, 2 pieces of labeled sample data is classified into class b, and 0 pieces of labeled sample data is classified into class c.

[0038] For example, assume that class c corresponding to label C has a low frequency (e.g., the time it takes for one occurrence is short), and therefore its proportion of all sample data is also very small (e.g., 0.1%). In this case, when using the Random sampling method, this means that approximately 1,000 trials may be required. In addition, in the case of the diversity sampling method, sample data corresponding to class c, which has a low ratio, may be included in other classes. For example, there may be cases where cluster 1 contains sample data of class c, but the majority of cluster 1 is sample data of class b. In this case, it is expected that the number of times sample data is selected to classify class c will increase.

[0039] <Phase 3> FIG. 3 is an explanatory diagram showing the processing of phase 3.

[0040] [Machine Learning Engine 130] The machine learning engine 130 has a class classification model trained with the sample data Xi as an explanatory variable and the label Yk (k=1 to L) as a response variable using the labeled sample data accumulation unit 102. The machine learning engine 130 is based on general class classification.

[0041] [Uncertainty Sampling Part 13] In phase 3, the machine learning engine 130 is used to have the user assign labels to unlabeled sample data as active learning using an uncertainty sampling method. 3, in phase 3, a large number of sample data are already labeled in the sample data storage unit 102. Here, it is assumed that labeled sample data of all labels corresponding to all classes are stored in the labeled sample data storage unit 102.

[0042] 2 and 3 described above, it is assumed that there are three types of labels, for example, labels A to C, corresponding to target classes of a classification task. Here, when the machine learning engine 130 in Fig. 3 is trained by the labeled sample data accumulation unit 102 in Fig. 2, it is assumed that, for example, two unlabeled sample data are output as follows. (Sample data 1) Label A: 0.4, Label B: 0.6, Label C: 0.0 (Sample data 2): Label A: 0.9, Label B: 0.1, Label C: 0.0 The machine learning engine 130 outputs the probability of each class. The label corresponding to the class with the highest probability is automatically assigned. According to Figures 2 and 3, the machine learning engine 130 cannot train with the label C sample data, so the probability for label C corresponding to class c is 0.0.

[0043] According to margin-sampling in the uncertainty sampling method in Figure 3, the smaller the difference between the first and second probabilities, the more difficult the sample data is for the classification model to judge. Margin for sample data 1: 0.6-0.4=0.2 Margin for sample data 2: 0.9-0.1=0.8 In this case, sample data 1 with a small probability difference is selected. The sample data selected in this way is sample data close to the classification boundaries between the classes held internally by the classification model.

[0044] <Meaning of the present invention> Here, if the classification model of the machine learning engine is trained using the labeled sample data storage unit 102 in which a sufficient number of labeled sample data is not stored in all classes, the classification boundary of class c becomes unclear. That is, if unlabeled sample data corresponding to label C is selected from unlabeled sample data, the label will be incorrectly assigned. Specifically, the class classification model at this stage outputs only the probabilities of classes a and b, and the classification boundary appears between classes a and b. The probability of class c is not output, and there is no classification boundary between classes a and c, and between classes b and c. As a result, sample data near the classification boundary of classes a and b continues to be selected. Then, many unlabeled sample data must be selected until sample data of class c is selected.

[0045] That is, before moving to Phase 3, it is sufficient if class a contains label A sample data, class b contains label B sample data, and class c contains label C sample data. If labeled sample data can be accumulated for each class by Phase 1, it is possible to move to Phase 3.

[0046] According to the present invention, active learning in a classification task can be shifted from Phase 1 to Phase 3 with as few sample data selections (i.e., user labeling) as possible.

[0047] <Phase 2> FIG. 4 is an explanatory diagram showing the processing of phase 2 in the present invention.

[0048] According to FIG. 4, the active learning device 1 realizes a "knowledge-based sampling method" in phase 2. In phase 2 in the present invention, by eliminating empty classes in which the number of labeled sample data is less than a predetermined number (for example, 1), it is possible to transition to phase 3 as quickly as possible. In other words, it is possible to switch from the Random sampling method or the Diversity sampling method from a cold start to the Uncertainty sampling method as quickly as possible. During this time, knowledge conditions related to classes are used to select sample data that is likely to correspond to an empty class from a group of unlabeled sample data.

[0049] [Knowledge condition storage unit 120] The knowledge condition storage unit 120 stores the knowledge condition of each feature amount j for each label Yk corresponding to a class. The knowledge conditions include "general knowledge" and "target environment knowledge". "General knowledge" may be knowledge information collected from websites on the Internet for each label, or statistical information (such as average values) collected in other environments. This is an initial knowledge condition that is likely to be general, and once generated, it can be reused by even third parties. The "target environment knowledge" is information that is acquired sequentially by making inquiries to the user in the current environment. However, the user is burdened with answering each time the information is acquired. For this purpose, for example, it is assumed that general knowledge that can be used without burdening the user with responses is applied initially, and the user is queried for target environmental knowledge as necessary.

[0050] Knowledge conditions include the value and range of each feature j for each class. Let condition_k,j be the knowledge condition for feature j of class k (k=1 to L). If there is no knowledge condition for feature j of class k or if it has not been obtained, let condition_k,j=null. For example, some knowledge conditions are as follows: xi,j=value1 : The value of xi,j is value1 (constant) xi,j != value1: The value of xi,j is not value1 (constant). value1 < xi,j < value2: The value of xi,j is greater than value1 (constant) and less than value2. Furthermore, a specific example of the knowledge condition storage unit 120 will be described later with reference to FIG. 10.

[0051] [Empty class identification unit 121] When there is an empty class where the number of labeled sample data in the labeled sample data storage unit 102 is less than a predetermined number, the empty class identification unit 121 identifies the empty class. When the "predetermined number" is 1, the class with 0 labeled sample data becomes the empty class. Also, when it is 2, the class with 0 or 1 labeled sample data becomes the empty class. According to FIG. 4, the empty class identification unit 121 identifies class c as the empty class.

[0052] [Sample data search unit 122] The sample data search unit 122 uses the knowledge condition storage unit 120 to search for unlabeled sample data from the group of unlabeled sample data whose knowledge conditions of some or all of the feature amounts j corresponding to the empty class have a similarity equal to or higher than a predetermined similarity.

[0053] Here, there are the following two patterns according to the knowledge condition storage unit 120. (Pattern 1) Knowledge conditions for the label Yclass_target (label C in FIG. 4) corresponding to the empty class are stored in the knowledge condition storage unit 120, but there may be a case where there is an empty class in the labeled sample data storage unit 102 where the number of labeled (label C in FIG. 4) sample data is less than a predetermined number. In this case, the sample data search unit 122 selects unlabeled sample data that is most similar to the knowledge conditions of some or all of the feature amounts j of the label Yclass_target (label C in FIG. 4) corresponding to the empty class.

[0054] (Pattern 2) There is a case where the knowledge condition storage unit 120 does not store a knowledge condition for the label Yclass_target (label C in FIG. 4) corresponding to the empty class. In this case, the sample data search unit 122 searches for unlabeled sample data having features least similar to the features of the multiple labeled sample data in the labeled sample data storage unit 102. According to Fig. 4, the unlabeled sample data to be selected is assumed to be classified into the class farthest from classes A and B.

[0055] [Confirmation flag assignment unit 123] The confirmation flag assigning unit 123 assigns a confirmation flag to the searched unlabeled sample data as active learning.

[0056] The confirmation flag assigning unit 123 outputs the unlabeled sample data to which the confirmation flag has been assigned to the user interface, and the user assigns the label Yk to the unlabeled sample data. As a result, the labeled sample data storage unit 102 stores the labeled sample data to which the user has assigned the label Yk. As shown in FIG. 4, the label C has been assigned to the sample data selected by the user.

[0057] The machine learning engine 130 then trains a classification class model using labeled sample data that has been assigned labels Yk by the user.

[0058] [Knowledge condition update section 124] The knowledge condition update unit 124 outputs the knowledge condition items of some feature quantities j in the label Yclass_target (label C in FIG. 3) corresponding to the empty class to the user interface, and updates the knowledge condition storage unit 120 with the knowledge conditions input by the user. This allows us to appropriately classify the sample data in Phase 2 by filling in the knowledge conditions of the features in the labels corresponding to the empty classes.

[0059] FIG. 5 is an example of a program code according to the present invention.

[0060] According to active learning, labeled sample data is selected as the model progresses from phase 1 to phase 2 to phase 3. According to FIG. 5, the input information is given as follows: (input) US: Unlabeled sample data class_list={1,2,…,L} : list of classes for classification task BUDGET: Maximum number of labelings phase_definition : Phase definition strategy : Sampling method to be used in each phase knowledge=condition_k,j :General knowledge

[0061] (Lines 1-3) Perform initialization processing. LS=0: Labeled sample data 0 phase=1: Phase 1 strategy: Random sampling F: Classification model is set to null.

[0062] <Phase 1: Random sampling method> (Lines 4-17) Through loop processing, sample data is selected and the class classification model is updated repeatedly as active learning. (Line 5) Randomly select sample data from the unlabeled sample data US. (Line 6) Query the user for unlabeled sample data and get the labels. (Lines 7-9) Update the unlabeled sample data US and the labeled sample data LS. (Line 10) Train a classification model F. (Line 11) When the total number of labeled sample data becomes less than a predetermined number (N1) (e.g., N1=10), Phase 1 is repeated. After the total number of labeled sample data becomes the predetermined number, Phase 2 is started and the method switches to the knowledge-based sampling method. If labeled sample data has been collected for all classes, Phase 1 has already satisfied the termination condition for Phase 2. In this case, Phase 2 is skipped and the method moves to Phase 3, where the method switches to the uncertainty sampling method.

[0063] <Phase 2: Knowledge-based sampling method> (Line 5) Select sample data using the following steps. (S1) Determine whether there is an empty class in which the total number of labeled sample data is equal to or greater than a predetermined number (N1) and the number of labeled sample data LS is less than a predetermined number (e.g., 1). The empty class is class_target (1 to L). The label of the empty class is Yclass_target. If there are multiple empty classes, select one of them (randomly) and set it as class_target. (S2) When a knowledge condition condition_class_target,j (j = 1 to N) corresponding to an empty class class_target is stored, select unlabeled sample data that is most similar to the knowledge condition of some or all of the features j of the label Yclass_target corresponding to the empty class. On the other hand, if some or all of the knowledge conditions condition_class_target,j corresponding to the empty class class_target are null (no knowledge conditions are stored), a search is made for unlabeled sample data from the group LS of labeled sample data that has features that are least similar to the features of multiple labeled sample data.

[0064] (Line 11) If the total number of labeled sample data is equal to or greater than a predetermined number (N1) and there is no empty class with less than the predetermined number of labeled sample data, switch the sample data selection method to the Uncertainty sampling method to proceed to Phase 3.

[0065] (Lines 14-16) Execute the knowledge update process. Check the following three conditions for the empty class class_target. (Condition 1) If some or all of the knowledge conditions condition_class_target,j are null (no knowledge conditions are stored) (Condition 2) If the number of sample data that matches the knowledge condition condition_class_target,j in the unlabeled sample data US is less than a certain number (for example, 1). (Condition 3) For the empty class class_target, sample data selection has already been performed a certain number of times (e.g., 5 times) or more, but sample data with labels has not been collected a certain number of times (e.g., 1). As a supplement to condition 3, when sample data is selected using the knowledge condition of the empty class class_target, there are success (sample data with the label of class_target can be obtained) and failure. Here, if failures occur multiple times, condition 3 is considered to be satisfied. If any of these three conditions is met, it is determined that "knowledge updating is necessary" and the user is asked for environment-specific knowledge conditions regarding the empty class class_target. The knowledge conditions of the empty class condition_class_target,j are updated based on the knowledge conditions obtained from the user. These are then used in the subsequent knowledge base sampling method. On the other hand, if all of these three conditions are not met, it is determined that "knowledge updating is not necessary" and the process moves on to the selection of the next sample data.

[0066] <Phase 3: Uncertainty Sampling Method> Repeatedly select sample data using the uncertainty sampling method, and terminate when the number of labeling attempts reaches BUDGET.

[0067] The present invention can also be applied to, for example, human behavior estimation in a smart home. In such applications, a large number of labeled samples are essentially required, and from the viewpoint of privacy and accuracy improvement, it is desirable for the user to assign labels themselves. In this case, the present invention can significantly reduce the burden of labeling on the user side.

[0068] FIG. 6 shows an example of a user's home in which multiple sensors are arranged.

[0069] According to FIG. 6, a wide variety of indoor sensors, such as those listed below, are placed in a user's home as IoT devices. Bedroom human sensor (ON / OFF) Temperature Sensor Living room motion sensor (ON / OFF) Bathroom occupancy sensor (ON / OFF) Kitchen motion sensor (ON / OFF) Entrance motion sensor (ON / OFF) Door sensor (ON / OFF) The sensors detect the indoor environment and the state and changes of indoor devices as events. The human presence sensor outputs ON when it detects human movement and OFF when it can no longer detect human movement. The temperature sensor periodically measures the room temperature. The door sensor outputs ON / OFF depending on whether the door is opened or closed. Each sensor then transmits the detected event to the active learning device 1.

[0070] FIG. 6 shows a time series of events input to the active learning device 1. The active learning device 1 configures events by arranging them in chronological order. A group of events in the time series can be said to be the result of actions (going out, sleeping, cooking, eating, working, taking medicine, etc.) of people staying in the room. By analyzing the patterns of these sensor events, it becomes possible to recognize people's actions.

[0071] FIG. 7 is an explanatory diagram in which a group of time-series events is divided into sensor data.

[0072] A time series of events detected by multiple sensors is divided into multiple sensor data by a sliding window. According to Fig. 7, the sensor data is divided into, for example, 10 events {x1, x2,..., x10}, and slid a predetermined number of events at a time (for example, one by one). x1 represents the oldest event, and x10 represents the newest event.

[0073] According to FIG. 7, the initial sensor data has the following data elements: x1: Living room motion sensor ON x2: Bedroom motion sensor ON x3: Bedroom motion sensor OFF x4: Living room motion sensor ON x5: Bathroom occupancy sensor ON x6: Living room motion sensor OFF x7: Temperature sensor 18 degrees x8: Kitchen motion sensor ON x9: Bathroom occupancy sensor OFF x10: Living room motion sensor ON

[0074] FIG. 8 is an explanatory diagram illustrating the process of the feature vector conversion unit.

[0075] [Feature Vector Conversion Unit 15] The feature vector conversion unit 15 converts sensor data from a plurality of sensors into sample data Xi={xi,1, xi,2, . . . , xi,N} consisting of vectors of a plurality of feature amounts j (j=1 to N). According to Figure 8, the following N=11 dimensional features are defined. 1 Time period (0~23) 2 Time difference between window data (seconds) 3. Sensor with the most detections 4. Number of detections by the human sensor 5 Door sensor detection count 6 Temperature sensor detection count 7. Number of detections by the living room motion sensor 8. Number of detections by the kitchen motion sensor 9. Bath occupancy sensor detection count 10 Number of detections by the bedroom motion sensor 11 Number of detections by the entrance motion sensor

[0076] Also, according to FIG. 8, the initial sensor data is converted into sample data consisting of 11-dimensional features as shown below. Sample data X1 = {19,300,2,9,0,1,4,1,1,2,0}

[0077] FIG. 9 is a table showing the knowledge condition storage unit in the present invention.

[0078] The labels correspond to the recognition of the behavior of people in a room. The knowledge condition storage unit 120 stores knowledge conditions for each feature amount for each label corresponding to a class. According to FIG. 9, the labels of behavior recognition corresponding to classes are six (L=6): "Going out," "Sleeping," "Cooking," "Eating," "Working," and "Taking medicine." For example, the knowledge conditions for "Sleeping" are the time period from 0:00 to 8:00, and the sensor with the highest number of detections is the bedroom motion sensor. Also, for example, the knowledge conditions for "Cooking" are the time period from 11:00 to 13:00 and from 17:00 to 20:00, and the sensor with the highest number of detections is the kitchen motion sensor. As described above, these knowledge conditions include "general knowledge" and "target environment knowledge."

[0079] The above-mentioned sample data search unit 122 searches for unlabeled sample data that has a predetermined similarity or higher with the knowledge condition of the feature amount j of the label Yclass_target corresponding to the empty class, using the knowledge condition storage unit 120. Here, the knowledge conditions of some of the feature amounts j (for example, only j=1, 3) may be compared, or the knowledge conditions of all the feature amounts j (for example, all of j=1 to N) may be compared.

[0080] FIG. 10 shows a user interface for querying the user about the behavioral state.

[0081] FIG. 10 shows an example of a screen display that asks the user to assign labels to sample data in all phases (phase 1, phase 2, and phase 3). In phase 1, the user is asked to assign labels to sample data selected by the random sampling unit 11. In phase 2, the user is asked to assign a label to the sample data to which the flag has been assigned by the confirmation flag assigning unit 123. In phase 3, the user is asked to label the sample data selected by the uncertainty sampling unit 13.

[0082] FIG. 11 shows a user interface for querying the user for knowledge conditions.

[0083] 11, the knowledge condition update unit 124 in phase 2 queries the user for the knowledge condition of feature j for the label corresponding to the empty class. For example, when the label Yclass_target corresponding to the empty class is “taking medicine”, the knowledge conditions “time period” and “location” are queried to the user. (inquiry) "Please tell us the [time of day] and [location] of your 'medication'." (User Answer) Time: 10:00~12:00 Location: Living room These knowledge conditions are stored in the knowledge condition storage unit 120. Then, they are used in the selection of sample data in the subsequent phase 2.

[0084] <Other embodiments> According to the above-described embodiment, a classification task in the application of human behavior recognition has been described. However, the present invention can of course be applied to a classification task with an imbalanced class distribution, even in the application of image classification or anomaly detection. An example of image classification is image classification of animal species. When the ratio of sample data for rare species is low and there is no labeled sample data, this can be applied to knowledge of rare species (such as color and shape characteristics). For anomaly detection, for example, in camera footage, knowledge of an abnormal state (e.g., a large number of people in the footage) can be applied when the frequency of occurrence of an abnormal state is low and there is no labeled sample data.

[0085] In addition, as an example of a scenario other than one in which the AI ​​selects sample data and the user assigns labels, there is also the case in which the user selects the samples, as shown below. (Example 1) The user searches for sample data of each class from unlabeled sample data. However, if the absolute number of unlabeled sample data is huge, it is expected that the user's load will be very high. (Example 2) The user notifies the AI ​​the moment sample data for each class is generated. In the case of behavior recognition, the user notifies the AI ​​the moment the behavior is performed, which allows labeled sample data for each class to be steadily collected. However, it is expected that there may be a high burden on the user to go about their daily life while remembering to notify the AI ​​when the behavior is performed.

[0086] As described above in detail, according to the active learning program, device, and method of the present invention, labels can be assigned to sample data in order to quickly accumulate labeled sample data for all classes of a classification task. Furthermore, according to the present invention, even in the case of a cold start, active learning in a classification task can be transitioned to active learning with as few sample data selections as possible. In other words, the number of times that a user or annotator assigns labels is reduced, and the associated work costs can be reduced.

[0087] Furthermore, this will, for example, "reduce the burden of labeling sample data on users," making it possible to contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0088] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0089] 1. Active learning device 101 Unlabeled sample data storage unit 102 Labeled sample data storage unit 11 Random Sampling Section 120 Knowledge condition storage unit 121 Empty Class Specification 122 Sample Data Search Section 123 Confirmation flag assignment section 124 Knowledge Condition Update Department 130 Machine Learning Engine 13 Uncertainty Sampling Section 14 Progress Management Department 15 Feature vector conversion unit

Claims

1. A program for causing a computer to function in such a manner that, for a plurality of sample data Xi = {xi,1, xi,2, ..., xi,N} each of which is a vector of a plurality of feature quantities j (j = 1 to N), a group of labeled sample data is used to assign labels to a group of unlabeled sample data, a labeled sample data storage means for storing the labeled sample data Xi; A machine learning engine having a class classification model that is trained using a labeled sample data storage means, with sample data Xi as explanatory variables and labels Yk (k = 1 to L) as objective variables; a knowledge condition storage means for storing a knowledge condition for each feature value j for each label Yk corresponding to a class; an empty class identifying means for identifying an empty class in which the number of labeled sample data is less than a predetermined number in the labeled sample data storage means; a sample data search means for searching, from a group of unlabeled sample data, for unlabeled sample data that has a predetermined similarity or more with the knowledge conditions of a part or all of the feature quantities j of the label Yclass_target corresponding to the empty class, using a knowledge condition storage means; A confirmation flag assignment means for assigning a confirmation flag as active learning to the retrieved unlabeled sample data; A program for causing a computer to function.

2. The confirmation flag assigning means outputs the unlabeled sample data to which the confirmation flag has been assigned to a user interface, and allows a user to assign a label Yk to the unlabeled sample data; The labeled sample data storage means stores labeled sample data to which a label Yk has been assigned by a user.

2. The program according to claim 1, which causes a computer to function as described above.

3. a knowledge condition updating means for outputting to a user interface knowledge condition items of a part of feature quantities j in a label Yclass_target corresponding to an empty class, and updating the knowledge condition storage means with the knowledge conditions input by a user; 3. The program according to claim 1, further comprising:

4. In the knowledge condition storage means, a knowledge condition is stored for the label Yclass_target corresponding to an empty class, but in the labeled sample data storage means, there is an empty class in which the number of labeled sample data is less than a predetermined number. The sample data search means selects unlabeled sample data that is most similar to the knowledge condition of part or all of the feature quantities j of the label Yclass_target corresponding to the empty class.

2. The program according to claim 1, which causes a computer to function as described above.

5. If the knowledge condition storage means does not store a knowledge condition for the label Yclass_target corresponding to the empty class, The sample data search means searches the labeled sample data storage means for unlabeled sample data having a feature quantity that is the least similar to the feature quantities of the plurality of labeled sample data.

2. The program according to claim 1, which causes a computer to function as described above.

6. The method further comprises a progress management means for managing a progress stage of creating the labeled sample data, The progress management method is a first phase in which the total number of labeled sample data is less than a predetermined number; a second phase in which the total number of labeled sample data is equal to or greater than a predetermined number and there is an empty class in which the number of labeled sample data is less than a predetermined number; a third phase in which the total number of labeled sample data is equal to or greater than a predetermined number and there is no empty class in which the number of labeled sample data is less than the predetermined number; We manage In the second phase, the empty class identification means, the sample data search means, and the confirmation flag assignment means are executed.

2. The program according to claim 1, which causes a computer to function as described above.

7. In the first phase, the user is asked to assign labels to unlabeled sample data as active learning using either random sampling or diversity sampling. In the third phase, we label the unlabeled sample data using the uncertainty sampling method.

7. The program according to claim 6, which causes a computer to function as described above.

8. A feature vector conversion means for converting sensor data from a plurality of sensors into sample data Xi consisting of vectors of a plurality of feature amounts j; 3. The program according to claim 1, further comprising:

9. The feature vector conversion means divides the time series events detected by the multiple sensors into multiple sensor data by a sliding window, and converts the sensor data into sample data Xi.

9. The program according to claim 8, which causes a computer to function in such a manner.

10. The sensor is placed in the room. The labels correspond to the recognition of people's activities in the room.

10. The program according to claim 9, further comprising:

11. In an active learning device, for a plurality of sample data Xi = {xi,1, xi,2, ..., xi,N} consisting of vectors of a plurality of feature quantities j (j = 1 to N), a label is assigned to a group of unlabeled sample data by using a group of labeled sample data, a labeled sample data storage means for storing the labeled sample data Xi; A machine learning engine having a class classification model that is trained using a labeled sample data storage means, with sample data Xi as explanatory variables and labels Yk (k = 1 to L) as objective variables; a knowledge condition storage means for storing a knowledge condition for each feature value j for each label Yk corresponding to a class; an empty class identifying means for identifying an empty class in which the number of labeled sample data is less than a predetermined number in the labeled sample data storage means; a sample data search means for searching, from a group of unlabeled sample data, for unlabeled sample data that has a predetermined similarity or more with the knowledge conditions of a part or all of the feature quantities j of the label Yclass_target corresponding to the empty class, using a knowledge condition storage means; A confirmation flag assignment means for assigning a confirmation flag as active learning to the retrieved unlabeled sample data; An active learning device comprising:

12. In an active learning method for a device that assigns labels to a group of unlabeled sample data using a group of labeled sample data for a plurality of sample data Xi = {xi,1, xi,2, ..., xi,N} consisting of vectors of a plurality of feature amounts j (j = 1 to N), the method comprises the steps of: The device is a labeled sample data storage unit that stores the labeled sample data Xi; A machine learning engine having a class classification model that uses a labeled sample data storage unit to train sample data Xi as explanatory variables and labels Yk (k = 1 to L) as objective variables; a knowledge condition storage unit that stores knowledge conditions for each feature j for each label Yk corresponding to a class; having a first step of identifying an empty class in which the number of labeled sample data is less than a predetermined number in the labeled sample data storage unit; A second step of searching for unlabeled sample data that has a predetermined similarity or more with the knowledge conditions of some or all of the feature quantities j of the label Yclass_target corresponding to the empty class from the group of unlabeled sample data using a knowledge condition storage unit; A third step of assigning a confirmation flag to the retrieved unlabeled sample data as active learning.

1. An active learning method comprising:

Citation Information

Patent Citations

  • Numerical data-oriented rare class detection method

    CN114881102A

  • Learning data creation support system and learning data creation support method

    JP2020187408A

  • Method for generating prediction results for early prediction of occurrence of fatal symptoms in a subject, and device using the same

    JP2020532025A

  • Methods and systems for mining minority-class data samples for training a neural network

    US20220188568A1

  • Information processing apparatus and information processing method

    JP2023048357A