DETECTION METHOD, DETECTION APPARATUS, AND PROGRAM

Abstract

This detection method according to one embodiment involves execution by a computer of: a meta-learning procedure for training a common model, which is a machine learning model that is common among a plurality of tasks, by using a first learning data set configured from the learning data of said plurality of tasks, which include a target task which is the task to be targeted; and a detection procedure for detecting data having a label which has not been assigned to the learning data included in a second learning data set as out-of-distribution data, by using the trained common model, a second learning data set configured from the target task learning data, and a target task-specific model which is a machine learning model for the target task.

Description

[Technical field]

[0001] The present invention relates to a detection method, a detection device, and a program. [Background technology]

[0002] When predicting classes (or classifying classes) using a machine learning model, predictions regarding data of classes not included in the training data are often significantly incorrect. For this reason, out-of-distribution data detection techniques (e.g., Non-Patent Document 1), which are techniques for detecting data of classes not included in the training data, are important. The out-of-distribution data detection technique makes it possible to detect data that may be significantly incorrect and to process the data separately. In addition, the out-of-distribution data detection technique can also detect abnormal data and check the quality of collected data. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Liang, Shiyu, Yixuan Li, and R. Srikant. "Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks." International Conference on Learning Representations. 2018. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional out-of-distribution data detection methods have a problem in that they require a large amount of training data.

[0005] An embodiment of the present invention has been made in consideration of the above points, and has an object to detect out-of-distribution data with high accuracy. [Means for solving the problem]

[0006] In order to achieve the above object, a detection method according to one embodiment includes a meta-learning procedure in which a first training data set consisting of training data for a plurality of tasks including a target task, which is a target task, is used to train a common model, which is a machine learning model common to the plurality of tasks, and a detection procedure in which data having a label not assigned to training data included in the second training data set is detected as out-of-distribution data, using the trained common model, a second training data set consisting of training data for the target task, and a target task-specific model, which is a machine learning model for the target task, executed by a computer. Effect of the Invention

[0007] Out-of-distribution data can be detected with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a detection device according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a detection device according to the present embodiment. [Diagram 3] 11 is a flowchart illustrating an example of a meta-learning process according to the present embodiment. [Figure 4] 10 is a flowchart illustrating an example of a detection process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An embodiment of the present invention will be described below. In this embodiment, a detection device 10 will be described that can detect out-of-distribution data in a target task with high accuracy even when there is only a small amount of training data for the target task. Here, the training data refers to data represented by a pair of a feature of a case and a label indicating to which class the feature belongs.

[0010] The detection device 10 according to this embodiment has a meta-learning period in which meta-learning is performed using learning data of multiple tasks including a target task, and a detection period in which, when data (feature values) of a target task is given, it is determined whether the feature values ​​are out-of-distribution data.

[0011] During meta-learning, a model for each task (hereinafter also referred to as a task-specific model), a model common to each task (hereinafter also referred to as a common model), and a learning data set for meta-learning are

[0012]

number

[0013]

number

[0014] During detection, we use the common model learned during meta-learning, the task-specific model for the target task t', and the training data set D t' ={(x t'n ,y t'n )|n=1,...,N t'} and the feature x to be judged as out-of-distribution data (or a set of such features to be judged {x n |n=1, ,N}, where N is the number of features. Here, the training data set D for the target task t' is given. t' The number of training data included in N t' is assumed to be relatively small (i.e., smaller than the number of training data required by conventional out-of-distribution data detection methods).

[0015] As each task-specific model (machine learning model), any model is given depending on the task, but in the following, as an example, it is assumed that an arbitrary density distribution model is given.

[0016] <Hardware Configuration of the Detection Device 10> An example of a hardware configuration of a detection device 10 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the detection device 10 according to this embodiment has an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0017] The input device 101 is, for example, a keyboard, a mouse, a touch panel, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the detection device 10 does not necessarily have to include at least one of the input device 101 and the display device 102, for example.

[0018] The external I / F 103 is an interface with an external device such as a recording medium 103a. The detection device 10 can read and write data from and to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), a SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0019] The communication I / F 104 is an interface for connecting the detection device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as a hard disk drive (HDD) or a solid state drive (SSD). The processor 108 is an arithmetic device such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0020] The detection device 10 according to this embodiment has the hardware configuration shown in Fig. 1 and is therefore capable of implementing meta-learning processing and detection processing, which will be described later. Note that the hardware configuration shown in Fig. 1 is merely an example, and the hardware configuration of the detection device 10 is not limited thereto. For example, the detection device 10 may have multiple auxiliary storage devices 107 and multiple processors 108, or may have various other hardware in addition to the hardware shown in the figure.

[0021] <Functional configuration of the detection device 10> FIG. 2 shows an example of a functional configuration of the detection device 10 according to this embodiment. As shown in FIG. 2, the detection device 10 according to this embodiment has a meta-learning unit 201, a matching unit 202, a termination determination unit 203, and a detection unit 204. Each of these units is realized, for example, by a process in which one or more programs installed in the detection device 10 are executed by the processor 108. The detection device 10 according to this embodiment also has a storage unit 205. The storage unit 205 is realized, for example, by the auxiliary storage device 107. Note that the storage unit 205 may be realized, for example, by a storage device (such as a database server) connected to the detection device 10 via a communication network.

[0022] During meta-learning, the meta-learning unit 201 uses a learning data set D for meta-learning to learn a common model so as to improve the performance of detecting out-of-distribution data.

[0023] The adaptation unit 202 trains a task-specific model so as to adapt to a certain training data set (a support set, described later) during both meta-learning and detection.

[0024] The termination determination unit 203 determines whether or not a termination condition for the iterations for learning the common model is satisfied during meta-learning.

[0025] During detection, the detection unit 204 detects out-of-distribution data using the common model learned during meta-learning and the task-specific model for the target task t'.

[0026] During meta-learning, the storage unit 205 stores at least the common model (and its parameters) to be learned, the task-specific model (and its parameters) for each task, and a learning data set D for meta-learning. In addition, during detection, the storage unit 205 stores the learned common model (and its parameters) and the learning data set D for the target task t'. t'The storage unit 205 stores at least the task specific model (and its parameters) of the target task t', and the feature (or a set of features) to be used for determining whether the data is out-of-distribution data. In addition to these, the storage unit 205 may store various other data, such as intermediate calculation results during meta-learning or detection.

[0027] <Meta-learning processing> The meta-learning process according to this embodiment will be described with reference to FIG.

[0028] First, the meta-learning unit 201 initializes a parameter Φ of a common model (hereinafter also referred to as a common model parameter Φ) (step S101). Note that the common model parameter Φ may be initialized by any method (for example, random initialization, initialization according to a certain distribution, etc.).

[0029] Next, the meta-learning unit 201 randomly selects a task t ∈ {1, . . . , T} (step S102).

[0030] Next, the meta-learning unit 201 calculates the support set S ⊂ D for the task t selected in step S102 above. t In other words, the meta-learning unit 201 randomly samples the learning data set D for the task t selected in step S102 above. t From some training data (x tn ,y tn ) are randomly sampled, and a set of these training data is defined as a support set S. The number of samples of training data may be determined randomly or may be a predetermined number.

[0031] Next, the meta-learning unit 201 selects the in-distribution query set Q of the task t selected in step S102 above. I In other words, the meta-learning unit 201 randomly samples the learning data set D for the task t selected in step S102 above. t The set of features contained in {x tn |n=1,...,Nt} to some features x tn We randomly sample,and define the set of features as a set of queries in the distribution,Q,. I The number of samplings of the feature amounts may be determined randomly or may be a predetermined number.

[0032] Next, the meta-learning unit 201 selects the out-of-distribution query set Q for the task t selected in step S102 above. O (step S105). That is, the meta-learning unit 201 randomly samples the learning data set D\D t We randomly sample some features from the set of features contained in and classify the set of features into an out-of-distribution query set Q O The number of samplings of the feature amounts may be determined randomly or may be a predetermined number.

[0033] Next, the adaptation unit 202 estimates parameters of the task specific model for the task t selected in the above step S102 so as to adapt to the support set S obtained in the above step S103 (step S106). Hereinafter, the parameters of the task specific model are referred to as Θ. Note that since a density distribution model is assumed as the task specific model, the parameters can be estimated by, for example, maximum likelihood estimation, and a specific example will be described later.

[0034] Next, the meta-learning unit 201 uses the task-specific model for the task t selected in step S102 above to I ∪Q O The out-of-distribution score is calculated (step S107).

[0035] Here, the out-of-distribution score of feature x when support set S and common model parameters Φ are given is defined as u(x|S;Φ). The out-of-distribution score u(x|S;Φ) takes a high value if the class of feature x is not included in the classes represented by the labels of each piece of training data that constitutes support set S, and takes a low value if the class is included. As will be described later, at the time of detection, the value of this out-of-distribution score is used to determine whether or not a given feature is out-of-distribution data. An example of such an out-of-distribution score u(x|S;Φ) is the negative log-likelihood in the latent space,

[0036]

number

[0037]

number

[0038] Therefore, the meta-learning unit 201 calculates, for each x∈Q, an out-of-distribution score u(x|S;Φ) shown in the above formula (1), for example.

[0039] Here, when the negative log-likelihood is used as the out-of-distribution score u(x|S;Φ), any density distribution model can be used as the task-specific model. Therefore, for example, when a Gaussian mixture distribution is used as the task-specific model, in the above step S106, the adaptation unit 202 can calculate the parameters shown in the following formula (2) by maximum likelihood estimation.

[0040]

number

[0041] Next, the meta-learning unit 201 evaluates the out-of-distribution scores calculated in step S107 above (that is, the out-of-distribution scores for the query set Q) (step S108).

[0042] Here, for example, AUC (Area under the ROC curve) can be used as an evaluation criterion for out-of-distribution scores (in other words, a criterion for evaluating the detection performance of out-of-distribution data using out-of-distribution scores). In this case, the AUC for the query set Q (more precisely, the AUC for the query set Q when the parameters ^Θ(S;Φ) estimated in the above step S106 and the common model parameters Φ are given) is given by

[0043]

number

[0044] Next, the meta-learning unit 201 updates the common model parameters Φ so as to improve the detection performance of out-of-distribution data (step S109).

[0045] For example, the common model parameter Φ is learned so as to maximize the expected value of AUC for the query set Q. That is, the common model parameter Φ is learned by the following formula (4).

[0046]

number

[0047] Next, the end determination unit 203 determines whether a predetermined end condition is satisfied (step S110). Examples of the end condition include the number of repetitions of steps S102 to S109 being equal to or greater than a predetermined threshold, the amount of change in the objective function to be optimized (i.e., the expected value E in the above formula (4)) being equal to or less than a predetermined threshold, the amount of change in the model common parameter Φ being equal to or less than a predetermined threshold, etc.

[0048] If it is not determined in step S110 that the predetermined termination condition is satisfied, the process returns to step S102. Thus, steps S102 to S110 are repeatedly executed until it is determined that the termination condition is satisfied. On the other hand, if it is determined in step S110 that the predetermined termination condition is satisfied, the meta-learning process is terminated. In this case, a trained common model is obtained.

[0049] <Detection process> The detection process according to this embodiment will be described with reference to FIG.

[0050] The adaptation unit 202 is a training data set D t' is set as a support set S, and the task-specific model of the target task t' is adapted to the support set S (step S201). That is, similar to step S106 in Fig. 3, the adaptation unit 202 estimates parameters ^Θ(S;Φ) of the task-specific model of the target task t' so as to adapt to the support set S, where Φ is a parameter of the trained common model.

[0051] Next, the detection unit 204 calculates the out-of-distribution score u(x|S:Φ) of the feature quantity x to be determined as being out-of-distribution data (step S202). That is, the detection unit 204 calculates the out-of-distribution score u(x|S:Φ) by, for example, the above formula (1). Note that the set of features {x n |n=1, ,N}, for n=1, ,N, we define the out-of-distribution score u(x n |S:Φ) can be calculated separately.

[0052] Then, the detection unit 204 judges whether or not the corresponding feature is out-of-distribution data from the out-of-distribution score calculated in the above step S202 (step S203). Specifically, for example, if the out-of-distribution score u(x|S:Φ) is equal to or greater than a predetermined threshold, the feature x is judged to be out-of-distribution data, and if not, it is judged not to be out-of-distribution data. As a result, the feature x whose out-of-distribution score u(x|S:Φ) is equal to or greater than a predetermined threshold is detected as out-of-distribution data.

[0053] <Evaluation> The results of evaluating the detection device 10 according to this embodiment using handwritten characters are shown in Table 1 below.

[0054] [Table 1] In the above Table 1, the proposed method is the detection device 10. On the other hand, the existing method is a prototypical network, which is an existing meta-learning method. In addition, AUC was used as the evaluation index. As shown in the above Table 1, it can be seen that the detection device 10 according to the present embodiment achieves a higher AUC than the existing method.

[0055] <Summary> As described above, the detection device 10 according to the present embodiment uses a training data set of a plurality of tasks including a target task to learn parameters of a common model common to these plurality of tasks through meta-learning. Then, the detection device 10 according to the present embodiment can detect out-of-distribution data of a target task from given features using the trained common model and a task-specific model adapted to the training data set of the target task. Moreover, even if the training data set of the target task contains only a small amount of training data, it can detect out-of-distribution data with higher accuracy than existing methods.

[0056] Although the above describes a case where out-of-distribution data of a target task is detected from a given feature amount, the detection device 10 according to the present embodiment may execute various processes based on the detection result. For example, when the out-of-distribution data represents abnormal data, control processes such as stopping the abnormal device or switching to another device may be executed.

[0057] The present invention is not limited to the above-described embodiments specifically disclosed, and various modifications and variations, combinations with known technologies, and the like are possible without departing from the scope of the claims. [Explanation of symbols]

[0058] 10. Detection Device 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 201 Meta-Learning Department 202 Conformity 203 End Judgment Section 204 Detection unit 205 Storage section

Claims

1. a meta-learning procedure for learning a common model, which is a machine learning model common to the plurality of tasks, using a first learning data set composed of learning data of a plurality of tasks including a target task that is a target task; a determination step of determining whether or not given data is out-of-distribution data using the trained common model, a second training data set composed of training data for the target task, and a target task-specific model which is a machine learning model for the target task; The computer executes The determination procedure includes: A detection method comprising: calculating an out-of-distribution score for the data; determining that the data is out-of-distribution data if the out-of-distribution score is equal to or greater than a predetermined threshold; and determining that the data is not out-of-distribution data if the out-of-distribution score is less than the threshold.

2. The detection method includes: an adaptation step of training a task-specific model, which is a machine learning model of the task, using training data of each task included in the plurality of tasks; The meta-learning procedure comprises: calculating the out-of-distribution score for each of the one or more first data and the one or more second data using one or more first data for a task selected from the plurality of tasks, one or more second data for a task other than the one task, and the trained task-specific model for the one task; The method of claim 1 , further comprising training the common model such that the out-of-distribution scores improve the performance of detecting the out-of-distribution data.

3. The out-of-distribution score is the negative log-likelihood in the latent space, The meta-learning procedure comprises: Calculating the expected AUC as a measure of performance for detecting out-of-distribution data; The method of claim 2 , wherein the common model is trained to maximize an expected value of the AUC.

4. the task specific model is a predetermined density distribution model; The adaptation procedure comprises: The method of claim 3 , further comprising training the task-specific model by estimating parameters of the task-specific model by maximum likelihood estimation.

5. The adaptation procedure comprises: training the target task-specific model using the second training data set; The determination procedure includes: Calculating the out-of-distribution score for a given data set using the trained target task-specific model; The method of claim 2 , further comprising determining that the data is out-of-distribution data if the out-of-distribution score is equal to or greater than the threshold.

6. a meta-learning unit configured to learn a common model, which is a machine learning model common to the plurality of tasks, using a first learning data set composed of learning data of a plurality of tasks including a target task that is a target task; a determination unit configured to determine whether or not given data is out-of-distribution data by using the trained common model, a second training data set composed of training data for the target task, and a target task-specific model which is a machine learning model for the target task; and having The determination unit is A detection device that calculates an out-of-distribution score for the data, and determines that the data is out-of-distribution data if the out-of-distribution score is equal to or greater than a predetermined threshold, and determines that the data is not out-of-distribution data if the out-of-distribution score is less than the threshold.

7. A program for causing a computer to execute the detection method according to any one of claims 1 to 5.

Images