Model evaluation device and model evaluation method

Abstract

To provide a novel technique for evaluating a model.SOLUTION: A model evaluation device 2000 acquires a first data set 20 containing a plurality of pieces of normal data, and a second data set 30 containing a plurality of pieces of abnormal data. The model evaluation device 2000 divides the first data set 20 into a training data set and a test data set, and trains a model 10 by using the training data set. The model evaluation device 2000 acquires output data from the trained model 10 by inputting each piece of normal data contained in the test data set into the trained model 10. The model evaluation device 2000 acquires output data by inputting each piece of abnormal data contained in the second data set 30 into the trained model 10. The model evaluation device 2000 calculates evaluation data of the model 10 by using each piece of output data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a model evaluation device and a model evaluation method. [Background technology]

[0002] Techniques for evaluating models that perform inference have been developed. For example, Patent Literature 1 discloses a technique for evaluating a model generated by supervised learning using receiver operating characteristic analysis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-042669 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 does not mention the evaluation of a model generated by unsupervised learning. The disclosure has been made in consideration of the above-mentioned problems, and one of the purposes is to provide a new technique for evaluating a model. [Means for solving the problem]

[0005] The model evaluation device provided by the present disclosure includes an acquisition unit that acquires a first dataset including a plurality of normal data and a second dataset including a plurality of abnormal data, and an evaluation unit that evaluates a model using the first dataset and the second dataset. The evaluation unit divides the first dataset into a training dataset and a test dataset, trains the model using the training dataset, and calculates evaluation data of the model based on output data obtained from the trained model by inputting each of the normal data included in the test dataset to the trained model, and output data obtained by inputting each of the abnormal data to the trained model.

[0006] The present disclosure provides a model evaluation method that is executed by a computer, and includes an acquisition step of acquiring a first data set including a plurality of normal data and a second data set including a plurality of abnormal data, and an evaluation step of evaluating a model using the first data set and the second data set. In the evaluation step, the first data set is divided into a training data set and a test data set, the model is trained using the training data set, and evaluation data for the model is calculated based on output data obtained from the trained model by inputting each of the normal data included in the test data set to the trained model, and output data obtained by inputting each of the abnormal data to the trained model. Effect of the Invention

[0007] In accordance with the present disclosure, new techniques for evaluating models are provided. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of an overview of a model evaluation device. [Diagram 2] FIG. 2 is a block diagram illustrating a functional configuration of a model evaluation device. [Diagram 3] FIG. 2 is a block diagram illustrating a hardware configuration of a computer that realizes the model evaluation device. [Figure 4] 11 is a flowchart illustrating a flow of a process executed by the model evaluation device. [Diagram 5] 11 is a flowchart illustrating the flow of a process executed by an evaluation unit. [Figure 6] 13 is a second flowchart illustrating the flow of the process executed by the evaluation unit. [Figure 7] 11 is a flowchart illustrating a flow in which a model is evaluated for each of a plurality of configurations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are given the same reference numerals, and duplicated explanations are omitted as necessary for clarity of explanation. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage unit accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or more arbitrary number of storage devices.

[0010] <Summary> Fig. 1 is a diagram illustrating an example of an overview of the model evaluation device 2000. Fig. 1 is a diagram conceptually illustrating an example of the operation of the model evaluation device 2000 to facilitate understanding of the model evaluation device 2000, and the operation of the model evaluation device 2000 is not limited to that shown in Fig. 1.

[0011] The model evaluation device 2000 trains a model 10 and evaluates the trained model 10. To this end, the model evaluation device 2000 acquires a first data set 20 and a second data set 30. The first data set 20 includes a plurality of normal data, which is normal data. The second data set 30 includes a plurality of abnormal data, which is abnormal data.

[0012] The model evaluation device 2000 divides the first data set 20 into a training data set and a test data set. The model evaluation device 2000 uses the training data set to train the model 10. Here, the model 10 is trained to determine whether input data is normal data or abnormal data. For example, the model 10 is trained to perform the operation of "outputting 1 when normal data is input, and outputting 0 when abnormal data is input."

[0013] Only normal data, and no abnormal data, is used to train the model 10. Therefore, the method in which the model evaluation device 2000 trains the model 10 is classified as unsupervised learning.

[0014] After the model 10 is trained using the training dataset, the model evaluation device 2000 uses the test dataset and the second dataset 30 to evaluate the trained model 10. Specifically, the model evaluation device 2000 obtains output data from the model 10 by inputting each normal data included in the test dataset to the model 10. The model evaluation device 2000 also obtains output data from the model 10 by inputting each abnormal data included in the second dataset 30 to the model 10. Then, the model evaluation device 2000 calculates evaluation data representing an evaluation of the model 10 based on each output data obtained from the model 10.

[0015] <Examples of effects> According to the model evaluation device 2000, both normal data and abnormal data are used to evaluate the model 10 trained by unsupervised learning. Thus, according to the present disclosure, a new technique for evaluating a model trained by unsupervised learning is provided.

[0016] Not using anomalous data to train the model 10 has the advantage that, for example, the amount of anomalous data that can be obtained may be small compared to the amount of normal data that can be obtained, since the normal data is used for both training and evaluation, whereas the anomalous data is not used for training.

[0017] The opportunities to obtain abnormal data are considered to be fewer than the opportunities to obtain normal data. For example, as described below, assume that normal data and abnormal data are data that represent the state of an observation target such as a machine. In this case, the period during which the state of the observation target is abnormal is considered to be shorter than the period during which the state of the observation target is normal. Therefore, the opportunities to obtain abnormal data are fewer than the opportunities to obtain normal data.

[0018] In this way, when the opportunities to obtain abnormal data are fewer than the opportunities to obtain normal data, it is preferable to be able to tolerate a situation in which the number of abnormal data is smaller than the number of normal data in the evaluation of the model 10. In this regard, the model evaluation device 2000 uses normal data for both training and evaluation of the model 10, but does not use abnormal data for training the model 10. Therefore, the model evaluation device 2000 can be used even in a situation in which the number of abnormal data is smaller than the number of normal data.

[0019] It is also possible to use only normal data for both training and evaluation of the model 10. In other words, the model 10 is trained using only normal data, and then evaluated using only normal data.

[0020] However, in this method, the model 10 is evaluated only from the perspective of "whether or not the input data can be correctly determined to be normal data." Therefore, the model 10 is not evaluated from the perspective of "whether or not the input data can be correctly determined to be abnormal data." Therefore, a model 10 that is highly evaluated in this method may not be able to correctly determine abnormal data.

[0021] In this regard, the model evaluation device 2000 uses abnormal data to evaluate the model 10. Therefore, the model 10 can be evaluated not only from the perspective of "whether or not it is possible to correctly determine that normal data is input as normal data" but also from the perspective of "whether or not it is possible to correctly determine that abnormal data is input as abnormal data". Therefore, the model 10 can be evaluated more accurately.

[0022] The model evaluation device 2000 of this embodiment will be described in more detail below.

[0023] <Example of functional configuration> 2 is a block diagram illustrating an example of a functional configuration of a model evaluation device 2000. In this example, the model evaluation device 2000 includes an acquisition unit 2020 and an evaluation unit 2040. The acquisition unit 2020 acquires a first data set 20 and a second data set 30. The evaluation unit 2040 evaluates the model 10 using the first data set 20 and the second data set 30.

[0024] Specifically, the evaluation unit 2040 divides the first data set 20 into a training data set and a test data set. The evaluation unit 2040 trains the model 10 using the training data set. The evaluation unit 2040 obtains output data from the trained model 10 by inputting each normal data included in the test data set to the trained model 10. The evaluation unit 2040 also obtains output data from the trained model 10 by inputting each abnormal data included in the second data set 30 to the trained model 10. The evaluation unit 2040 calculates evaluation data for the model 10 based on each obtained output data.

[0025] <Example of hardware configuration> Each functional component of the model evaluation device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the model evaluation device 2000 is realized by a combination of hardware and software.

[0026] 3 is a block diagram illustrating a hardware configuration of a computer 500 that realizes the model evaluation device 2000. For example, the computer 500 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 500 is a portable computer such as a smartphone or a tablet terminal. Alternatively, the computer 500 is realized by a semiconductor chip such as a SoC (System on Chip). The computer 500 may be a dedicated computer designed to realize the model evaluation device 2000, or may be a general-purpose computer.

[0027] For example, by installing a specific application on the computer 500, each function of the model evaluation device 2000 is realized by the computer 500. The application is configured with a program for realizing each functional component of the model evaluation device 2000. The method of acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD (Digital Versatile Disk) or a USB (Universal Serial But) memory) in which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device in which the program is stored.

[0028] The computer 500 has a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. The bus 502 is a data transmission path for the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 to transmit and receive data to and from each other. However, the method of connecting the processor 504 and other components to each other is not limited to a bus connection.

[0029] The processor 504 is one of various processors such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 506 is a main storage device realized using a random access memory (RAM) or the like. The storage device 508 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0030] The input / output interface 510 is an interface for connecting the computer 500 to an input / output device, and the network interface 512 is an interface for connecting the computer 500 to a network. The network may be a LAN (Local area network) or a WAN (Wide area network).

[0031] The storage device 508 stores programs (programs for realizing the above-mentioned applications) for realizing each functional component of the model evaluation device 2000. The processor 504 reads the programs into the memory 506 and executes them to realize each functional component of the model evaluation device 2000. The storage device 508 may function as a storage unit for storing values ​​and information used by the model evaluation device 2000.

[0032] The model evaluation device 2000 may be realized by one computer 500, or may be realized by multiple computers 500. In the latter case, the multiple computers 500 may have the same hardware configuration as each other, or may have different hardware configurations from each other.

[0033] <Processing flow> 4 is a flowchart illustrating a process executed by the model evaluation device 2000. The acquisition unit 2020 acquires the first data set 20 and the second data set 30 (S102). The evaluation unit 2040 evaluates the model 10 using the first data set 20 and the second data set 30 (S104).

[0034] FIG. 5 is a flowchart illustrating a process flow executed by the evaluation unit 2040. The evaluation unit 2040 divides the first data set 20 into a training data set and a test data set (S202). The evaluation unit 2040 trains the model 10 using the training data set (S204). The evaluation unit 2040 inputs each normal data included in the test data set to the trained model 10 to obtain output data from the model 10 (S206). The evaluation unit 2040 inputs each abnormal data included in the second data set 30 to the trained model 10 to obtain output data from the model 10 (S208). The evaluation unit 2040 calculates evaluation data for the model 10 using each output data obtained from the trained model 10 (S210). Note that S206 and S208 may be performed in the reverse order to that shown in FIG. 5, or may be performed in parallel.

[0035] <Regarding normal and abnormal data> Normal data and abnormal data are, for example, data related to a specific observation target. In this case, for example, normal data is data obtained by observing an observation target in a normal state. On the other hand, abnormal data is data obtained by observing an observation target in an abnormal state.

[0036] The observation target is, for example, various machines and devices. For example, the observation target is assumed to be the blades of a wind turbine generator. In this case, for example, audio data obtained by recording the sound emitted from the blades in a normal state can be used as normal data. On the other hand, audio data obtained by recording the sound emitted from a blade in an abnormal state (for example, a blade with a hole) can be used as abnormal data. Note that, instead of observation data such as audio data, feature quantities obtained from the observation data may be used as normal data and abnormal data.

[0037] The normal data and abnormal data are not limited to data obtained by actually observing the observation target, and may be data obtained by simulation or data augmentation. For example, abnormal data may be generated by adding various noises to normal data.

[0038] For example, it is known that a wind power generator blade with a hole generates a whistling sound. Therefore, for example, by superimposing a whistling sound on the sound data of the sound emitted from the wind power generator blade in a normal state, it is possible to artificially generate the sound data of the sound emitted from the wind power generator blade in an abnormal state.

[0039] The observation target is not limited to a machine or a device. For example, the observation target may be various living organisms. For example, assume that the observation target is a human being. In this case, for example, video data of a person behaving normally can be treated as normal data, while video data of a person behaving abnormally can be treated as abnormal data.

[0040] <About Model 10> The model 10 is trained to determine whether input data is normal data or abnormal data. Normal data is used for training the model 10. Therefore, for example, the model 10 is trained to determine whether "the input data belongs to the same group as the data used for training." Data that belongs to the same group as the input data is normal data. On the other hand, data that belongs to a different group from the input data is abnormal data. As a model that can be trained in this way, for example, a machine learning model such as a one-class support vector machine or a k-nearest neighbor model can be used.

[0041] For example, as described above, it is assumed that the blades of a wind power generator are the object of observation. It is also assumed that the normal data and the abnormal data are audio data of sounds emitted from the blades in a normal state and audio data of sounds emitted from the blades in an abnormal state, respectively. In this case, the model 10 is trained to determine whether the input audio data represents sounds emitted from the blades in a normal state or not.

[0042] Here, the abnormal data is data that does not belong to the same group as the data used for training, and therefore can be called an outlier. Therefore, the process performed by the model 10 can be described as outlier detection.

[0043] <Acquisition of the first data set 20: S102> The acquisition unit 2020 acquires the first data set 20 and the second data set 30 (S102). There are various methods for acquiring the first data set 20 and the second data set 30. For example, the first data set 20 and the second data set 30 are stored in advance in a storage unit accessible from the model evaluation device 2000. In this case, the acquisition unit 2020 accesses this storage unit to acquire the first data set 20 and the second data set 30.

[0044] Alternatively, for example, the first data set 20 and the second data set 30 may be transmitted to the model evaluation device 2000 from a device other than the model evaluation device 2000. In this case, the acquisition unit 2020 acquires the first data set 20 and the second data set 30 by receiving the first data set 20 and the second data set 30 transmitted from the other device.

[0045] The first data set 20 and the second data set 30 may be obtained by the same method or by different methods.

[0046] <Model 10 Evaluation: S104> The evaluation unit 2040 evaluates the model 10 using the first data set 20 and the second data set 30 (S104). The evaluation of the model 10 will be described in detail below.

[0047] <<Splitting the Data Set: S202>> The evaluation unit 2040 divides the first data set 20 into a training data set and a test data set (S202). There are various methods for dividing the first data set 20 into a training data set and a test data set. For example, the evaluation unit 2040 divides the first data set 20 into a predetermined number of partial data sets. Here, it is assumed that the number of partial data sets is previously determined as N. It is also assumed that the number of data sets used as test data sets is previously determined as M, where N>M. Instead of determining the number of partial data sets used as test data sets, the number of partial data sets used as training data sets may be determined.

[0048] For example, the evaluation unit 2040 extracts any M partial datasets from the N partial datasets and treats each of the extracted partial datasets as a test dataset. The evaluation unit 2040 also treats the remaining (NM) partial datasets as training datasets. Alternatively, for example, the evaluation unit 2040 may first determine the (NM) partial datasets to be treated as training datasets, and treat the remaining M partial datasets as test datasets.

[0049] As will be described in detail later, there may be a plurality of ways to divide the first data set 20 into a training data set and a test data set. Therefore, the evaluation unit 2040 may evaluate the model 10 for each of the plurality of ways.

[0050] <<Training Model 10: S204>> The evaluation unit 2040 uses the training data set to train the model 10 (S204). Specifically, the evaluation unit 2040 inputs normal data included in each of one or more training data sets to the model 10. The evaluation unit 2040 trains the model 10 by updating trainable parameters (in other words, updatable parameters) of the model 10 using output data obtained from the model 10.

[0051] <Calculation of evaluation value: S210> The evaluation unit 2040 calculates evaluation data for the model 10 using each output data obtained from the trained model 10 (S210). The evaluation data for the model 10 indicates, for example, one or more evaluation values ​​related to the accuracy of the model 10.

[0052] Here, various evaluation values ​​can be used to indicate the accuracy of the model. For example, the evaluation unit 2040 calculates the accuracy defined by the following formula (1) as the evaluation value of the model 10.

number

[0053] In formula (1), TP is the proportion of data determined to be normal data among the normal data input to the trained model 10. TN is the proportion of data determined to be abnormal data among the abnormal data input to the trained model 10. FP is the proportion of data determined to be normal data among the abnormal data input to the trained model 10. FN is the proportion of data determined to be abnormal data among the normal data input to the trained model 10.

[0054] Alternatively, for example, the evaluation unit 2040 may calculate the recall defined by the formula (2) as the evaluation value of the model 10.

number

[0055] In equation (2), Er represents the recall rate. TP and FN in equation (2) are the same as TP and FN in equation (1), respectively.

[0056] The evaluation unit 2040 may calculate an evaluation value such as a correct answer rate using all output data obtained from the trained model 10, or may calculate an evaluation value such as a correct answer rate for each data set. In the latter case, for example, the evaluation unit 2040 calculates an evaluation value of the model 10 for each of a plurality of test data sets. The evaluation value of the model 10 calculated for a certain test data set is calculated using output data obtained by inputting each normal data included in the test data set to the trained model 10.

[0057] When an evaluation value is calculated for each data set, the evaluation unit 2040 further calculates an evaluation value of the model 10 for the second data set 30. The evaluation value of the model 10 calculated for the second data set 30 is calculated using output data obtained by inputting each abnormal data included in the second data set 30 to the trained model 10.

[0058] Here, the evaluation unit 2040 may divide the second data set 30 into a plurality of partial data sets and calculate an evaluation value of the model 10 for each partial data set. The evaluation value of the model 10 calculated for a partial data set obtained from the second data set 30 is calculated using output data obtained by inputting each abnormal data included in the partial data set to the trained model 10.

[0059] The size of each partial data set obtained by dividing the second data set 30 is arbitrary. For example, the size of this partial data set is set to the same size as the test data set.

[0060] The evaluation value calculated for each dataset is the true positive rate or true negative rate defined by the following formula (3).

number

[0061] When the data set i is a normal data set, the true positive rate is calculated as an evaluation value using TP[i] and FN[i]. TP[i] represents the number of data in the normal data set i that is determined to be normal data by the trained model 10. FN[i] represents the number of data in the normal data set i that is determined to be abnormal data by the trained model 10.

[0062] When the data set i is an abnormal data set, the true positive rate is calculated as an evaluation value using TN[i] and FP[i]. TN[i] represents the number of data in the abnormal data set i that is determined to be abnormal data by the trained model 10. FP[i] represents the number of data in the abnormal data set i that is determined to be normal data by the trained model 10.

[0063] The evaluation unit 2040 may calculate the variance of the evaluation value calculated for each data set. In this case, the variance of the evaluation value calculated for each data set can also be treated as an evaluation value related to the accuracy of the model 10. The variance of a certain evaluation value can be expressed, for example, by the variance or standard deviation of the evaluation value.

[0064] Here, the variance of the evaluation value may be calculated separately for the data set of normal data and the data set of abnormal data. For example, assume that there are M test data sets, and U partial data sets are generated from the second data set 30. In this case, the evaluation unit 2040 calculates the variance of the evaluation value calculated for each of the M test data sets as the variance of the evaluation value of the model 10 regarding the treatment of normal data. Also, the evaluation unit 2040 calculates the variance of the evaluation value calculated for each of the U partial data sets as the variance of the evaluation value of the model 10 regarding the treatment of abnormal data.

[0065] The evaluation unit 2040 calculates evaluation data for the model 10 using one or more evaluation values ​​calculated for the model 10. When only one evaluation value is calculated for the model 10, the evaluation unit 2040 treats the evaluation value as the evaluation data.

[0066] When multiple evaluation values ​​are calculated for the model 10, for example, the evaluation unit 2040 calculates evaluation data indicating the multiple evaluation values. Alternatively, for example, the evaluation unit 2040 may use the multiple evaluation values ​​to calculate an evaluation value (hereinafter, a comprehensive evaluation value) that indicates a comprehensive evaluation of the model 10. The comprehensive evaluation value can be expressed, for example, as a weighted sum of the evaluation values ​​calculated for the model 10. The weight for each evaluation value is assumed to be determined in advance.

[0067] The evaluation data may further include TP, TN, FP, and FN used in calculating the evaluation score. In this case, for example, the evaluation data includes a confusion matrix composed of TP, TN, FP, and FN.

[0068] <Case of dividing the first data set 20 in multiple ways> As described above, there may be multiple ways to divide the first data set 20 into a training data set and a test data set. For example, assume that the first data set 20 is divided into N partial data sets. Also assume that the number of partial data sets (M described above) treated as a test data set is 1. In this case, the evaluation unit 2040 divides the N partial data sets into (N-1) training data sets and one test data set. There are N ways to divide the data in this way.

[0069] Therefore, the evaluation unit 2040 may evaluate the model 10 for each of the multiple ways of dividing the data. Fig. 6 is a second flowchart illustrating the flow of the process performed by the evaluation unit 2040. In the example of Fig. 6, it is assumed that there are a total of K ways of dividing the first data set 20.

[0070] S302 to S314 are loop processing L1 that is executed for each of the K ways of dividing the first data set 20. In S302, the evaluation unit 2040 determines whether or not the loop processing L1 has been executed for all of the K ways of dividing.

[0071] If the loop process L1 has already been executed for all K division methods, the evaluation unit 2040 ends the process in Fig. 6. On the other hand, if there is a division method that has not yet been the target of the loop process L1, the evaluation unit 2040 selects one of the division methods that has not yet been the target of the loop process L1.

[0072] In S304, the evaluation unit 2040 divides the first data set 20 into a training data set and a test data set in the division method selected in S302. The evaluation unit 2040 trains the model 10 using the training data set obtained by the division in S304 (S306). The evaluation unit 2040 inputs each normal data included in the test data set obtained by the division in S304 to the trained model 10 to obtain output data from the model 10 (S308). The evaluation unit 2040 inputs each abnormal data included in the second data set to the trained model 10 to obtain output data from the model 10 (S310). The evaluation unit 2040 calculates evaluation data for the model 10 using each output data obtained from the trained model 10 (S312). Since S314 is the end of the loop process L1, the evaluation unit 2040 executes S302 again.

[0073] The evaluation unit 2040 may use the evaluation data generated for each of the multiple division methods to calculate overall evaluation data for the model 10. For example, the evaluation unit 2040 calculates an average value of the evaluation values ​​calculated for each of the multiple division methods. The evaluation unit 2040 may further calculate the variation (such as variance or standard deviation) of the evaluation values ​​calculated for each of the multiple division methods.

[0074] <Evaluation of Model 10 for each of several configurations> For example, the model evaluation device 2000 can be used to specify a configuration to be adopted (e.g., an optimal configuration) from among multiple configurations of the model 10. The configuration of the model 10 is represented by, for example, a combination of values ​​of multiple hyperparameters that are set for the model 10.

[0075] The model evaluation device 2000 evaluates the model 10 for each of a plurality of configurations of the model 10. Then, based on the evaluation data obtained for each configuration, the configuration of the model 10 to be adopted is specified. For example, when the evaluation of the model 10 is expressed by an overall evaluation value, the evaluation unit 2040 specifies the configuration with the maximum overall evaluation value as the configuration of the model 10 to be adopted. Alternatively, for example, when the evaluation of the model 10 is expressed by the variation in the evaluation value, the evaluation unit 2040 may specify the configuration with the minimum variation in the evaluation value of the model 10 as the configuration of the model 10 to be adopted.

[0076] 7 is a flowchart illustrating a flow in which a plurality of configurations are evaluated for each of the model 10. In the example of FIG. 7, W configurations, from a first configuration to a Wth configuration, are prepared in advance as evaluation targets.

[0077] S402 to S406 constitute a loop process L2. The loop process L2 is executed for each of the W configurations of the model 10. In S402, the evaluation unit 2040 judges whether or not the loop process L2 has already been executed for all of the W configurations. If the loop process L2 has already been executed for all of the W configurations, the evaluation unit 2040 ends the processing in FIG. 7. On the other hand, if there is a configuration that has not yet been the target of the loop process L2, the evaluation unit 2040 selects one of the configurations that has not yet been the target of the loop process L2.

[0078] In S404, the evaluation unit 2040 evaluates the model 10 having the selected configuration using the first data set 20 and the second data set 30. In S404, the model 10 is evaluated, for example, according to the flow shown in Fig. 5 or 6. Since S406 is the end of the loop process L2, the evaluation unit 2040 executes S402 again.

[0079] In S408, the evaluation unit 2040 uses the evaluation data generated for each configuration to identify the configuration of the model 10 to be adopted. Note that, when the first data set 20 is divided in a plurality of ways as illustrated in FIG. 7, a plurality of evaluation data are generated for the model 10 of each configuration. Therefore, as described above, it is preferable that the evaluation unit 2040 calculates comprehensive evaluation data by using the evaluation data generated for each of the plurality of ways of dividing the first data set 20. The evaluation unit 2040 identifies the configuration of the model 10 to be adopted by comparing the comprehensive evaluation data generated for each configuration of the model 10.

[0080] 7, a method for identifying a configuration to be adopted from among multiple configurations based on evaluation data has been described, but the model evaluation device 2000 may determine multiple configurations based on evaluation data. For example, consider a case where the hyperparameter is incremented from 1, 2, . . . , to 10 to evaluate 10 configurations of the model 10 and identify a configuration of the model 10 to be adopted. In this case, the model evaluation device 2000 first obtains evaluation values ​​of five configurations in which the hyperparameter is an even number. Of these five evaluation values, it is assumed that the evaluation values ​​are good when the hyperparameter is 4 and when the hyperparameter is 6.

[0081] In this case, when the number of hyperparameters is 1 or 9, which is far from both 4 and 6, it is considered that the evaluation value of the model 10 cannot be expected to be good (the probability that the evaluation value is good is low). Therefore, the model evaluation device 2000 obtains evaluation values ​​of the model 10 only for three configurations, 3, 5, and 7, in which the hyperparameters are close to 4 or 6. In this way, when there is a noticeable tendency in a part of the range of the hyperparameters, it is possible to reduce the number of searches for hyperparameters with high evaluation values, and it is possible to reduce the amount of calculation for identifying the configuration of the model 10 to be adopted.

[0082] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0083] In the above embodiment, the program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiment. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals. [Explanation of symbols]

[0084] 10 Models 20 First Data Set 30 Second Data Set 500 Computers 502 Bus 504 Processor 506 Memory 508 Storage Devices 510 Input / Output Interface 512 Network Interface 2000 model evaluation device 2020 Acquisition Department 2040 Evaluation Department

Claims

1. An acquisition unit that acquires a first data set including a plurality of normal data and a second data set including a plurality of abnormal data; An evaluation unit that evaluates a model using the first data set and the second data set, The evaluation unit is Splitting the first data set into a training data set and a test data set; training the model utilizing the training data set; A model evaluation device that calculates evaluation data for the model based on output data obtained from a trained model by inputting each of the normal data included in the test data set to the trained model, and output data obtained by inputting each of the abnormal data to the trained model.

2. The model evaluation device according to claim 1 , wherein the evaluation unit trains the model so as to determine whether input data is data belonging to the same group as data used for the training.

3. The evaluation unit is calculating one or more evaluation values ​​for the model using each of the output data; Calculating the evaluation data based on the calculated evaluation value; The model evaluation device according to claim 1 , wherein the evaluation value includes a precision rate, a recall rate, a true positive rate, or a true negative rate.

4. The evaluation unit is Dividing the second data set into a plurality of partial data sets; calculating, for each of the partial data sets, a first evaluation value of the model using the output data obtained for that partial data set; The model evaluation device according to claim 3 , wherein a value representing a variance in the first evaluation value is calculated as a second evaluation value of the model.

5. acquiring a first data set including a plurality of normal data and a second data set including a plurality of abnormal data; and evaluating a model using the first data set and the second data set; In the evaluation step, Splitting the first data set into a training data set and a test data set; training the model utilizing the training data set; A model evaluation method executed by a computer, which calculates evaluation data for the model based on output data obtained from a trained model by inputting each of the normal data included in the test data set to the trained model, and output data obtained by inputting each of the abnormal data to the trained model.

Images