Accuracy calculation program, accuracy calculation method, and information processing device

Abstract

On the basis of generation conditions, which are based on the correspondence between one or a plurality of feature combinations and label information, and an input data set, which is a set of input data including at least one data item, an information processing device (10) obtains, as rules to be selected, data items or data item combinations that match generation conditions. The information processing device (10) calculates the accuracy of each of a plurality of machine learning algorithms using a data set that includes only data items used by a selection rule, which is a rule to be selected among said rules to be selected that satisfies a specific condition.

Description

[Technical field]

[0001] The present invention relates to an accuracy calculation program, an accuracy calculation method, and an information processing device. [Background technology]

[0002] Known machine learning models generated by machine learning include white-box models, in which the basis for a judgment is easy to explain, and black-box models, in which the basis for a judgment is difficult to explain. For example, known machine learning algorithms used in white-box models include decision trees, random forests, and logistic regression, while known machine learning algorithms used in black-box models include neural networks and deep learning.

[0003] When using such machine learning algorithms, if accuracy is the only important factor, it is common to select from all methods, and if explainability is particularly important, it is common to use white-box methods. On the other hand, when performing machine learning using data in a table format, the efficiency and accuracy of machine learning can be improved by knowing in advance the optimal method or which machine learning algorithm will produce useful results. In recent years, various tools that perform model calculations automatically have become known. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-4509 A [Patent Document 2] JP 2020-46888 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when selecting a machine learning algorithm using the above tools, it is difficult to determine whether the model comparison is appropriate, and the selection results have low reliability.

[0006] For example, because the data types and machine learning algorithms supported by each tool are different, the optimal machine learning algorithm may not be included in the selection candidates in the first place. In addition, although it is possible to run multiple tools and make a selection, different machine learning algorithms may be judged to be the best by a small margin, but the basis for the narrow margin may be unclear, and the judgment results of each tool may not be compatible.

[0007] In one aspect, an object of the present invention is to provide an accuracy calculation program, an accuracy calculation method, and an information processing device that can optimize accuracy comparison between multiple machine learning algorithms. [Means for solving the problem]

[0008] In the first proposal, the accuracy calculation program causes the computer to execute a process of acquiring, as a selection target rule, a data item or a combination of data items that meets a generation condition based on a correspondence relationship between one or more combinations of features and label information, and an input dataset that is a collection of input data including at least one data item. The accuracy calculation program causes the computer to execute a process of calculating the accuracy of each of a plurality of machine learning algorithms using a dataset that holds only data items used in a selection rule that is a selection target rule that satisfies a specific condition among the selection target rules. Effect of the Invention

[0009] According to one embodiment, it is possible to optimize the accuracy comparison of multiple machine learning algorithms. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an information processing apparatus according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram of the information processing apparatus according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of input data stored in the input data DB. [Figure 4] FIG. 4 is a diagram for explaining the generation of rules. [Diagram 5] FIG. 5 is a diagram for explaining the selection of a rule. [Figure 6] FIG. 6 is a diagram illustrating the evaluation of a machine learning model. [Figure 7] FIG. 7 is a diagram illustrating accuracy prediction of a machine learning model. [Figure 8] FIG. 8 is a diagram illustrating an example of the output result. [Figure 9] FIG. 9 is a flowchart of the process according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining the generation of rules according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating the selection of a rule according to the second embodiment. [Figure 12] FIG. 12 is a diagram illustrating an evaluation of the machine learning model according to the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating the accuracy prediction of the machine learning model according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of an output result according to the second embodiment. [Figure 15] FIG. 15 is a diagram for explaining another example of rule selection. [Figure 16] FIG. 16 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the accuracy calculation program, the accuracy calculation method, and the information processing device according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to these embodiments. In addition, each embodiment can be appropriately combined within a range that does not cause a contradiction. EXAMPLES

[0012] [Description of information processing device] Fig. 1 is a diagram illustrating an information processing device 10 according to a first embodiment. The information processing device 10 illustrated in Fig. 1 is an example of a computer device capable of generating an evaluation index for selecting an optimal machine learning algorithm that is expected to be effective from among a plurality of machine learning algorithms when a machine learning model is to be generated by machine learning using input data in a table format or the like.

[0013] In recent years, various tools that perform model calculations automatically have become known. These automated tools do not try every hyperparameter of every machine learning algorithm, but rather use their own methods to reduce execution time, and furthermore, they limit the execution time and randomly search for parameters within that time. For this reason, it is difficult to optimize the accuracy comparison of multiple machine learning algorithms using automated tools.

[0014] Therefore, the information processing device 10 according to the first embodiment enables sufficient comparison and verification of each model by maximally covering the possibilities of parameter search during model calculation using a machine learning algorithm within a certain period of time.

[0015] 1, the information processing device 10 acquires a data item or a combination of data items that meets the generation condition based on the correspondence between one or more combinations of features and label information, as a selection target rule, based on an input dataset that is a collection of input data including at least one data item. Then, the information processing device 10 calculates the accuracy of each of the multiple machine learning algorithms using a dataset that holds only data items used in the selection rule that is a selection target rule that satisfies a specific condition among the selection target rules.

[0016] For example, the information processing device 10 extracts a combination (rule) of columns of data items corresponding to each hypothesis, which is a set of rules that explain the prediction according to the objective variable. At this time, each rule (combination of columns) can be generated using, for example, the technology of JP 2020-46888 A, and can also include the importance (weight) corresponding to each hypothesis. Note that the weight can also be calculated by logistic regression using each rule as training data.

[0017] Then, the information processing device 10 selects, as selected data, a combination of a plurality of columns that satisfies a preset specific condition from the extracted combinations of a plurality of columns. After that, the information processing device 10 executes parameter search and evaluation of each machine learning algorithm using the selected combination of columns, and outputs the execution result.

[0018] In this way, the information processing device 10 can search for a highly accurate machine learning model under the same conditions with many hypotheses, and present the results to the user.

[0019] [Function configuration] 2 is a functional block diagram showing a functional configuration of the information processing device 10 according to the first embodiment. As shown in FIG. 2, the information processing device 10 includes a communication unit 11, an output unit 12, a storage unit 13, and a control unit 20.

[0020] The communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface, etc. For example, the communication unit 11 receives various instructions from an administrator terminal, etc., and transmits execution results, etc. to the administrator terminal, etc.

[0021] The output unit 12 is a processing unit that outputs various information, and is realized by, for example, a display, a touch panel, etc. For example, the output unit 12 displays and outputs the generated evaluation result to the control unit 20, which will be described later.

[0022] The storage unit 13 is an example of a storage device that stores various data, programs, etc., and is realized by, for example, a memory, a hard disk, etc. The storage unit 13 stores an input data DB 14 and an evaluation result DB 15.

[0023] The input data DB14 is an example of a data set used for evaluating a machine learning algorithm. FIG. 3 is a diagram for explaining an example of input data stored in the input data DB14. FIG. 3 shows an example in which the label "1" is set to input data determined to be a mammal, and the label "0" is set to input data not determined to be a mammal. As shown in FIG. 3, the input data is composed of "data name, feather, egg, . . . , label". Here, the "data name" is information for identifying the input data. "Feather, egg" and the like are data items and correspond to explanatory variables, and the "label" is a data item indicating correct answer information and corresponds to a target variable indicating a class.

[0024] In this way, the input data set stored in the input data DB 14 is data in a table format with each data item such as data name, feather, label, etc. as a column, and input data (such as a value to be set) consisting of each data item as a row. Note that in this embodiment, for the sake of simplicity, each data item may be abbreviated to item A, item B, label, etc., as shown in the lower diagram of Fig. 3.

[0025] The evaluation result DB 15 is a database that stores the evaluation results of each machine learning algorithm executed by the control unit 20, which will be described later. This evaluation result DB 15 stores the evaluation results in association with the execution timestamp, the user name that requested the execution of the evaluation, information related to the input data used in the evaluation, and the machine learning algorithm to be evaluated. Note that the stored information will be described later, so a detailed description will be omitted here.

[0026] The control unit 20 is a processing unit that controls the entire information processing device 10, and is realized by, for example, a processor. The control unit 20 has a generation unit 21, a selection unit 22, an evaluation execution unit 23, and a display control unit 24. Note that the generation unit 21, the selection unit 22, the evaluation execution unit 23, and the display control unit 24 can be realized, for example, as electronic circuits included in the processor, or can be realized as processes executed by the processor.

[0027] The generating unit 21 is a processing unit that extracts, according to a generating condition based on the correspondence between one or more combinations of features and label information, a data item or a combination of data items that meets the generating condition (extraction condition) as selection target data (rule) from the input data DB 14. Here, the extraction condition can also be said to be a condition used for generating a rule.

[0028] Specifically, the generation unit 21 performs preprocessing on the input data, performing one-hot representation for categorical columns and discretization such as minimum description length principle discretization (MDLP) for numeric columns. Next, the generation unit 21 extracts (generates) rules corresponding to hypotheses from the preprocessed input data. Here, a hypothesis is a set of multiple rules, and each rule is a combination of columns. In addition, since the extracted rules (combinations of columns) differ depending on the setting of the minimum value of the sample ratio (hereinafter referred to as the column extraction parameter or simply the parameter), multiple values ​​are set between "0.0 and 1.0" and extracted.

[0029] Fig. 4 is a diagram for explaining the generation of rules. As shown in Fig. 4, the generation unit 21 extracts, from the preprocessed input data, hypothesis 1 with a parameter (column extraction parameter) set to 0.0, hypothesis 2 with a parameter set to 0.4, hypothesis 3 with a parameter set to 0.8, and hypothesis 4 with a parameter set to 1.0. Here, each hypothesis is composed of a "rule, weight." A "rule" is a combination of columns that corresponds to a hypothesis, and a "weight" is the weight of that rule in the hypothesis.

[0030] In the example of Figure 4, "A, 0.5", "B, 0.4", "C&E, 0.25", and "G, 0.1" are extracted as "rule, weights" corresponding to hypothesis 1. Also, "A, 0.5", "B, 0.4", "E, 0.25", and "G, 0.1" are extracted as "rule, weights" corresponding to hypothesis 2. "A&F, 0.5", "B, 0.4", and "C&E, 0.25" are extracted as "rule, weights" corresponding to hypothesis 3. "C&D&F, 0.5", "C&E, 0.4", and "B&F&G, 0.25" are extracted as "rule, weights" corresponding to hypothesis 4.

[0031] For example, "A, 0.5" in hypothesis 1 indicates that the column "item A" is extracted as data belonging to hypothesis 1, and the weight in hypothesis 1 is "0.5". Also, "C&E, 0.5" in hypothesis 3 indicates that the AND condition of columns "item C" and "item E" is extracted as data belonging to hypothesis 3, and the weight in hypothesis 3 is "0.5". "Item" here is not the data value itself, but an item that is a higher-level concept of data. In other words, in the example of Figure 3, for data with "feather (none)" and "egg (none)" as "label (1)", the above A corresponds to the item "feather", which is the data item name, rather than "feather (none)". Note that although "none" is used here for simplicity, a "numerical value" is set.

[0032] The generation unit 21 then outputs each hypothesis thus generated and the "rule, weight" belonging to each hypothesis to the selection unit 22. Note that, although an example of generating hypotheses and extracting rules corresponding to each hypothesis has been described here, the present invention is not limited to this. For example, the "rule, weight" can be generated in advance by an administrator or the like, or a "rule" used in past machine learning can be extracted and a weight (importance) can be calculated from the rate at which the "rule" was used.

[0033] 2, the selection unit 22 is a processing unit that selects selection data (selection rules) that satisfy a specific condition from the rules of each hypothesis generated by the generation unit 21. Specifically, the selection unit 22 extracts a set of rules that meet the specific condition "the number of columns is two or more" from the rules (combinations of columns) of each hypothesis, and outputs these rules to the evaluation execution unit 23 as selection data.

[0034] FIG. 5 is a diagram for explaining the selection of rules. As shown in FIG. 5, the selection unit 22 selects the rule "C&E" which is a combination of two columns from among the rules belonging to hypothesis 1. The selection unit 22 does not select any rules belonging to hypothesis 2 because there is no corresponding rule. Similarly, the selection unit 22 selects the rules "A&F" and "C&E" which are a combination of two columns from among the rules belonging to hypothesis 3. Furthermore, the selection unit 22 selects the rule "C&E" which is a combination of two columns, and the rules "C&D&F" and "B&F&G" which are a combination of three columns from among the rules belonging to hypothesis 4.

[0035] Next, the selection unit 22 extracts data (selected data) corresponding to the selected rule from the input data. For example, as shown in FIG. 5, for the selected rule "C&E", the selection unit 22 generates a data set in which a column (item C), a column (item E), and a column (label) are extracted from the input data. Similarly, for the rule "A&F", the selection unit 22 generates a data set in which a column (item A), a column (item F), and a column (label) are extracted from the input data. For the rule "B&F&G", the selection unit 22 generates a data set in which a column (item B), a column (item F), a column (item G), and a column (label) are extracted from the input data. For the rule "C&D&F", the selection unit 22 generates a data set in which a column (item C), a column (item D), a column (item F), and a column (label) are extracted from the input data.

[0036] The evaluation execution unit 23 is a processing unit that executes evaluation of a plurality of machine learning algorithms to be evaluated, using each data set selected by the selection unit 22. Specifically, the evaluation execution unit 23 executes machine learning using data belonging to each selected rule by all machine learning algorithms (machine learning methods), and executes evaluation by searching all parameters without optimization. Note that the parameter search executed here can adopt general machine learning.

[0037] FIG. 6 is a diagram for explaining the evaluation of the machine learning model. As shown in FIG. 6, the evaluation execution unit 23 executes machine learning using each of the machine learning algorithms X, Y, and Z with respect to the data of "item C, item E, label" corresponding to the number of columns 2, with the row data of "item C, item E, label" as training data, and obtains the accuracy of each machine learning algorithm. That is, the evaluation execution unit 23 executes machine learning with each data of "item C, item E" as explanatory variables and the data of "label" as objective variables by each machine learning algorithm, and obtains the accuracy of the machine learning model X of "0.5", the accuracy of the machine learning model Y of "0.6", and the accuracy of the machine learning model Z of "0.7".

[0038] Similarly, the evaluation execution unit 23 performs machine learning using each machine learning algorithm with “item A, item F, label” corresponding to the number of columns 2 as training data, and obtains an accuracy of “0.8” for machine learning model X, an accuracy of “0.8” for machine learning model Y, and an accuracy of “0.75” for machine learning model Z.

[0039] In addition, the evaluation execution unit 23 performs machine learning using each machine learning algorithm with “item B, item F, item G, label” corresponding to the number of columns 3 as training data, and obtains an accuracy of “0.5” for machine learning model X, an accuracy of “0.6” for machine learning model Y, and an accuracy of “0.68” for machine learning model Z.

[0040] In addition, the evaluation execution unit 23 performs machine learning using each machine learning algorithm with "item C, item D, item G, label" corresponding to the number of columns 3 as training data, and obtains an accuracy of "0.75" for machine learning model X, an accuracy of "0.5" for machine learning model Y, and an accuracy of "0.6" for machine learning model Z.

[0041] Thereafter, the evaluation execution unit 23 calculates pseudo accuracy based on the results of the machine learning and the weights of the rules, and searches for a machine learning algorithm that has high accuracy across many hypotheses.

[0042] 7 is a diagram for explaining the accuracy prediction of the machine learning model. As shown in FIG. 7, the evaluation execution unit 23 calculates the multiplication result of the accuracy of each machine learning algorithm and the weight of the rule "C&E" selected from each rule of hypothesis 1 as the pseudo accuracy. That is, the evaluation execution unit 23 calculates "0.5×0.25=0.125" as "pseudo accuracy of machine learning algorithm X=accuracy×weight", calculates "0.6×0.25=0.15" as "pseudo accuracy of machine learning algorithm Y", and calculates "0.7×0.25=0.175" as "pseudo accuracy of machine learning algorithm Z".

[0043] Similarly, the evaluation execution unit 23 calculates each pseudo-accuracy "(Accuracy × Weight of A&F) + (Accuracy × Weight of C&E)" using each weight of the selected rules "A&F" and "C&E" from among the rules of hypothesis 3. That is, the evaluation execution unit 23 calculates "0.8 × 0.5 + 0.5 × 0.25 = 0.525" as the "pseudo-accuracy of machine learning algorithm X", "0.8 × 0.5 + 0.6 × 0.25 = 0.55" as the "pseudo-accuracy of machine learning algorithm Y", and "0.75 × 0.5 + 0.7 × 0.25 = 0.55" as the "pseudo-accuracy of machine learning algorithm Z".

[0044] Similarly, the evaluation execution unit 23 calculates each pseudo-accuracy "(Accuracy x Weight of C&D&F) + (Accuracy x Weight of C&E) + (Accuracy x Weight of B&F&G)" using the weights of the selected rules "C&D&F", "C&E", and "B&F&G" from among the rules of hypothesis 4. That is, the evaluation execution unit 23 calculates "0.75 x 0.5 + 0.5 x 0.4 + 0.5 x 0.25 = 0.7" as the "pseudo-accuracy of machine learning algorithm X", "0.5 x 0.5 + 0.6 x 0.4 + 0.6 x 0.25 = 0.64" as the "pseudo-accuracy of machine learning algorithm Y", and "0.6 x 0.5 + 0.7 x 0.4 + 0.68 x 0.25 = 0.75" as the "pseudo-accuracy of machine learning algorithm Z".

[0045] The evaluation execution unit 23 judges that hypothesis 2 is a data set that can be sufficiently classified with a single column (no difference due to the model).

[0046] As a result of these, the evaluation execution unit 23 determines that a model with high accuracy in many hypotheses is an appropriate machine learning model. That is, the evaluation execution unit 23 evaluates "machine learning algorithm Z" as an appropriate model for hypothesis 1, "machine learning algorithm Y" and "machine learning algorithm Z" as an appropriate model for hypothesis 3, and "machine learning algorithm Z" as an appropriate model for hypothesis 4, and outputs the results to the display control unit 24.

[0047] The display control unit 24 is a processing unit that displays and outputs the execution results by the evaluation execution unit 23. Specifically, the display control unit 24 displays and outputs each hypothesis and the evaluation result by the evaluation execution unit 23 to the output unit 12, stores them in the storage unit 13, and transmits them to the administrator terminal.

[0048] Fig. 8 is a diagram for explaining an example of an output result. As shown in Fig. 8, the display control unit 24 generates and displays a screen including information on the optimal model that has high accuracy in many hypotheses, the basis for the optimal model, each hypothesis used to select the optimal model, the pseudo-accuracy of each machine learning model, and the like.

[0049] For example, the "optimal model" field contains "machine learning model Z" determined by the evaluation execution unit 23, and the basis field contains content such as "machine learning model Z achieved the highest pseudo accuracy for combinations (1) to (4) of columns that are correlated with the label columns of the data." Additionally, (1) to (4) each correspond to the above hypotheses 1 to 4, respectively, and the calculation results of the pseudo accuracy described above are listed under (1), etc.

[0050] [Processing flow] Fig. 9 is a flowchart showing a process flow according to the embodiment 1. As shown in Fig. 9, when a process start is instructed (S101: Yes), the generating unit 21 reads input data (S102) and generates hypotheses and rules included in the hypotheses (S103).

[0051] Then, the selection unit 22 selects a rule based on a specific condition (S104). After that, the evaluation execution unit 23 executes machine learning by each machine learning model using the selected selection data (selection rule) (S105), and executes accuracy prediction (pseudo accuracy) of each machine learning model using the machine learning result (S106). Then, the display control unit 24 outputs the result of the accuracy prediction (S107).

[0052] [effect] As described above, the information processing device 10 can perform sufficient comparison and verification of each model in the divided data by performing sufficient parameter search only on data that is meaningful for classification. In addition, the information processing device 10 can increase the user's satisfaction with the presented model by presenting the result.

[0053] For example, when using the tools mentioned above, it is nearly impossible to perform every possible parameter search for all the data, and calculation time can be reduced by using only a portion of the original data. However, the number of combinations that select a portion of the original data increases exponentially as the number of columns increases, which is not realistic. For example, if the original data has 200 columns, the number of combinations for selecting only three columns from that is (200 x 199 x 198) / (3 x 2 x 1) = 1,313,400, which is not realistic.

[0054] In contrast, the information processing device 10 according to the first embodiment does not optimize the parameter adjustment, i.e., does not execute only a part of all parameters, but narrows down the input data, thereby enabling the parameter adjustment to be executed in a full search (without optimization). As a result, the information processing device 10 can calculate and evaluate a pseudo-precision that is highly reliable, even though it is pseudo, and therefore can evaluate the machine learning model at high speed and with high precision. EXAMPLES

[0055] Incidentally, in the first embodiment, an example was described in which a set of rules that meet the specific condition "the number of columns is 2 or more" is extracted from the hypothesis rules (combination of columns), but the specific condition is not limited to this. This specific condition may be a condition for extracting rules that are included in a certain number of hypotheses from hypotheses that explain belonging to a certain class at a certain sample ratio or more. Therefore, in the second embodiment, a condition for "extracting rules that are included in 70% or more of all hypotheses" is described as another example of the specific condition. Note that the functional configuration, etc. are the same as in the first embodiment.

[0056] First, the generation unit 21 performs preprocessing on the input data, and extracts rules corresponding to hypotheses from the preprocessed input data. Fig. 10 is a diagram for explaining the generation of rules according to the second embodiment. As shown in Fig. 10, the generation unit 21 extracts, from the preprocessed input data, hypothesis 1 in which 0.0 is set as a parameter (column extraction parameter), hypothesis 2 in which 0.4 is set as a parameter, hypothesis 3 in which 0.8 is set as a parameter, and hypothesis 4 in which 1.0 is set as a parameter.

[0057] In the example of Figure 10, "A, 0.5", "B, 0.4", "C&E, 0.25", and "G, 0.1" are extracted as "rule, weights" corresponding to hypothesis 1. Also, "A, 0.5", "B, 0.4", "E, 0.25", and "G, 0.1" are extracted as "rule, weights" corresponding to hypothesis 2. "A&F, 0.5", "B, 0.4", and "C&E, 0.25" are extracted as "rule, weights" corresponding to hypothesis 3. "C&D&F, 0.5", "C&E, 0.4", and "B&F&G, 0.25" are extracted as "rule, weights" corresponding to hypothesis 4.

[0058] Next, the selection unit 22 selects selection data that satisfies a specific condition "extract rules that are included in 70% or more of all hypotheses" from the rules of each hypothesis generated by the generation unit 21. Fig. 11 is a diagram for explaining the selection of rules according to the second embodiment. As shown in Fig. 11, the selection unit 22 selects rules "B" and "C&E" that are included in three or more hypotheses ((3 / 4) x 100 = 75%) from among the rules included in each of the four hypotheses.

[0059] Then, the selection unit 22 extracts data (selected data) corresponding to the selected rule from the input data. For example, as shown in Fig. 11, the selection unit 22 generates a data set for the selected rule "B" by extracting "column (item B) and column (label)" from the input data. Also, the selection unit 22 generates a data set for the selected rule "C&E" by extracting "column (item C), column (item E), and column (label)" from the input data.

[0060] Next, the evaluation execution unit 23 executes evaluation of a plurality of machine learning algorithms to be evaluated using each data set selected by the selection unit 22. FIG. 12 is a diagram for explaining the evaluation of a machine learning model according to the second embodiment. As shown in FIG. 12, the evaluation execution unit 23 executes machine learning using each of the machine learning algorithms X, Y, and Z with the data of "item B, label" corresponding to the number of columns 1 as training data, and obtains the accuracy of each machine learning algorithm. That is, the evaluation execution unit 23 executes machine learning with the data of "item B" as an explanatory variable and the data of "label" as an objective variable by each machine learning algorithm, and obtains the accuracy of the machine learning model X of "0.15", the accuracy of the machine learning model Y of "0.3", and the accuracy of the machine learning model Z of "0.28".

[0061] Similarly, the evaluation execution unit 23 performs machine learning using each machine learning algorithm with “item C, item E, label” corresponding to the number of columns 2 as training data, and obtains an accuracy of “0.7” for machine learning model X, an accuracy of “0.2” for machine learning model Y, and an accuracy of “0.75” for machine learning model Z.

[0062] Next, the evaluation execution unit 23 calculates the pseudo accuracy based on the results of the machine learning and the weights of the rules, and searches for a machine learning algorithm that has high accuracy in many hypotheses. FIG. 13 is a diagram for explaining the accuracy prediction of the machine learning model according to the second embodiment. As shown in FIG. 13, the evaluation execution unit 23 calculates the pseudo accuracy "(Accuracy × Weight of B) + (Accuracy × Weight of C & E)" using the accuracy of each machine learning algorithm and the weights of the rules "B" and "C & E" selected from among the rules of hypothesis 1. That is, the evaluation execution unit 23 calculates "0.15 × 0.4 + 0.7 × 0.25 = 0.235" as the "pseudo accuracy of machine learning algorithm X", calculates "0.3 × 0.4 + 0.2 × 0.25 = 0.17" as the "pseudo accuracy of machine learning algorithm Y", and calculates "0.28 × 0.4 + 0.75 × 0.25 = 0.2995" as the "pseudo accuracy of machine learning algorithm Z".

[0063] Similarly, the evaluation execution unit 23 calculates each pseudo-accuracy "(accuracy × weight of B)" using the weight of rule "B" selected from among the rules of hypothesis 2. That is, the evaluation execution unit 23 calculates "0.15 × 0.4 = 0.06" as the "pseudo-accuracy of machine learning algorithm X", "0.3 × 0.4 = 0.12" as the "pseudo-accuracy of machine learning algorithm Y", and "0.28 × 0.4 = 0.112" as the "pseudo-accuracy of machine learning algorithm Z".

[0064] Furthermore, the evaluation execution unit 23 calculates the pseudo-accuracy "(Accuracy × Weight of B) + (Accuracy × Weight of C & E)" using the accuracy of each machine learning algorithm and the weights of the rules "B" and "C & E" selected from among the rules of hypothesis 3. That is, the evaluation execution unit 23 calculates "0.15 × 0.4 + 0.7 × 0.25 = 0.235" as the "pseudo-accuracy of machine learning algorithm X", "0.3 × 0.4 + 0.2 × 0.25 = 0.17" as the "pseudo-accuracy of machine learning algorithm Y", and "0.28 × 0.4 + 0.75 × 0.25 = 0.2995" as the "pseudo-accuracy of machine learning algorithm Z".

[0065] Furthermore, the evaluation execution unit 23 calculates each pseudo-accuracy "(Accuracy × Weight of C&E)" using the weight of the rule "C&E" selected from among the rules of hypothesis 4. That is, the evaluation execution unit 23 calculates "0.7 × 0.25 = 0.175" as the "pseudo-accuracy of machine learning algorithm X", "0.2 × 0.25 = 0.05" as the "pseudo-accuracy of machine learning algorithm Y", and "0.75 × 0.25 = 0.1875" as the "pseudo-accuracy of machine learning algorithm Z".

[0066] As a result, the evaluation execution unit 23 determines that a model with high accuracy in many hypotheses is an appropriate machine learning model. That is, the evaluation execution unit 23 evaluates "machine learning algorithm Z" as an appropriate model for hypotheses 1, 3, and 4, and "machine learning algorithm Y" as an appropriate model for hypothesis 2, and outputs the results to the display control unit 24.

[0067] Finally, the display control unit 24 displays and outputs the execution result by the evaluation execution unit 23. Fig. 14 is a diagram for explaining an example of the output result according to the second embodiment. As shown in Fig. 14, the display control unit 24 generates and displays a screen including information on the optimal model that achieved high accuracy with many hypotheses, the basis for the optimal model, each hypothesis used to select the optimal model, the pseudo-accuracy of each machine learning model, and the like.

[0068] For example, "optimal model" would be "machine learning model Z," and "reason" would be "machine learning model Z achieved the highest pseudo accuracy for combinations (1) to (4) of columns that are correlated with the label column of the data." Additionally, (1) to (4) each correspond to the above hypotheses 1 to 4, respectively, and the calculation results of the pseudo accuracy mentioned above are listed under (1), etc.

[0069] In addition, various conditions can be set other than the specific condition "extract rules that are included in 70% or more of all hypotheses" described in the second embodiment, so an example will be described below. For example, the specific condition can be set to "the top two most frequently included rules of all hypotheses."

[0070] Fig. 15 is a diagram for explaining another example of rule selection. As shown in Fig. 15, it is assumed that the generation unit 21 extracts hypothesis 1 with a parameter (string extraction parameter) set to 0.0, hypothesis 2 with a parameter set to 0.4, hypothesis 3 with a parameter set to 0.8, and hypothesis 4 with a parameter set to 1.0 from the input data after preprocessing.

[0071] In this state, the selection unit 22 selects, from among the rules included in each of the four hypotheses, rules that meet the specific condition "top two rules that are included most frequently among all hypotheses."

[0072] That is, for rule "A", the selection unit 22 calculates "(2 / 4) x 100 = 50%" since rule "A" is included in hypotheses 1 and 2 of the four hypotheses, and for rule "B", the selection unit 22 calculates "(3 / 4) x 100 = 75%" since rule "B" is included in hypotheses 1, 2, and 3 of the four hypotheses. For rule "C&E", the selection unit 22 calculates "(3 / 4) x 100 = 75%" since rule "C&E" is included in hypotheses 1, 2, and 3 of the four hypotheses, and for rule "G", the selection unit 22 calculates "(2 / 4) x 100 = 50%" since rule "G" is included in hypotheses 1 and 2 of the four hypotheses.

[0073] Furthermore, the selection unit 22 calculates "(1 / 4)×100=25%" for the rule "E" since it is included in hypothesis 2 of the four hypotheses, and "(1 / 4)×100=25%" for the rule "A&F" since it is included in hypothesis 3 of the four hypotheses. The selection unit 22 calculates "(1 / 4)×100=25%" for the rule "C&D&F" since it is included in hypothesis 4 of the four hypotheses, and "(1 / 4)×100=25%" for the rule "B&F&G" since it is included in hypothesis 4 of the four hypotheses.

[0074] As a result, the selection unit 22 selects the rules "B" and "C&E" as rules that meet the specific condition "top two most frequently included among all hypotheses". Note that the subsequent processing is the same as in the first and second embodiments, and therefore the description of the processing is omitted.

[0075] As described above, even if a huge number of rules are generated in the hypothesis, the information processing device 10 can appropriately narrow down the rules for accuracy evaluation by specifying a specific condition. Furthermore, since the information processing device 10 can arbitrarily set the specific condition, it is possible to provide an environment for accuracy evaluation suited to the needs of each user, such as a user who wants to use many rules with an emphasis on accuracy, or a user who uses narrowed down rules with an emphasis on speed. EXAMPLES

[0076] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments.

[0077] [Numbers, etc.] The number of hypotheses, the number of rules, the number of machine learning algorithms, the hypothesis generation technology, the rule generation technology, various numerical examples, various thresholds, information to be written on the output screen, and the like used in the above embodiment are merely examples and can be changed as desired. The weight of a rule can be, for example, a weight calculated by logistic regression, or various values ​​obtained when extracting a rule. As merely an example, a "chi-squared value" or "normalized mutual information" can be used. The "chi-squared value" is a value calculated by using an expected value (theoretical value) of the frequency of positive examples and negative examples in the entire sample, and the frequency of positive examples and negative examples in the sample that hits the rule is used as an observed value to test whether the rule and the positive examples and negative examples are independent. The "normalized mutual information" indicates what percentage of the amount of information held by the label is obtained by knowing the information on whether the rule is hit.

[0078] [system] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0079] Moreover, each component of each device shown in the figure is functionally conceptual, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure. In other words, all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. The generation unit 21 is an example of a generation unit, and the selection unit 22 and the evaluation execution unit 23 are examples of a calculation unit. Moreover, the hypothesis is an example of a generation condition, and the selection target rule is an example of a rule.

[0080] Furthermore, each processing function performed by each device may be realized, in whole or in part, by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0081] [Hardware] Next, an example of the hardware configuration of each device described in the above embodiment will be described. Since each device has the same hardware configuration, the information processing device 10 will be described here as an example. Fig. 16 is a diagram for explaining an example of the hardware configuration. As shown in Fig. 16, the information processing device 10 has a communication device 10a, a HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. Moreover, each unit shown in Fig. 16 is connected to each other by a bus or the like.

[0082] The communication device 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores programs and DBs that operate the functions shown in FIG.

[0083] The processor 10d reads out a program that executes the same processes as the respective processing units shown in FIG. 2 and the like from the HDD 10b and expands it in the memory 10c, thereby operating a process that executes the respective functions described in FIG. 2 and the like. For example, this process executes the same functions as the respective processing units possessed by the information processing device 10. Specifically, the processor 10d reads out a program having the same functions as the generation unit 21, the selection unit 22, the evaluation execution unit 23, the display control unit 24, and the like from the HDD 10b and the like. Then, the processor 10d executes a process that executes the same processes as the generation unit 21, the selection unit 22, the evaluation execution unit 23, the display control unit 24, and the like.

[0084] In this way, the information processing device 10 operates as an information processing device that executes various processing methods by reading and executing a program. The information processing device 10 can also realize functions similar to those of the above-mentioned embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing device 10. For example, the present invention can be similarly applied to cases where another computer or server executes a program, or where these execute a program in cooperation with each other.

[0085] This program can be distributed via a network such as the Internet. This program can also be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer. [Explanation of symbols]

[0086] 10. Information processing device 11 Communications Department 12 Output section 13 Storage section 14 Input Data DB 15 Evaluation result database 20 Control section 21 Generation part 22 Selection Department 23 Evaluation Execution Department 24 Display control section

Claims

1. On the computer, obtaining, from an input data set which is a set of input data having columns of at least one data item and label information, a column combination corresponding to a hypothesis which is a generation condition based on a correspondence between one or more combinations of features and the label information, the column combination being a combination of column data of the label information and column data of the data item; Among the obtained combinations of columns, a combination of columns that satisfies a specific condition set using the number of columns or the combination of columns is selected, and for each of the selected combinations of columns, machine learning is performed using a plurality of machine learning algorithms using training data in which column data of label information included in the combination of columns is used as a target variable and column data of data items is used as an explanatory variable, and the accuracy of each of the plurality of machine learning algorithms is calculated.

4. An accuracy calculation program for executing a process.

2. The input data set is a set of the input data in which the at least one data item and the label information are arranged as columns, The obtaining process obtains, from the input dataset, a combination of columns that combines column data of the label information indicating that the data item belongs to a specific class with column data of the data item that is correlated with the column data, 2. The accuracy calculation program according to claim 1, wherein the calculating process uses a combination of columns that satisfies the specific condition from among the obtained combinations of columns, and for each of the selected combinations of columns, performs machine learning with a plurality of machine learning algorithms using training data in which column data of label information included in the combination of columns is an objective variable and column data of data items is an explanatory variable, and calculates an accuracy of each of the plurality of machine learning algorithms.

3. the hypothesis being a set of rules, each rule being a combination of sequences; 3. The accuracy calculation program according to claim 2, characterized in that the calculation process obtains, as the combination of columns that satisfies the specific condition, a combination of columns that corresponds to a rule included in each of a predetermined number or more of all hypotheses determined as the generation condition.

4. 4. The accuracy calculation program according to claim 3, wherein the calculation process calculates the accuracy of each of the plurality of machine learning algorithms using a combination of columns that correspond to a rule included in each of the predetermined number or more of hypotheses.

5. 5. The accuracy calculation program according to claim 4, wherein the calculation process calculates the accuracy of each of the plurality of machine learning algorithms using combinations of columns that correspond to rules included in each of the predetermined number or more of hypotheses, the number of columns being a predetermined value or more.

6. 5. The accuracy calculation program according to claim 4, wherein the calculation process calculates the accuracy of each of the plurality of machine learning algorithms using a top predetermined number of column combinations that are included in each hypothesis at a high rate among combinations of columns that correspond to rules included in each of the predetermined number or more hypotheses.

7. Each hypothesis includes weight information indicating the importance of that hypothesis; The calculation process includes: predicting accuracy of each of the machine learning algorithms using the selected combination of columns and weight information for each of the selected combinations of columns for each of the hypotheses; The accuracy calculation program according to claim 1 , further comprising: determining an optimal machine learning model based on the accuracy predicted in each of the hypotheses.

8. 8. The accuracy calculation program according to claim 7, further comprising: causing the computer to execute a process of outputting information about the optimal machine learning model, information about each of the hypotheses, and predicted accuracy for each of the multiple machine learning algorithms for each of the hypotheses.

9. The computer obtaining, from an input data set which is a set of input data having columns of at least one data item and label information, a column combination corresponding to a hypothesis which is a generation condition based on a correspondence between one or more combinations of features and the label information, the column combination being a combination of column data of the label information and column data of the data item; Among the obtained combinations of columns, a combination of columns that satisfies a specific condition set using the number of columns or the combination of columns is selected, and for each of the selected combinations of columns, machine learning is performed using a plurality of machine learning algorithms using training data in which column data of label information included in the combination of columns is used as a target variable and column data of data items is used as an explanatory variable, and the accuracy of each of the plurality of machine learning algorithms is calculated.

4. An accuracy calculation method comprising the steps of:

10. obtaining, from an input data set which is a set of input data having columns of at least one data item and label information, a column combination corresponding to a hypothesis which is a generation condition based on a correspondence between one or more combinations of features and the label information, the column combination being a combination of column data of the label information and column data of the data item; using selected combinations of columns that satisfy a specific condition set using the number of columns or the combination of columns from among the obtained combinations of columns, and performing machine learning with a plurality of machine learning algorithms for each of the selected combinations of columns using training data in which column data of the label information included in the combination of columns is an objective variable and column data of the data items is an explanatory variable, and calculating the accuracy of each of the plurality of machine learning algorithms; An information processing device comprising a control unit.

Images