Pipeline set generation program, pipeline set generation method, and information processor

Abstract

To provide a pipeline set generation program, a pipeline set generation method, and an information processor that improve convenience of automating machine learning.SOLUTION: A pipeline set generation program causes a computer to execute processing of: acquiring, on the basis of a plurality of tasks, a first set of pipelines, each pipeline including a machine learning model; adding a specific component corresponding to each type of variable included in data of the plurality of tasks to each pipeline included in the first set of pipelines to generate a second set of pipelines; executing a second set of pipelines for a plurality of tasks to acquire an evaluation value of each pipeline included in the second set of pipelines; and selecting, on the basis of the evaluation value, a plurality of pipelines from the second set of pipelines to generate a third set of pipelines.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a pipeline set generating program, a pipeline set generating method, and an information processing device. [Background technology]

[0002] In analysis using machine learning, the machine learning algorithms, such as the machine learning model to be used and the format of the input data, vary depending on the data to be analyzed and the purpose. In addition, the prediction accuracy of the machine learning model can be improved by appropriately tuning the hyperparameters. For this reason, in the past, when performing analysis using machine learning, data processing and formatting, feature engineering, hyperparameter optimization, and machine learning model design were performed manually by machine learning experts.

[0003] Thus, to properly perform analysis using machine learning, advanced data science knowledge and techniques are required. This has made it difficult for general users to use machine learning. Therefore, research is being conducted into automating machine learning using AutoML (Auto Machine Learning), which enables analysis using machine learning without advanced techniques or knowledge. AutoML automatically processes and formats data, engineer features, and generates machine learning models.

[0004] A series of processes including multiple preprocessing steps such as data processing and formatting, feature engineering, and hyperparameter adjustment, as well as prediction using the data generated by each process and a machine learning model, is called a "pipeline." In other words, a pipeline represents a series of processes in which prediction processing using a learning device is performed after zero or more preprocessing steps. The pipeline includes hyperparameters for each process.

[0005] Moreover, the information to be predicted, which is a dataset with a target variable and an evaluation index, is called a "task." In other words, AutoML is required to determine a suitable pipeline for a specific task. However, the suitable pipeline here is a pipeline with a prediction accuracy within an acceptable range, and may not be the optimal pipeline.

[0006] In AutoML, by retaining a set of a small number of robust pipelines that have a certain degree of prediction accuracy for many tasks, it becomes possible to quickly determine an appropriate pipeline for a specific task. Therefore, retaining a set of a small number of robust pipelines that have a certain degree of prediction accuracy for many tasks as pipelines to be selected is a promising method for quickly achieving prediction performance that falls within the usable range in AutoML.

[0007] One technique for retaining a set of candidate pipelines is to run AutoML on a large number of tasks and select a subset of the suitable pipelines for each task selected by AutoML.

[0008] There are other techniques for pipeline selection, such as the following. For example, a technique has been proposed for selecting a pipeline for a new dataset based on a representative dataset in each class obtained by clustering multiple datasets, and determining a pipeline set based on a rating value indicating the performance of the selected pipeline. A technique has also been proposed for determining a pipeline set suitable for a given data set, generating a preprocessed dataset of the given data, and generating a pipeline set based on a score obtained by applying a hyperparameter set and a pipeline selected based on performance. A technique has also been proposed for ranking each pipeline to determine the price of the pipeline using a surrogate model based on a user-specified index or a combination thereof, and determining the price of each pipeline according to the rank. A technique has also been proposed for generating a first ranking set using a first feature extracted from an existing machine learning project, generating a second ranking set using a second feature generated based on the first feature, and selecting the second ranking set with the highest rank. A technique has also been proposed for generating a feature construction pipeline optimized for deployment from a feature construction pipeline constructed during training of a machine learning model, evaluating the latency requirements of data transformation operators, and creating an optimized feature construction pipeline based on an impact evaluation of the latency. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent Publication No. 2022-87842 [Patent Document 2] US Patent Application Publication No. 2022 / 0051049 [Patent Document 3] US Patent Application Publication No. 2022 / 0207444 [Patent Document 4] JP 2022-44016 A [Patent Document 5] JP 2022-159132 A Summary of the Invention [Problem to be solved by the invention]

[0010] However, there are cases where a pipeline generated by AutoML for a certain task does not include pre-processing parts for successfully executing processing for other tasks. A pipeline that does not include appropriate pre-processing parts may be excluded because it is difficult to execute other tasks in a technology that selects a pipeline set that is a subset of suitable pipelines for each task. However, in reality, even a pipeline that does not include appropriate pre-processing parts for such other tasks may be preferable to be considered as a robust pipeline. Therefore, a pipeline set obtained as a subset of a set of suitable pipelines for each task is not necessarily sufficiently robust. Therefore, it is difficult to improve the convenience of automating machine learning.

[0011] Additionally, the other pipeline selection techniques discussed above do not take into account pipelines that lack pre-processing components to perform a given task, making it difficult to properly evaluate and incorporate such pipelines into the pipeline collection of robust pipelines.

[0012] The disclosed technology has been made in consideration of the above, and aims to provide a pipeline set generation program, a pipeline set generation method, and an information processing device that improve the convenience of automating machine learning. [Means for solving the problem]

[0013] In one aspect of the pipeline set generation program, pipeline set generation method, and information processing device disclosed in the present application, a computer is caused to execute a process of obtaining a first pipeline set based on a plurality of tasks, each pipeline including a machine learning model, generating a second pipeline set by adding specific components corresponding to each type of variable included in data of the plurality of tasks to each pipeline included in the first pipeline set, obtaining evaluation values ​​of each pipeline included in the second pipeline set by executing the second pipeline set for the plurality of tasks, and generating a third pipeline set by selecting a plurality of pipelines from the second pipeline set based on the evaluation values. Effect of the Invention

[0014] In one aspect, the present invention can improve the convenience of automating machine learning. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a pipeline. [Diagram 3] FIG. 3 is a diagram showing an example of a suitable pipeline and deviation degree for each task. [Figure 4] FIG. 4 is a diagram showing an example of the deviation of the preferred pipeline for each task after modification to a complete pipeline. [Diagram 5] FIG. 5 is a diagram illustrating an example of a first selection of pipelines to be included in a pipeline set. [Figure 6] FIG. 6 is a diagram illustrating an example of the second selection of pipelines to be included in the pipeline set. [Figure 7] FIG. 7 is a flowchart of a process of generating a pipeline set of a robust pipeline by the information processing device according to the first embodiment. [Figure 8]FIG. 8 is a flowchart of a process of generating a default pre-processing component by the information processing device according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a method for generating a default pre-processing component by the information processing device according to the second embodiment. [Figure 10] FIG. 10 is a flowchart of a process of generating a default pre-processing component by the information processing device according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the pipeline set generating program, the pipeline set generating method, and the information processing device disclosed in the present application will be described in detail with reference to the drawings. Note that the pipeline set generating program, the pipeline set generating method, and the information processing device disclosed in the present application are not limited to the following embodiments. EXAMPLES

[0017] 1 is a block diagram of an information processing device. The information processing device 1 is connected to a user terminal 2 which is an input source of data and an output destination of processing results. The information processing device 1 has a control unit 10, a reception unit 11, and an output unit 12.

[0018] The reception unit 11 receives input of a task set and an AutoML program from the user terminal 2. The task set is a set of multiple tasks. The tasks are preferably similar to tasks used by the user when performing machine learning using the AutoML program. The AutoML program is a program used by the user for analysis using machine learning. The AutoML program outputs a suitable pipeline based on the input task. A suitable pipeline for a specific task is a pipeline that has a top predictive performance among pipelines that can be generated for a specific task.

[0019] 2 is a diagram showing an example of a pipeline, and the pipeline used in the following description will be described with reference to FIG.

[0020] The pipeline 200 includes components 201 to 205. The components 201 to 205 are programs that perform various processes such as conversion and prediction at each stage included in the pipeline 200. In general, the data handled by each component of the pipeline may differ. Therefore, the data handled by each component of the pipeline is called a variable. Also, information indicating the type of information that the variable is is called a "variable type." This variable may also be called a "column." The components 201 to 205 handle different data.

[0021] For example, the part 201 is a preprocessing called SimpleImputer. The type of the variable used in the part 201 is "numeric". In addition, as the part 202, one of the preprocessing is selected from OneHotEncoder and OrdinalEncoder. The type of the variable used in the part 202 is "categorical". For example, gender is an example of a variable whose type is categorical. Since the data input to the learning device is a numerical value, the part 202 converts the input data whose type is categorical into a numerical value. As the part 203, one of the preprocessing is selected from CountVectorizer and TfidfVectorizer. The type of the variable used in the part 203 is "text". The part 204 converts the input data whose type is text into a numerical value. The type of the variable used in the part 204 is "all", that is, it includes all the types used in the parts 201 to 203 up to that point. The part 204 performs processing such as normalization of the input data. The part 205 is a learning device. The component 205 handles variables whose type is a numeric value. The component 205 has a machine learning model and performs predictions using the machine learning model.

[0022] The AutoML program generates a pipeline 200 including pre-processing components corresponding to each variable, which correspond to components 201 to 204, and a learning device, which corresponds to component 205. The pre-processing components include components that perform processing on the entire data, such as component 204. However, the pipeline generated by the AutoML program does not need to include a pre-processing component that performs processing on the entire data. The AutoML program selects an optimal combination from among multiple pre-processing components and hyperparameters as a pre-processing component for each variable type.

[0023] Continuing the explanation by returning to Fig. 1, the reception unit 11 outputs the received task set and AutoML program to the execution unit 101 of the control unit 10.

[0024] Here, in this embodiment, when a suitable pipeline obtained by AutoML for a certain task is determined, the predicted performance of the target pipeline to be evaluated is expressed as a deviation corresponding to the difference from the predicted performance of the suitable pipeline. In detail, the deviation is an index showing how much the predicted performance when the target pipeline is used for a certain task deviates from the predicted performance when a suitable pipeline for the specific task is used. That is, the deviation is expressed by a formula: "deviation=(predicted performance of suitable pipeline)-(predicted performance of target pipeline)". The deviation is also sometimes called "regret".

[0025] Fig. 3 is a diagram showing an example of suitable pipelines and deviations for each task. Here, as shown in table 211 in Fig. 3, a case will be described in which tasks #1 to #8 exist and pipelines P1 to P8 are suitable pipelines for each of tasks #1 to #8.

[0026] Task #1 is a task having data whose variable type is a numeric value. Pipeline P1 is a suitable pipeline for task #1, and has a pre-processing component N1 for data whose variable type is a numeric value and a machine learning model M1 which is a learning device. The incremented symbols written on the lower side of each of the pipelines P1 to P8 in Table 211 represent the pre-processing components and machine learning models included in each of the pipelines P1 to P8.

[0027] Furthermore, task #2 is a task having data whose variable type is a numeric value. A pipeline P2 is a suitable pipeline for task #2, and has a pre-processing component N2 and a machine learning model M2 for data whose variable type is a numeric value. A task #3 is a task having data whose variable type is a numeric value. A pipeline P3 is a suitable pipeline for task #3, and has a pre-processing component N3 and a machine learning model M3 for data whose variable type is a numeric value.

[0028] Task #4 is a task with numeric and categorical data, and pipeline P4 is a suitable pipeline for task #4, with pre-processing component N4 for numeric data, pre-processing component C4 for categorical data, and machine learning model M4.

[0029] Task #5 is a task with numeric and categorical data, and pipeline P5 is a preferred pipeline for task #5, with pre-processing component N5 for numeric data, pre-processing component C5 for categorical data, and machine learning model M5.

[0030] Task #6 is a task with data whose variable type is text, and pipeline P6 is a suitable pipeline for task #6, and includes preprocessing component T6 for data whose variable type is text, preprocessing component C5 for data whose variable type is categorical, and machine learning model M5.

[0031] Task #7 is a task with numeric and text data, and pipeline P7 is a suitable pipeline for task #7, with pre-processing component N7 for numeric data, pre-processing component T7 for text data, and machine learning model M7.

[0032] Task #8 is a task with data whose variable types are numeric, categorical, and text. And pipeline P8 is a suitable pipeline for task #8. Pipeline P8 has preprocessing component N8 for data whose variable types are numeric, preprocessing component C8 for data whose variable types are categorical, preprocessing component T8 for data whose variable types are text, and machine learning model M8.

[0033] As shown in Table 211, the deviation of each of the pipelines P1 to P8 can be calculated for each of the tasks #1 to #8. However, since the pipelines P1 to P3 do not have preprocessing components for data whose variable type is categorical and data whose variable type is text, it is difficult to make predictions for tasks #4 to #8, and the deviation for each is not calculated. Furthermore, since the pipelines P4 and P5 do not have preprocessing components for data whose variable type is text, it is difficult to make predictions for tasks #6 to #8, and the deviation for each is not calculated. Furthermore, since the pipelines P6 and P7 do not have preprocessing components for data whose variable type is categorical, it is difficult to make predictions for tasks #4, #5, and #8, and the deviation for each is not calculated.

[0034] Returning to Fig. 1, the explanation will be continued. The reception unit 11 also receives information on a target deviation, which is a target value of the deviation for each task, from the user terminal 2. Here, the target deviation may be different for each task or may be the same. The reception unit 11 then outputs the received target deviation to the pipeline set generation unit 103 of the control unit 10. This target deviation may be input in advance.

[0035] The control unit 10 generates a suitable pipeline for each task based on a plurality of tasks, and adds pre-processing to each pipeline so that all variables included in each task can be processed. The control unit 10 then executes the pipeline with the pre-processing added for each task, and generates a pipeline set of robust pipelines from the obtained deviation. Here, the suitable pipeline set for each task is an example of a "first pipeline set". Furthermore, the pipeline set obtained by adding pre-processing to each suitable pipeline for each task is an example of a second pipeline set. Furthermore, the deviation obtained by executing the pipeline with the pre-processing added for each task is an example of an "evaluation value". Furthermore, the pipeline set of robust pipelines generated based on the deviation is an example of a "third pipeline set".

[0036] That is, the control unit 10 obtains a first pipeline set in which each pipeline includes a machine learning model based on a plurality of tasks. Next, the control unit 10 generates a second pipeline set by adding a specific component corresponding to each type of variable included in the data of the plurality of tasks to each pipeline included in the first pipeline set. Next, the control unit 10 obtains an evaluation value of each pipeline included in the second pipeline set by executing the second pipeline set for the plurality of tasks. Next, the control unit 10 generates a third pipeline set by selecting a plurality of pipelines from the second pipeline set based on the evaluation value.

[0037] The control unit 10 will be described in detail below. The control unit 10 includes an execution unit 101, a default pre-processing generation unit 102, and a pipeline set generation unit 103, as shown in FIG.

[0038] The execution unit 101 receives an input of a task set and an AutoML program from the reception unit 11. Next, the execution unit 101 executes the AutoML program for each task included in the task set. Then, the execution unit 101 obtains a suitable pipeline for each task that is output as a result of the execution of the AutoML program. Then, the execution unit 101 outputs the task set and the obtained suitable pipeline for each task to the default pre-processing generation unit 102 and the pipeline set generation unit 103.

[0039] The default preprocessor generation unit 102 acquires a task set and a suitable pipeline for each task from the execution unit 101. Here, the default preprocessor generation unit 102 has in advance information on the type of variables input to each component in each task. Then, the default preprocessor generation unit 102 selects one variable type from the types of variables used in each task.

[0040] Next, the default pre-processing generation unit 102 obtains from each pipeline a pre-processing component that performs processing using data of the selected variable type as input. Hereinafter, a pre-processing component that performs processing using data of the selected variable type as input is referred to as a "pre-processing component for the selected variable type." Here, the pre-processing component for the selected variable type may be different for each pipeline or may be the same. Therefore, the default pre-processing generation unit 102 calculates the frequency of occurrence of the same pre-processing component in all pipelines among the pre-processing components for the selected variable type.

[0041] Next, the default pre-processing generation unit 102 extracts a pre-processing component for the selected variable type that appears most frequently, and sets the extracted pre-processing component as the default pre-processing component for the selected variable type.

[0042] The default preprocessing generation unit 102 performs the above-mentioned process for all variable types included in each task, and determines default preprocessing components for all variable types. For example, if there are three types of variables included in each task, namely, "numeric", "category", and "character string", the default preprocessing generation unit 102 determines default preprocessing components for each of "numeric", "category", and "character string". However, the default preprocessing generation unit 102 may omit determining default preprocessing components for preprocessing components for variable types included in all tasks. The default preprocessing generation unit 102 then outputs information on default preprocessing components for each variable type included in each task to the pipeline set generation unit 103.

[0043] This default pre-processing component is an example of a "specific component". That is, the default pre-processing generation unit 102 selects the specific component based on the components included in the first pipeline set. More specifically, for a specific pipeline included in the first pipeline set, the default pre-processing generation unit 102 selects a component that is not included in the specific pipeline among the components included in the first pipeline set as a specific component for the specific pipeline. Furthermore, when there are multiple types of components that are not included in the specific pipeline among the components included in the first pipeline set for a specific type of variable, the default pre-processing generation unit 102 performs the following process. For example, the default pre-processing generation unit 102 selects a specific component corresponding to a specific type of variable based on the frequency of occurrence of each type in the first pipeline set.

[0044] Here, an example will be described in which there are tasks #1 to #8 and pipelines P1 to P8 shown in table 211 of Fig. 3. Table 211 also shows the degree of deviation for each pair of tasks #1 to #8 and pipelines P1 to P8.

[0045] For example, when preprocessing components C4 and C5 are the same preprocessing component, the default preprocessing generation unit 102 selects a default preprocessing component for data whose variable type is categorical as follows: Since preprocessing component C4 appears twice in pipelines P4 and P5, and preprocessing component C8 appears once in pipeline P8, the default preprocessing generation unit 102 selects pipeline P4, which appears more frequently, as the default preprocessing component for data whose variable type is categorical. The default preprocessing generation unit 102 similarly determines a default preprocessing component for data whose variable type is numeric and a default preprocessing component for data whose variable type is text.

[0046] The pipeline set generation unit 103 acquires a task set and a suitable pipeline for each task from the execution unit 101. The pipeline set generation unit 103 also receives an input of information on default pre-processing components for each variable type from the default pre-processing generation unit 102. The pipeline set generation unit 103 also receives an input of a target deviation degree from the reception unit 11.

[0047] Next, the pipeline set generation unit 103 selects one pipeline from the suitable pipelines for each task, and determines whether there is a pre-processing component that exists in other pipelines but does not exist in the selected pipeline. Here, a pre-processing component that exists in other pipelines but does not exist in the selected pipeline is called a "missing pre-processing component." If a missing pre-processing component exists in the selected pipeline, the pipeline set generation unit 103 complements the inter-missing processing using a default pre-processing component for the type of variable processed by the pre-processing component corresponding to the missing pre-processing component. Here, a pipeline in which all missing pre-processing components are stored, that is, a pipeline that has all pre-processing components for the types of variables used in each task, is called a "complete pipeline." The pipeline set generation unit 103 modifies all suitable pipelines for each task into complete pipelines having the same pre-processing components.

[0048] Next, the pipeline set generation unit 103 transmits the suitable pipeline for each task corrected to a complete pipeline to the execution unit 101, and causes each pipeline to execute processing for each task. After that, the pipeline set generation unit 103 acquires the execution result from the execution unit 101. Next, the pipeline set generation unit 103 uses the acquired execution result to calculate the deviation degree for each task of the suitable pipeline for each task corrected to a complete pipeline.

[0049] Fig. 4 is a diagram showing an example of the degree of deviation of a suitable pipeline for each task corrected to a complete pipeline. For example, the pipeline set generation unit 103 corrects each of the pipelines P1 to P8 shown in table 211 in Fig. 3 to a complete pipeline. Here, the pipeline set generation unit 103 sets the default pre-processing component for data whose variable type is category to pre-processing component C0, and the default pre-processing component for data whose variable type is text to pre-processing component T0.

[0050] The pipeline set generation unit 103 adds pre-processing components C0 and T0 to each of the pipelines P1 to P3 to form a complete pipeline. The pipeline set generation unit 103 also adds pre-processing component T0 to pipelines P4 and P5 to form a complete pipeline. The pipeline set generation unit 103 also adds pre-processing component N0 and pre-processing component C0 to pipeline P6 to form a complete pipeline. The pipeline set generation unit 103 also adds pre-processing component C0 to pipeline P7 to form a complete pipeline.

[0051] Here, in table 211 of Fig. 3, pipelines P1 to P7 have missing pre-processing components and difficult-to-execute tasks, so executable pipeline P8 becomes a pipeline set of robust pipelines. In contrast, by correcting all pipelines P1 to P8 to complete pipelines as shown in table 212 of Fig. 4, the pipeline set generation unit 103 can calculate the deviations of each of pipelines P1 to P8 from all tasks #1 to #8. Therefore, the pipeline set generation unit 103 can include all of pipelines P1 to P8 in the pipeline set.

[0052] Next, the pipeline set generation unit 103 calculates an error representing the difference between the deviation and the target deviation for each pair of a task and a pipeline. The minimum value of the error is set to 0. That is, the error can be expressed as an equation: "error = max (deviation - target deviation, 0)". In this equation, "max" is a function that selects the larger value of two elements. The error is sometimes called "loss". Note that when the target deviation is set to 0, that is, when the target deviation is set to the case where the deviation matches the predicted performance of a suitable pipeline, the error matches the deviation.

[0053] Next, the pipeline set generation unit 103 initializes the pipeline set with an empty set. Then, the pipeline set generation unit 103 repeats the following process to generate a pipeline set of robust pipelines. Specifically, the pipeline set generation unit 103 provisionally adds, to the pipeline set, each of the suitable pipelines for each task that is not included in the pipeline set, one by one. Then, for each pipeline set to which a different pipeline is provisionally added, the pipeline set generation unit 103 calculates a total error by adding, for all tasks, the minimum value of the error for each task included in the pipeline set. Then, the pipeline set generation unit 103 actually adds, to the pipeline set, the provisionally added pipeline that has the minimum total error, to generate a new pipeline set.

[0054] The pipeline set generation unit 103 repeats adding pipelines to the pipeline set until a predetermined generation end condition for the pipeline set is satisfied. The generation end condition is set based on, for example, an upper limit on the number of pipelines included in the pipeline set, an elapsed time, etc. Then, the pipeline set generation unit 103 outputs the pipeline set at the time when the generation end condition is satisfied to the output unit 12 as a pipeline set of robust pipelines.

[0055] Fig. 5 is a diagram showing an example of a first selection of pipelines to be included in a pipeline set. Fig. 6 is a diagram showing an example of a second selection of pipelines to be included in a pipeline set. Here, an example of generation of a pipeline set by the pipeline set generation unit 103 will be described with reference to Figs. 5 and 6. Here, a case where the target deviation is 0.1 will be described.

[0056] The pipeline set generation unit 103 subtracts the target deviation from the deviation of each combination of the pipelines P1 to P8 and the tasks #1 to #8 to calculate each error as shown in table 213 of FIG.

[0057] When the pipeline set is an empty set, the pipeline set generation unit 103 adds each of the pipelines P1 to P8 one by one to the pipeline set, and the minimum value of the error for each of the tasks #1 to #8 included in the pipeline set matches the error of each of the pipelines P1 to P8. Therefore, the pipeline set generation unit 103 calculates the total error by simply calculating the sum of the errors for each of the pipelines P1 to P8. In this case, as shown in table 213 in FIG. 5, the total error of the pipeline P1 is 0.5, which is the smallest, so the pipeline set generation unit 103 adds the pipeline P1 to the pipeline set.

[0058] Next, the pipeline set generation unit 103 adds each of the pipelines P2 to P8 to the pipeline set including the pipeline P1, and finds the minimum error value for each of the tasks #1 to #8. Table 214 shows the minimum error value for each of the tasks #1 to #4 when each of the pipelines P2 to P8 is added to the pipeline set including the pipeline P1. The pipeline P1 in table 214 is already included in the pipeline set, and is excluded from the pipeline to be added next. The grayed-out parts in table 214 show the error of the pipeline P1, since the error of the pipeline P1 is the minimum value. The other parts show the errors of the pipelines P2 to P8, since the errors of the added pipelines P2 to P8 are smaller than the error of the pipeline P1.

[0059] The pipeline set generation unit 103 calculates the sum of errors by adding the errors in table 214 each time pipelines P2 to P8 are added. In this case, since the sum of errors is 0.1 when pipeline P5 is added, which is the smallest, the pipeline set generation unit 103 adds pipeline P5 to the pipeline set. That is, the pipeline set includes pipelines P1 and P5.

[0060] Here, the suitable pipeline for each task is an example of a “reference pipeline.” That is, the pipeline set generation unit 103 obtains, for each task of the multiple tasks, the difference between the prediction accuracy of the reference pipeline and the prediction accuracy of each pipeline included in the second pipeline set as the evaluation value.

[0061] Returning to Fig. 1, the explanation will be continued. The output unit 12 receives an input of a pipeline set of robust pipelines from the pipeline set generation unit 103. Then, the output unit 12 transmits the acquired pipeline set to the user terminal 2 to notify the user. The user can acquire a pipeline having a certain degree of prediction performance in a short time by executing an AutoML program using the pipeline set of robust pipelines notified from the information processing device 1.

[0062] Fig. 7 is a flowchart of a process of generating a pipeline set of a robust pipeline by the information processing device according to the embodiment 1. Next, a flow of a process of generating a pipeline set of a robust pipeline by the information processing device 1 according to the embodiment will be described with reference to Fig. 7.

[0063] A user inputs a task set and an AutoML program to the information processing device 1 using the user terminal 2 (step S1).

[0064] The receiving unit 11 receives the task set and the AutoML program input from the user terminal 2, and outputs them to the executing unit 101. The executing unit 101 executes AutoML for all tasks in the task set, and obtains a set of suitable pipelines for each task (step S2).

[0065] The default pre-processing generation unit 102 determines default pre-processing components for the types of all variables included in each task (step S3).

[0066] The pipeline set generation unit 103 supplements the missing pre-processing components of the suitable pipelines having missing pre-processing components with default pre-processing components, thereby modifying each pipeline included in the pipeline set into a complete pipeline (step S4).

[0067] The pipeline set generation unit 103 causes the execution unit 101 to execute each of the suitable pipelines modified into complete pipelines for each task, and acquires the execution results. Then, the pipeline set generation unit 103 calculates the deviation degree for all tasks included in the task set for each suitable pipeline modified into a complete pipeline (step S5).

[0068] Next, the pipeline set generation unit 103 adds each suitable pipeline that is not included in the pipeline set to the pipeline set one by one, and calculates the sum of errors by adding the minimum value of errors included in the pipeline set for all tasks.The pipeline set generation unit 103 then repeats adding the pipeline with the minimum sum of errors to the pipeline set to generate a pipeline set (step S6).

[0069] The pipeline set generation unit 103 outputs the generated pipeline set to the output unit 12. The output unit 12 transmits the pipeline set of the robust pipeline input from the pipeline set generation unit 103 to the user terminal 2 (step S7).

[0070] Fig. 8 is a flowchart of a process of generating a default pre-processing component by the information processing device according to the embodiment 1. Each process shown in the flowchart of Fig. 8 corresponds to an example of the process executed in step S3 of Fig. 7. Next, a flow of a process of generating a default pre-processing component by the information processing device 1 according to the embodiment will be described with reference to Fig. 8.

[0071] The default pre-processing generation unit 102 acquires a task set and a suitable pipeline for each task from the execution unit 101 (step S101).

[0072] The default preprocessing generator 102 selects one variable type from the types of variables used in each task (step S102).

[0073] Next, the default pre-processing generation unit 102 acquires from each pipeline a pre-processing component that performs processing using data of the selected variable type as input. Then, the default pre-processing generation unit 102 calculates the appearance frequency of the same pre-processing component among the pre-processing components for the selected variable type in all suitable pipelines for each task (step S103).

[0074] Next, the default pre-processing generator 102 extracts the pre-processing component with the highest occurrence frequency from among the pre-processing components for the selected variable type, and sets the extracted pre-processing component as the default pre-processing component for the selected variable type (step S104).

[0075] Next, the default pre-processing unit 102 judges whether the generation of default pre-processing components has been completed for all variable types (step S105). If there are any variable types for which the generation of default pre-processing components has not been performed (step S105: No), the default pre-processing unit 102 returns to step S102.

[0076] On the other hand, if the generation of default pre-processing components has been completed for all variable types (step S105: Yes), the default pre-processing generation unit 102 outputs all the generated default pre-processing components to the pipeline set generation unit 103 (step S106).

[0077] As described above, the information processing device according to the present embodiment determines default pre-processing components capable of processing data having the types of all variables included in each task in the task set. Then, the missing pre-processing components of each suitable pipeline for each task generated by AutoML are complemented with default pre-processing components, and all suitable pipelines for each task are modified to complete pipelines. This makes it possible to process all tasks with each suitable pipeline for each task. Therefore, the information processing device according to the present embodiment generates a pipeline set of robust pipelines using the suitable pipelines for each task modified to complete pipelines.

[0078] As a result, all suitable pipelines for each task can be used to generate a pipeline set of robust pipelines, and a pipeline set of highly robust pipelines can be generated. By executing automation of machine learning using the pipeline set of highly robust pipelines thus obtained, it becomes easier to determine a suitable pipeline, and the convenience of automating machine learning can be improved.

[0079] Furthermore, since a preprocessing component that occurs frequently can be considered to be a preprocessing component with higher versatility, it is possible to further improve robustness by making the preprocessing component that occurs frequently the default preprocessing component. In particular, when the number of types of preprocessing components for the same variable type is small, the difference in application frequency is dominant over the high versatility, and robustness can be improved by making the preprocessing component that occurs frequently the default preprocessing component. An example of a case where the number of types of preprocessing components that can be used is a case where the number of types is half or less of the total number of tasks. EXAMPLES

[0080] Next, a second embodiment will be described. An information processing device 1 according to this embodiment is also shown in the block diagram of FIG. 1. The information processing device 1 according to this embodiment differs from the first embodiment in the method of generating a default pre-processing component by the default pre-processing generation unit 102. The method of generating a default pre-processing component by the default pre-processing generation unit 102 according to this embodiment will be described in detail below. In the following description, the operation of each unit similar to that of the first embodiment will be omitted.

[0081] The default preprocessing generation unit 102 extracts all variable types included in each task in the task set. Next, the default preprocessing generation unit 102 performs the following process for each of the extracted variable types to generate a default preprocessing component for each variable type.

[0082] The default pre-processing generation unit 102 extracts all pre-processing components corresponding to the target variable type from among the pre-processing components included in the suitable pipeline for each task. The default pre-processing generation unit 102 also extracts a pipeline having a pre-processing component corresponding to the target variable type from among the suitable pipelines for each task.

[0083] Next, for each of the extracted pipelines, the default preprocessing generation unit 102 replaces the preprocessing component for the target variable type with another preprocessing component of another pipeline for the same variable type. Next, the default preprocessing generation unit 102 causes the execution unit 101 to execute processing by the pipeline after the replacement of the preprocessing component for the task that became suitable before the replacement of the preprocessing component. Then, the default preprocessing generation unit 102 acquires the execution result by the execution unit 101, calculates the error before and after the replacement of the preprocessing component, and calculates the deterioration of prediction accuracy due to the replacement of the preprocessing component, that is, the amount of increase in error. The default preprocessing generation unit 102 determines that the smaller the increase in error, the less the deterioration of prediction accuracy.

[0084] The default pre-processing generator 102 then selects the pre-processing component with the smallest maximum value of deterioration in prediction accuracy when the pre-processing component is replaced as the default pre-processing component. The smaller the maximum value of deterioration in prediction accuracy, the more likely it is that the deterioration in prediction accuracy will be prevented from becoming extremely large for many tasks, and the more robustness can be improved.

[0085] However, the default pre-processing component generation unit 102 can use other methods for selecting default pre-processing components as long as the selection method can suppress deterioration of prediction accuracy for many tasks. For example, the default pre-processing component generation unit 102 can calculate the sum of deterioration of prediction accuracy when replacing each pre-processing component, and select the pre-processing component that minimizes the calculated sum as the default pre-processing component. It can be said that the smaller the sum of deterioration of prediction accuracy is, the more deterioration of prediction accuracy can be suppressed on average for any task, and the more robustness can be improved.

[0086] The default pre-processing generation unit 102 generates default pre-processing components for all variable types using the method described above. Then, the default pre-processing generation unit 102 outputs the generated default pre-processing components for all variable types to the pipeline set generation unit 103.

[0087] That is, when there are multiple types of components that are not included in a specific pipeline among the components included in the first pipeline set for a specific type of variable, the default preprocessing generation unit 102 executes the following process: The default preprocessing generation unit 102 selects a specific component that corresponds to the specific type of variable based on the deterioration of prediction accuracy by the machine learning model of each pipeline included in the first pipeline set when a component of a different type is replaced in the first pipeline set.

[0088] Fig. 9 is a diagram illustrating an example of a method for generating a default pre-processing component by the information processing device according to the embodiment 2. Next, an example of a method for generating a default pre-processing component by the information processing device 1 according to the embodiment will be described with reference to Fig. 9.

[0089] For example, when suitable pipelines P1 to P8 exist for tasks #1 to #8 shown in table 211 in FIG. 3, the type of the target variable is treated as a category, and a default pre-processing component for the category is determined.

[0090] In this case, the default pre-processing generation unit 102 extracts pre-processing components C4, C5, and C8 as pre-processing components for the categories included in the pipelines P1 to P8. Next, the default pre-processing generation unit 102 extracts pipelines P4, P5, and P8 having the extracted pre-processing components C4, C5, and C8 as shown in Table 221 as pipelines having pre-processing components for the categories.

[0091] Next, the default pre-processing generation unit 102 calculates the error of the pipeline P4. Next, the default pre-processing generation unit 102 changes the pre-processing component C4 of the pipeline P4 to the pre-processing component C5, and causes the execution unit 101 to execute task #4 to obtain the execution result. Then, the default pre-processing generation unit 102 calculates the error when the pre-processing component C4 of the pipeline P4 is changed to the pre-processing component C5. Also, the default pre-processing generation unit 102 changes the pre-processing component C4 of the pipeline P4 to the pre-processing component C8, and causes the execution unit 101 to execute task #4 to obtain the execution result. Then, the default pre-processing generation unit 102 calculates the error when the pre-processing component C4 of the pipeline P4 is changed to the pre-processing component C8. The default pre-processing generation unit 102 similarly replaces the pre-processing for the pipelines P5 and P8, and calculates the respective errors when the replacement is performed. As a result, the default pre-processing generation unit 102 calculates the respective errors when the pre-processing components for the categories are replaced for the pipelines P4, P5, and P8, as shown in Table 222.

[0092] Table 223 is a table summarizing the errors for each task when replaced by each preprocessing component. As shown in Table 223, the maximum value of deterioration in prediction accuracy, which is the amount of increase in error, is 0.2 for preprocessing component C4, 0.15 for preprocessing component C5, and 0.3 for preprocessing component C8. Therefore, the default preprocessing generation unit 102 sets preprocessing component C5, which has the smallest maximum value of deterioration in prediction accuracy, as the default preprocessing component for the category.

[0093] Furthermore, if the pre-processing component with the smallest sum of deterioration in prediction accuracy is to be set as the default pre-processing component, the default pre-processing generation unit 102 determines the default pre-processing component as follows: The sum of deterioration in prediction accuracy is 0.2 for pre-processing component C4, 0.25 for pre-processing component C5, and 0.4 for pre-processing component C8. Therefore, the default pre-processing generation unit 102 sets pre-processing component C4, with the smallest sum of deterioration in prediction accuracy, as the default pre-processing component for the category.

[0094] Fig. 10 is a flowchart of a process for generating a default pre-processing component by an information processing device according to Example 2. Each process shown in the flowchart of Fig. 10 corresponds to an example of the process executed in step S3 of Fig. 7. Next, a flow of a process for generating a default pre-processing component by the information processing device 1 according to this embodiment will be described with reference to Fig. 10.

[0095] The default pre-processing generation unit 102 acquires a task set and a suitable pipeline for each task from the execution unit 101 (step S201).

[0096] The default preprocessing generator 102 selects one variable type from the types of variables used in each task (step S202).

[0097] Next, the default pre-processing generation unit 102 extracts all pre-processing components for the selected variable type from among the pre-processing components included in the suitable pipeline for each task (step S203).

[0098] Next, the default pre-processing generator 102 selects one task including the selected variable type from the task set, and obtains a suitable pipeline for the selected task from the suitable pipelines for each task (step S204).

[0099] Next, the default pre-processing generation unit 102 replaces the pre-processing components for the selected variable types in the obtained preferred pipeline with the pre-processing components for the selected variable types, and calculates an error (step S205).

[0100] Next, the default preprocessing generation unit 102 determines whether the error calculation process is completed when the preprocessing components are replaced for all tasks including the selected variable type (step S206). If there are any tasks that have not been subjected to the error calculation process (step S206: No), the default preprocessing generation unit 102 returns to step S204.

[0101] On the other hand, when the error calculation process is completed for all tasks including the selected variable type (step S206: Yes), the default pre-processing generation unit 102 selects a default pre-processing component for the selected variable type using the error (step S207). For example, the default pre-processing generation unit 102 selects the pre-processing component with the smallest maximum value of the deterioration of prediction accuracy, which corresponds to the amount of increase in error, as the default pre-processing component.

[0102] Next, the default pre-processing unit 102 judges whether the generation of default pre-processing components has been completed for all variable types (step S208). If there are any variable types for which the generation of default pre-processing components has not been performed (step S208: No), the default pre-processing unit 102 returns to step S202.

[0103] On the other hand, if the generation of default pre-processing components has been completed for all variable types (step S208: Yes), the default pre-processing generation unit 102 outputs all the generated default pre-processing components to the pipeline set generation unit 103 (step S209).

[0104] As described above, the information processing device according to the present embodiment determines the default pre-processing part according to the deterioration of prediction accuracy represented by the increase in error. By setting the pre-processing part that causes little deterioration in prediction accuracy even when replaced as the default pre-processing part, it is possible to reduce the deterioration of prediction accuracy in more tasks and improve robustness. In particular, when there are many types of pre-processing parts for the same variable type, there is a risk that the difference in the occurrence frequency of each type will be small, so determining the default pre-processing part according to the deterioration of prediction accuracy, as in the information processing device according to the present embodiment, is effective in improving robustness. A large number of types of pre-processing parts refers to, for example, a case where the number of types is more than half of the total number of tasks.

[0105] In each of the above embodiments, the information processing device determines the default preprocessing component based on the frequency of occurrence of each preprocessing component and the degree of deterioration in prediction accuracy, but if robustness can be sacrificed to a certain extent, the default preprocessing component may be determined appropriately. For example, the information processing device may determine in advance that OneHotEncoder is used as the default preprocessing component for categories and CountVectorizer is used as the default preprocessing component for text. In addition, the information processing device may randomly select from all preprocessing components for the target variable type among the preprocessing components included in the preferred pipeline for each task.

[0106] (Hardware configuration) Fig. 11 is a hardware configuration diagram of a computer. The information processing device 1 can be realized by a computer 90 shown in Fig. 11. Next, an example of the hardware configuration of the information processing device 1 will be described with reference to Fig. 11.

[0107] 11, a computer 90 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0108] The network interface 94 is an interface for communication between the computer 90 and an external device. The network interface 94 relays communication between the user terminal 2 and the CPU 91, for example. The network interface 94 realizes the functions of the reception unit 11 and the output unit 12.

[0109] The hard disk 93 is an auxiliary storage device that stores various programs including a program for implementing the functions of the control unit 10 including the execution unit 101, the default pre-processing generation unit 102, and the pipeline set generation unit 103 illustrated in FIG.

[0110] The memory 92 is a main storage device, and may be, for example, a dynamic random access memory (DRAM).

[0111] The CPU 91 reads out various programs from the hard disk 93, and loads and executes them in the memory 92. In this way, the CPU 91 realizes the functions of the control unit 10 including the execution unit 101, the default pre-processing generation unit 102, and the pipeline set generation unit 103 illustrated in FIG.

[0112] Here, a GPU (Graphics Processing Unit) may be used instead of the CPU 91. Furthermore, a plurality of CPUs 91 or a combination of a CPU 91 and a GPU may be used in cooperation with each other to realize the functions of the control unit 10. [Explanation of symbols]

[0113] 1. Information processing device 2. User terminal 10 Control section 11 Reception 12 Output section 101 Executive Department 102 Default preprocessing generator 103 Pipeline Set Generation Unit

Claims

1. Obtaining a first set of pipelines based on the plurality of tasks, each pipeline including a machine learning model; generating a second pipeline set by adding specific components corresponding to each type of variable included in the data of the plurality of tasks to each pipeline included in the first pipeline set; acquiring evaluation values ​​of each pipeline included in the second pipeline set by executing the second pipeline set for the plurality of tasks; generating a third pipeline set by selecting a plurality of pipelines from the second pipeline set based on the evaluation values; A pipeline set generating program for causing a computer to execute a process.

2. selecting the particular component based on components included in the first pipeline set; 2. The pipeline set generating program according to claim 1, further comprising a step of causing the computer to execute a process.

3. The selection of the specific component includes, for a specific pipeline included in the first pipeline set, selecting a component that is not included in the specific pipeline among components included in the first pipeline set as the specific component for the specific pipeline.

3. The pipeline set generating program according to claim 2.

4. When there are a plurality of types of components that are not included in the specific pipeline among the components included in the first pipeline set for the specific type of the variable, the specific component corresponding to the specific type of the variable is selected based on the frequency of occurrence of each of the types in the first pipeline set.

4. The pipeline set generating program according to claim 3.

5. When there are a plurality of types of components that are not included in the specific pipeline among the components included in the first pipeline set for the specific type of the variable, the specific component corresponding to the specific type of the variable is selected based on a deterioration in prediction accuracy by the machine learning model of each of the pipelines included in the first pipeline set when a component of a different type is replaced in the first pipeline set.

4. The pipeline set generating program according to claim 3.

6. 2. The pipeline set generation program according to claim 1, wherein the process of acquiring the evaluation value includes a process of acquiring, for each of the plurality of tasks, a difference between a prediction accuracy of a reference pipeline and a prediction accuracy of each of the pipelines included in the second pipeline set as the evaluation value.

7. Obtaining a first set of pipelines based on the plurality of tasks, each pipeline including a machine learning model; generating a second pipeline set by adding specific components corresponding to each type of variable included in the data of the plurality of tasks to each pipeline included in the first pipeline set; acquiring evaluation values ​​of each pipeline included in the second pipeline set by executing the second pipeline set for the plurality of tasks; generating a third pipeline set by selecting a plurality of pipelines from the second pipeline set based on the evaluation values; A pipeline set generating method comprising the steps of: causing a computer to execute a process.

8. Obtaining a first set of pipelines based on the plurality of tasks, each pipeline including a machine learning model; generating a second pipeline set by adding specific components corresponding to each type of variable included in the data of the plurality of tasks to each pipeline included in the first pipeline set; acquiring evaluation values ​​of each pipeline included in the second pipeline set by executing the second pipeline set for the plurality of tasks; generating a third pipeline set by selecting a plurality of pipelines from the second pipeline set based on the evaluation values; A control unit that executes the processing; An information processing device comprising:

Images