Model search device, model search method, and program

Abstract

A task list output means in this model search device outputs a task list indicating candidates of a task. A data list output means receives the selection of a task included in the task list and outputs a data list indicating data candidates to be used in the selected task. A model list output means receives the selection of data included in the data list and outputs a model list that indicates model candidates that use the selected data. A model output means receives the selection of a model included in the model list and outputs the selected model.

Description

[Technical Field]

[0001] The present disclosure relates to exploring machine learning models. [Background technology]

[0002] In recent years, predictive models obtained by machine learning have been used in various fields. To make useful predictions in a given environment, it is necessary to select an appropriate model according to the target task and the data to be used. Patent Document 1 describes an example in which a user sets the details of a newly created model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2020 / 085114 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the user must input information for each setting item independently, which may make it difficult for an inexperienced user to select and input detailed information.

[0005] An object of the present disclosure is to provide a model search device that enables a user to easily search for a model for achieving a target task. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a model exploration device includes: a task list output means for outputting a task list showing candidate tasks; The task selected by the user from the output task list is associated with the correspondence information indicating the correspondence between the task, the data, and the trained model. a data list output means for outputting a data list showing candidate data; With reference to the correspondence information, the learned data associated with the data selected by the user from the output data list is a model list output means for outputting a model list showing candidate models; The user selects from the output model list Selected The above study a model output means for outputting a model; Equipped with.

[0007] In another aspect of the present disclosure, a computer-implemented model exploration method includes: Output a task list showing candidate tasks, The task selected by the user from the output task list is associated with the correspondence information indicating the correspondence between the task, the data, and the trained model. Output a data list showing candidate data, With reference to the correspondence information, the learned data associated with the data selected by the user from the output data list is Output a model list showing candidate models, The user selects from the output model list Selected The above study Output the model.

[0008] In yet another aspect of the invention, a program includes: Output a task list showing candidate tasks, The task selected by the user from the output task list is associated with the correspondence information indicating the correspondence between the task, the data, and the trained model. Output a data list showing candidate data, With reference to the correspondence information, the learned data associated with the data selected by the user from the output data list is Output a model list showing candidate models, The user selects from the output model list Selected The above study The computer is caused to execute a process for outputting the model. [Effects of the Invention]

[0009] According to the present disclosure, a user can easily search for a model to achieve a desired task. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the overall configuration of a model management system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of the model search device according to the first embodiment. [Figure 3] FIG. 1 is a block diagram showing a functional configuration of a model search device according to a first embodiment. [Figure 4] An example of the correspondence information is shown below. [Figure 5] An example of a model search screen is shown. [Figure 6] 10 shows another example of the model search screen. [Figure 7] 10 is a flowchart of a model search process performed by the model search device. [Figure 8] FIG. 1 is a block diagram showing a schematic configuration of a model management system using a server and a terminal device. [Figure 9] FIG. 10 is a block diagram showing the functional configuration of a model search device according to a second embodiment. [Figure 10] 10 is a flowchart of a process performed by a model searching device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Overall configuration] FIG. 1 is a block diagram showing the overall configuration of a model management system according to a first embodiment. The model management system 1 includes a model search device 100, a display device 2, and an input device 3. The model search device 100 is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays a model search screen generated by the model search device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by the user to make inputs and selections required when searching for a model.

[0012] First, an outline of the operation of the model search device 100 will be described. The model search device 100 stores various machine learning models. As an example, the machine learning models include models that have already been trained. As another example, the machine learning models may include models that have not been trained. Furthermore, the machine learning models may include both trained models and untrained models. Note that untrained models include, for example, untrained models in which hyperparameters have been set, and models in which parameters have been trained to a certain extent.

[0013] The model search device 100 also stores various data (hereinafter also referred to as "actual data") acquired in a real environment. A user uses the model search device 100 to search for a model suitable for a target task. Specifically, the model search device 100 first presents a list of task candidates (hereinafter also referred to as a "task list") to the user. When the user selects a target task from the task list, the model search device 100 presents a list of data candidates to be used for the task (hereinafter also referred to as a "data list"). When the user selects data from the data list, the model search device 100 presents the user with a list of model candidates suitable for the selected data (hereinafter also referred to as a "model list"). The user can determine a model suitable for the target task by selecting an appropriate model from the presented model list. Thereafter, the user can obtain data related to the determined model from the model management system and use it for actual operation.

[0014] A machine learning model is information that represents the relationship between explanatory variables and a response variable. A machine learning model is, for example, a component that estimates the result of an estimation target by calculating a response variable based on explanatory variables. A machine learning model is generated by executing a learning algorithm using training data, for which the response variable value has already been obtained, and arbitrary parameters as input. A machine learning model may be represented, for example, by a function c that maps an input x to a correct answer y. A machine learning model may estimate a numerical value of an estimation target, or may estimate a label of an estimation target. A machine learning model may output a variable that describes the probability distribution of a response variable. A machine learning model may also be referred to as a "learning model," an "analysis model," an "AI (artificial intelligence) model," a "trained model," an "inference model," or a "prediction formula." Here, an explanatory variable is a variable used as an input in a machine learning model. An explanatory variable may also be referred to as a "feature" or "characteristic."

[0015] A task refers to a type of prediction or inference that a machine learning model makes based on a problem or question and available data. For example, a classification task refers to assigning data to categories, and a clustering task refers to grouping data according to similarity. Specific examples of tasks in this embodiment are described below.

[0016] [Hardware configuration] 2 is a block diagram showing the hardware configuration of the model search device 100. As shown in the figure, the model search device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0017] The I / F 111 inputs and outputs data to and from external devices. Specifically, instructions and selections input by the user using the input device 3 are input to the model search device 100 through the I / F 111. In addition, data of a model selection screen including a list of tasks, data, and models generated by the model search device 100 is output to the display device 2 through the I / F 111.

[0018] The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model search device 100 by executing a pre-prepared program. The processor 112 may be a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, or an FPGA (Field-Programmable Gate Array). The processor 112 executes the model search process described below.

[0019] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0020] The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the model search device 100. The recording medium 114 records various programs executed by the processor 112. When the model search device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0021] DB 115 stores prepared models, various real data obtained in real environments, etc. DB 115 may also store training data used for training each learned model, evaluation data used for evaluation, etc.

[0022] (Functional configuration) 3 is a block diagram showing the functional configuration of the model search device 100 of the first embodiment. Functionally, the model search device 100 includes a data storage unit 121, a model storage unit 122, and a model search unit 123.

[0023] The data storage unit 121 stores various types of real data obtained in a real environment. Specifically, the real data stored includes various types of data such as image data, natural language data, table data, and time-series data. The data storage unit 121 may also store so-called open data, as well as training models and evaluation models used in learning a trained model. The data storage unit 121 also stores attribute information for each piece of stored data. The attribute information includes, for example, information such as the type and structure of the data, explanatory variables that the data has, and the date and time when the data was acquired.

[0024] The model storage unit 122 stores machine learning models that have been trained using various algorithms. For example, the model storage unit 122 stores machine learning models such as a decision tree model, a piecewise linear model using FAB inference (Factorized Asymptotic Bayesian Inference), and a neural network. Note that a piecewise linear model technique using FAB inference is disclosed in, for example, U.S. Patent Publication US2014 / 0222741A1.

[0025] The model storage unit 122 also stores correspondence information indicating the correspondence between tasks, data, and models. The correspondence information is information indicating the correspondence between tasks, data, and models, such as, for example, when executing a certain task A, data B is used, and model C is a model for processing data B.

[0026] FIG. 4 shows an example of correspondence information. In this example, multiple tasks, data, and models are prepared, and some of the correspondences between them are indicated by dotted lines. For example, it is shown that image data can be used for a face recognition task, which belongs to the classification task, and that in this case the image data can be processed by a neural network model. It is also shown that time-series data can be used for a regression task, and that in this case the time-series data can be processed by a decision tree model. In this way, the model storage unit 122 pre-stores information on the data used by each model and the tasks that each model can execute, regarding the stored models.

[0027] The model storage unit 122 also stores attribute information for each model. The attribute information includes, for example, information on related models related to each model, information on the creation date and time or update date and time of each model, etc. The information on related models includes, for example, information on whether the model is an original model or an updated model, version information of the same model, etc.

[0028] The model search unit 123 searches for models stored in the model storage unit 122 based on user input. The user inputs a target task and usage data to search for an appropriate model for achieving the target task using the usage data. Specifically, the model search unit 123 generates a model search screen and displays it on the display device 2. Then, based on the input or selection made by the user while viewing the model search screen, the model search unit 123 refers to the data storage unit 121 and the model storage unit 122 and presents candidate models appropriate for the user's input. The model search unit 123 refers to the correspondence information exemplified in FIG. 4, extracts models appropriate for the task and data selected by the user, and displays them as a model list on the model search screen.

[0029] 5 shows an example of a model search screen. The model search screen 40 includes a task column 41, a data column 42, and a model column 43. The model search screen 40 is designed to accept user selections basically in the order of task, data, and model. In the initial state, a default list is displayed in the task column 41, but the data column 42 and model column 43 are blank.

[0030] The display order of the model search screen 40 is not limited to the example in Fig. 5. For example, the model search screen 40 may be designed to first display the data column 42 or the model column 43 and then accept the user's selection.

[0031] The task column 41 displays a list of multiple tasks that can be achieved using the machine learning model. Each displayed task corresponds to a candidate task. In the example of Figure 5, the major categories of tasks are "classification," "regression," "sign detection," and "object recognition," and each category has its own subcategory of tasks.

[0032] Additionally, a sort button 41a is displayed in the task column 41. The sort button 41a is a button for sorting the display order of the tasks displayed in the task column 41. The user can change the display order of the tasks in the task column 41 by operating the sort button 41a and selecting a display order such as alphabetical order or hiragana order, for example.

[0033] Additionally, a narrowing down button 41b is displayed in the task column 41. The narrowing down button 41b is a button for narrowing down the tasks displayed in the task column 41 to those of a specific category. The user can narrow down the tasks displayed in the task column 41 to those of a specific major category by operating the narrowing down button 41b and selecting one of the tasks of the major category. For example, if the user operates the narrowing down button 41b to select "regression," only "regression" tasks are displayed in the task column 41.

[0034] Additionally, an OK button 41c is displayed in the task column 41. The OK button 41c is a button for confirming the user's selection in the task column 41. The user selects the desired task from the displayed task list by, for example, moving the cursor C. In the example of FIG. 5, the task "Demand Forecast" currently selected by the user is underlined. The user confirms the task selection by pressing the OK button 41c.

[0035] When the user selects a target task from the task column 41 and presses the OK button 41c to confirm the selection, a list of data is displayed in the data column 42. The data list displayed here is a list of data corresponding to the task selected by the user in the task column 41. Specifically, the model search unit 123 refers to the correspondence information described above to identify the type of data corresponding to the task selected by the user, and extracts data that corresponds to that type and is actually stored in the data storage unit 121 to create a data list. This allows the user to search for a model that uses data actually registered in the data storage unit 121. In the example of FIG. 5, the user selected the task "demand forecasting" in the task column 41, so a list of data used in demand forecasting is displayed in the data column 42. Each piece of data listed in the data column 42 corresponds to a data candidate.

[0036] A sort button 42a and a narrow down button 42b are also displayed in the data column 42. By operating the sort button 42a, the user can change the display order of the data displayed in the data column 42. For example, the user can change the display order of the data to order by date and time, newest or oldest data, or type of data.

[0037] Furthermore, the user can narrow down the data displayed in the data column 42 by operating the narrow down button 42b. For example, the user can narrow down the data by specifying the data period. Furthermore, when different types of data are displayed, such as image data, table data, and time-series data, the user can narrow down the data by specifying the type of data.

[0038] Furthermore, attribute information prepared for each data item can be displayed in the data column 42. For example, as shown in the example of FIG. 5, when a user selects the data item "Data202104," attribute information 42d for that data item is displayed as additional information in the form of a speech bubble or the like. This allows the user to select data after checking the attribute information for each data item.

[0039] The data column 42 also has an OK button 42c, and after the user selects data in the list with the cursor C or the like, the user presses the OK button 42c to confirm the selection.

[0040] When the user selects data from the data field 42 and presses the OK button 42c to confirm the selection, a list of models is displayed in the model field 43. The list of models displayed here is a list of models that use the data selected by the user in the data field 42. Here, the models that use the data selected by the user include, as an example, models that are applicable to the data selected by the user. As another example, the models that use the data selected by the user may include models that have been trained with the data selected by the user. Furthermore, the models that use the data selected by the user may include both models that are applicable to the data selected by the user and models that have been trained with the data selected by the user.

[0041] Specifically, the model search unit 123 references the correspondence information described above, extracts models that correspond to the data selected by the user and that are actually stored in the model storage unit 122, and creates a list of models. This allows the user to search for models that are actually registered in the model storage unit 122, i.e., models that can be used immediately. In the example of FIG. 5, the user has selected the data "Data202104" in the data column 42, and a list of models that use that data is displayed in the model column 43. Each model listed in the model column 43 corresponds to a candidate model.

[0042] A sort button 43a and a narrow down button 43b are also displayed in the model column 43. By operating the sort button 43a, the user can change the display order of the models displayed in the model column 43. For example, the user can change the display order of the models to the order of newest to oldest models (in order of version or update date). Furthermore, the model storage unit 122 stores attributes of each model as attribute information, specifically, characteristics such as accuracy, interpretability, fairness, and operation cost. Therefore, the user can sort the models based on these attributes.

[0043] Furthermore, the user can narrow down the models displayed in the model column 43 by operating the narrow down button 43b. For example, the user can narrow down the models by specifying whether the model is new or old (the version or the time of update). As described above, the user can also narrow down the models based on attributes such as accuracy, interpretability, fairness, and operation cost. Here, "accuracy" refers to the probability that the model output is correct. "Interpretability" refers to the ability to explain the basis for predictions and inferences made by the model, and to understand them as human-understandable. "Fairness" refers to the absence of bias in predictions and inferences made by the model based on the characteristics of the target. "Operation cost" refers to the cost required when actually using the model for operation.

[0044] Furthermore, the model column 43 can display attribute information prepared for each model. For example, when a user selects the model "LGBM" as in the example of FIG. 5, attribute information 43d of that model is displayed as additional information in the form of a speech bubble or the like. This allows the user to select a model after checking the attribute information of each model. When a user selects a model, related models of that model may also be displayed as additional information. FIG. 6 shows an example in which additional information 43e indicating related models for the model "LGBM" selected by the user is displayed. Related models include models that have undergone backward compatibility adjustments for the model selected by the user, as well as newer and older versions of that model.

[0045] The model column 43 also has an OK button 43c, and the user can select a model from the list using the cursor C or the like, and then press the OK button 43c to confirm the selection.

[0046] In the above example, additional information is displayed when the user selects data or a model using cursor C, and the selection is confirmed when the user presses OK button 42c or 43c. Alternatively, the OK button may not be displayed on the model search screen, and additional information may be displayed when the user places cursor C over the text of data or a model (so-called mouse over), and the selection may be confirmed when the user selects data or a model.

[0047] As described above, according to the model search screen of this embodiment, the user selects the target task and then the data to be used, and candidate data and models corresponding to the user's selection are displayed in a list. This allows the user to efficiently search for a model suitable for the target task. Furthermore, by selecting a task or data, candidate models suitable for those are displayed in a list, so even a user with little knowledge of models can easily search for a model suitable for the target task and the actual data that has already been acquired.

[0048] [Model search process] Next, a description will be given of the model search processing executed by the model search device 100. Fig. 7 is a flowchart of the model search processing. This processing is realized by the processor 112 shown in Fig. 2 executing a program prepared in advance and operating mainly as the model search unit 123.

[0049] First, the model search unit 123 displays an initial screen of the model search screen on the display device 2 (step S10). On the initial screen, the model search unit 123 displays a default task list in the task column 41, and leaves the data column 42 and the model column 43 blank.

[0050] Next, the model search unit 123 determines whether or not the user has selected a task (step S11). In this case, the selection of a task refers to a selection confirmed by the user by, for example, pressing the OK button 41c. If no task has been selected (step S11: No), the model search unit 123 waits for a selection by the user. On the other hand, if a task has been selected (step S11: Yes), the model search unit 123 refers to the correspondence information and data storage unit 121 described above, and displays a list of data corresponding to the selected task on the model search screen (step S12).

[0051] Next, the model search unit 123 determines whether or not the user has selected data (step S13). Note that the selection of data in this case refers to the user confirming the selection by, for example, pressing the OK button 42c. If no data has been selected (step S13: No), the model search unit 123 waits for a selection by the user. On the other hand, if data has been selected (step S13: Yes), the model search unit 123 refers to the correspondence information and model storage unit 122 described above, and displays a list of models corresponding to the selected data on the model search screen (step S14).

[0052] Next, the model search unit 123 determines whether or not the user has selected a model (step S15). In this case, the selection of a model refers to the user confirming the selection by pressing the OK button 43c, for example. If a model has not been selected (step S15: No), the model search unit 123 waits for the user to make a selection. On the other hand, if a model has been selected (step S15: Yes), the model search unit 123 determines the model selected by the user as the search result (step S16) and ends the process.

[0053] The model search unit 123 accepts changes to the task and data until the user finalizes the model selection. This allows the user to change the task and data selection to check various patterns and find the optimal model. The user can also obtain data to be used for the selected model from the data storage unit 121 and obtain the selected model from the model storage unit 122, thereby preparing the data and model to be used for the target task.

[0054] [Variations] In the above embodiment, the model search device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured by a server and a terminal device. FIG. 8 is a block diagram showing a schematic configuration of a model management system 1x using a server and a terminal device. In FIG. 8, the server 100x has the configuration of the model search device 100 shown in FIG. 3. In addition, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG. 3.

[0055] Second Embodiment 9 is a block diagram showing the functional configuration of a model search device 70 according to the second embodiment. The model search device 70 includes a task list output means 71, a data list output means 72, a model list output means 73, and a model output means 74.

[0056] 10 is a flowchart of processing by the model search device 70 of the second embodiment. Task list output means 71 outputs a task list indicating candidate tasks (step S71). Data list output means 72 receives a selection of a task included in the task list, and outputs a data list indicating candidate data to be used in the selected task (step S72). Model list output means 73 receives a selection of data included in the data list, and outputs a model list indicating candidate models that use the selected data (step S73). Model output means 74 receives a selection of a model included in the model list, and outputs the selected model (step S74).

[0057] According to the model search device 70 of the second embodiment, it becomes possible for a user to easily search for a model for achieving a target task.

[0058] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0059] (Appendix 1) a task list output means for outputting a task list showing candidate tasks; a data list output means for receiving a selection of a task included in the task list and outputting a data list showing candidates of data to be used in the selected task; a model list output means for receiving a selection of data included in the data list and outputting a model list showing candidate models that use the selected data; a model output means for receiving a selection of a model included in the model list and outputting the selected model; A model search device comprising:

[0060] (Appendix 2) 2. The model search device according to claim 1, wherein the data list output means rearranges the order of the data in the data list or narrows down the data in the data list based on an instruction from a user.

[0061] (Appendix 3) 3. The model search device according to claim 1, wherein the data list includes attribute information relating to the data included in the data list.

[0062] (Appendix 4) 4. The model search device according to claim 1, wherein the model list output means rearranges the order of models in the model list or narrows down the models in the model list based on an instruction from a user.

[0063] (Appendix 5) 5. The model search device according to claim 4, wherein the model list output means performs the sorting or the narrowing down based on at least one of the accuracy, interpretability, fairness, and operation cost of the model.

[0064] (Appendix 6) 6. The model search device according to claim 1, wherein the model list includes attribute information about the models included in the model list.

[0065] (Appendix 7) 7. The model search device according to claim 1, wherein the model list includes related models that are related to the models included in the model list.

[0066] (Appendix 8) 8. The model search device according to claim 1, wherein the data list includes actual data stored in a data storage unit.

[0067] (Appendix 9) Output a task list showing candidate tasks, receiving a selection of a task included in the task list, and outputting a data list indicating candidates of data to be used in the selected task; receiving a selection of data included in the data list, and outputting a model list indicating candidate models that use the selected data; A model search method that receives a selection of a model to be included in the model list and outputs the selected model.

[0068] (Appendix 10) Output a task list showing candidate tasks, receiving a selection of a task included in the task list, and outputting a data list indicating candidates of data to be used in the selected task; receiving a selection of data included in the data list, and outputting a model list indicating candidate models that use the selected data; A recording medium having recorded thereon a program for causing a computer to execute a process of receiving a selection of a model included in the model list and outputting the selected model.

[0069] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0070] 1. 1x Model Management System 2, 2x display device 3. 3x input devices 7 Terminal Equipment 100 Model Search Device 112 processors 121 Data storage unit 122 Model Memory Unit 123 Model Search Unit

Claims

1. a task list output means for outputting a task list showing candidate tasks; a data list output means for outputting a data list indicating candidates of the data associated with a task selected by a user from the output task list, by referring to correspondence information indicating correspondence between tasks, data, and trained models; a model list output means for outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, by referring to the correspondence information; a model output means for outputting the trained model selected by the user from the output model list; A model search device comprising:

2. 2. The model search device according to claim 1, wherein the data list output means rearranges the order of the data in the data list or narrows down the data in the data list based on an instruction from the user.

3. 3. The model search device according to claim 1, wherein the data list includes attribute information relating to the data included in the data list.

4. 4. The model search device according to claim 1, wherein the model list output means rearranges the order of trained models in the model list or narrows down the trained models in the model list based on an instruction from the user.

5. The model search device according to claim 4 , wherein the model list output means performs the sorting or the narrowing down based on at least one of accuracy, interpretability, fairness, and operation cost of the trained models.

6. The model search device according to claim 1 , wherein the model list includes attribute information about trained models included in the model list.

7. The model search device according to claim 1 , wherein the model list includes related models that are related to the trained models included in the model list.

8. 8. The model search device according to claim 1, wherein the data list includes actual data stored in a data storage unit.

9. A model search device according to claim 1, wherein the task list output means outputs a major classification of tasks and a minor classification of tasks corresponding to the major classification.

10. 1. A computer-implemented method for model exploration, comprising: Output a task list showing candidate tasks, outputting a data list indicating candidates for the data associated with a task selected by a user from the output task list, with reference to correspondence information indicating correspondence between tasks, data, and trained models; outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, with reference to the correspondence information; A model search method that outputs the trained model selected by the user from the output model list.

11. Output a task list showing candidate tasks, outputting a data list indicating candidates for the data associated with a task selected by a user from the output task list, with reference to correspondence information indicating correspondence between tasks, data, and trained models; outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, with reference to the correspondence information; A program that causes a computer to execute a process of outputting the trained model selected by the user from the output model list.

Images