Model analysis device, model analysis method, and program

Abstract

Provided is a model analysis device wherein an index acquisition means acquires designation of a plurality of evaluation indexes relating to a model. An importance acquisition means acquires degrees of importance for the plurality of evaluation indexes. An index value acquisition means acquires the values of the plurality of evaluation indexes for a plurality of models. An analysis means analyzes the ranking of the plurality of models on the basis of the acquired evaluation index values and degrees of importance.

Description

[Technical Field]

[0001] The present disclosure relates to the analysis of machine learning models. [Background technology]

[0002] In recent years, predictive models obtained by machine learning have been used in various fields. Patent Document 1 relates to a system for learning and / or optimizing processes related to semiconductor manufacturing, and describes a method for ranking candidate process models based on quality values ​​and diversity values. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-504872 Summary of the Invention [Problem to be solved by the invention]

[0004] When selecting a predictive model to use in machine learning, it is desirable to evaluate the predictive model by comparing and considering multiple evaluation indicators depending on the task to which the predictive model will be applied, and select an appropriate predictive model.

[0005] One object of the present disclosure is to evaluate models and select an appropriate model by taking into account multiple evaluation indicators. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a model analysis apparatus includes: index acquisition means for acquiring a designation of a plurality of evaluation indices for the model; an importance obtaining means for obtaining importance for the plurality of evaluation indexes; index value acquisition means for acquiring values ​​of the plurality of evaluation indexes for a plurality of models; an analysis means for analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance; a display means for displaying an evaluation index space for the plurality of models based on the values ​​of the plurality of evaluation indexes; Equipped with the evaluation index space has the plurality of evaluation indexes as its coordinate axes, each of the plurality of models is embedded in the evaluation index space at a position corresponding to the value of the evaluation index of the model; The evaluation index sky A plurality of line segments are drawn on the space, where the plurality of evaluation indexes are at the same level.

[0007] In another aspect of the present disclosure, a computer-implemented model analysis method includes: Obtaining multiple evaluation metric specifications for the model; Obtaining the importance of the plurality of evaluation indexes; Obtaining values ​​of the evaluation indexes for a plurality of models; Analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance; displaying an evaluation index space for the plurality of models based on the values ​​of the plurality of evaluation indexes; the evaluation index space has the plurality of evaluation indexes as its coordinate axes, each of the plurality of models is embedded in the evaluation index space at a position corresponding to the value of the evaluation index of the model; The evaluation index sky A plurality of line segments are drawn on the space, where the plurality of evaluation indexes are at the same level.

[0008] In yet another aspect of the invention, a program includes: Obtaining multiple evaluation metric specifications for the model; Obtaining the importance of the plurality of evaluation indexes; Obtaining values ​​of the evaluation indexes for a plurality of models; Analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance; causing a computer to execute a process of displaying an evaluation index space for the plurality of models based on values ​​of the plurality of evaluation indexes; the evaluation index space has the plurality of evaluation indexes as its coordinate axes, each of the plurality of models is embedded in the evaluation index space at a position corresponding to the value of the evaluation index of the model; The evaluation index sky A plurality of line segments are drawn on the space, where the plurality of evaluation indexes are at the same level. [Effects of the Invention]

[0009] According to the present disclosure, it becomes possible to evaluate models and select appropriate models by taking multiple evaluation indicators into consideration. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the model generating device. [Figure 3] FIG. 1 is a block diagram showing a functional configuration of a model generation device according to a first embodiment. [Figure 4] 10 shows an example of displaying evaluation information according to the first embodiment. [Figure 5] 10 shows another example of displaying evaluation information according to the first embodiment. [Figure 6] 10 is a flowchart of a model analysis process according to the first embodiment. [Figure 7] FIG. 1 is a block diagram showing a schematic configuration of a model generation system using a server and a terminal device. [Figure 8] 10 shows an example of displaying evaluation information according to the second embodiment. [Figure 9] 10 is a flowchart of a process performed by a model analysis device according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing the functional configuration of a model analysis device according to a third embodiment. [Figure 11]10 is a flowchart of a process performed by a model analysis device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Overall configuration] FIG. 1 is a block diagram showing the overall configuration of a model generation system according to the first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is an application of the model analysis device of the present disclosure, and is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by the user to give instructions and input necessary when modifying a model or viewing the evaluation information.

[0012] First, the operation of the model generation system 1 will be briefly described. The model generation device 100 generates a machine learning model (hereinafter simply referred to as a "model") using training data prepared in advance. The model generation device 100 also analyzes and evaluates the generated model. Specifically, the model generation device 100 performs model prediction using evaluation data and evaluates the model based on the prediction result. The model generation device 100 evaluates multiple models using multiple evaluation indices. The evaluation indices include various indices such as the prediction accuracy, compatibility, and fairness of the model. Note that prediction accuracy indicates the probability that the prediction result of a model is correct. Compatibility indicates the degree of agreement between the prediction results of one model and another model. Furthermore, fairness indicates the degree of difference or variation between the prediction results of models for data with different attributes. The model generation device 100 then presents the results of evaluating the multiple models in relation to the multiple evaluation indices to the user as evaluation information. The user can check the evaluation information and select an appropriate model from the multiple models, or operate the input device 3 to input correction information for correcting the model.

[0013] A "machine learning model" is information that represents the relationship between explanatory variables and a response variable. A machine learning model is, for example, a component that estimates the results of an estimation target by calculating a response variable based on explanatory variables. A machine learning model is generated by executing a learning algorithm using training data, for which the response variable value has already been obtained, and arbitrary parameters as input. A machine learning model may be represented, for example, by a function c that maps an input x to a correct answer y. A machine learning model may estimate a numerical value of an estimation target, or may estimate a label of an estimation target. A machine learning model may output a variable that describes the probability distribution of a response variable. A machine learning model may also be referred to as a "learning model," an "analysis model," an "AI (artificial intelligence) model," or a "prediction formula." An explanatory variable is a variable used as input in a machine learning model. An explanatory variable may also be referred to as a "feature" or "characteristic."

[0014] Furthermore, the learning algorithm for generating the machine learning model is not particularly limited and may be an existing learning algorithm. For example, the learning algorithm may be a random forest, a support vector machine, a naive Bayes, a piecewise linear model using FAB inference (Factorized Asymptotic Bayesian Inference), or a neural network. Note that a piecewise linear model technique using FAB inference is disclosed, for example, in U.S. Patent Publication US2014 / 0222741A1.

[0015] [Hardware configuration] 2 is a block diagram showing the hardware configuration of model generation device 100. As shown in the figure, model generation device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0016] The I / F 111 inputs and outputs data to and from external devices. Specifically, training data and evaluation data used in generating a model, as well as instructions and inputs entered by a user using the input device 3, are input to the model generation device 100 via the I / F 111. Furthermore, evaluation information for a model generated by the model generation device 100 is output to the display device 2 via the I / F 111.

[0017] The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generation device 100 by executing a prepared program. The processor 112 may be a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, or an FPGA (Field-Programmable Gate Array). The processor 112 executes the model analysis process described below.

[0018] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0019] The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 records various programs to be executed by the processor 112. When the model generation device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0020] DB 115 stores information about the model generated by model generation device 100 (hereinafter referred to as the "existing model") and the model corrected by retraining (hereinafter referred to as the "corrected model"). DB 115 also stores training data, evaluation data, correction information input by the user, etc., input via I / F 111 as necessary.

[0021] [Function Configuration] 3 is a block diagram showing the functional configuration of the model generation device 100 according to the first embodiment. Functionally, the model generation device 100 includes a training data DB 121, a model training unit 122, a model DB 123, an evaluation data DB 124, and an analysis unit 125.

[0022] The training data DB 121 stores training data used to generate a model. The training data D1 is input to the model training unit 122. The training data D1 is composed of multiple combinations of input data and correct answers (correct values ​​or correct labels) for the input data.

[0023] The model training unit 122 trains a model using the training data D1 and generates a model. The model training unit 122 outputs model data M corresponding to the generated model to the model DB 123 and the analysis unit 125. The model data M includes multiple pieces of parameter information constituting the model. The parameter information includes, for example, information on explanatory variables used as input to the model, information on weights for each explanatory variable, and information on weights for each sample constituting the input data.

[0024] The model DB 123 stores multiple models for each task. A "task" refers to a type of prediction or inference made by a machine learning model based on a problem or question and available data. Tasks include classification tasks, regression tasks, clustering tasks, object recognition tasks, and various other tasks. For example, a classification task involves assigning data to a category. A regression task involves predicting the value of a target variable based on the values ​​of other explanatory variables. A clustering task involves grouping data according to similarity. An object recognition task involves detecting objects in an image and outputting their location and class. The model DB 123 stores multiple models for each task. Note that the multiple models corresponding to a given task may use different algorithms, may be different models obtained using the same algorithm on different data, or may be different models obtained using the same algorithm with different settings (e.g., hyperparameters).

[0025] The model training unit 122 retrains the existing model to generate a modified model. In this case, the model training unit 122 modifies the parameters that configure the model based on modification information D3 input by the user using the input device 3, and retrains the model using training data for retraining as necessary. The model training unit 122 stores model data M of the modified model obtained by the retraining in the model DB 123 and outputs it to the analysis unit 125.

[0026] Furthermore, the method by which the model training unit 122 retrains the model is not particularly limited. For example, the model training unit 122 may use a method of retraining a completely new, modified model while ignoring the existing model, or a method of updating the existing model using new training data. Other methods by which the model training unit 122 retrains the existing model include a method of updating only the coefficients by retraining without changing the types of explanatory variables used in the model, and a method of retraining that also includes the selection of explanatory variables.

[0027] The evaluation data DB 124 stores evaluation data used to evaluate the generated model. The evaluation data may be, for example, a portion of collected data that was not used as training data, newly collected data, or validation data prepared for verification. Training data may also be used as evaluation data. The evaluation data is composed of multiple combinations of input data and correct answers (correct values ​​or correct labels) for the input data.

[0028] The analysis unit 125 analyzes the performance of multiple models using the evaluation data. Specifically, the analysis unit 125 acquires multiple models corresponding to a task selected by the user from the model DB 123, and calculates evaluation indexes for each model using the evaluation data. In the following description, the analysis unit 125 calculates an accuracy index and a compatibility index for each model as multiple evaluation indexes specified by the user. The analysis unit 125 then evaluates each model in relation to the accuracy index and the compatibility index, and outputs the result to the display device 2 as evaluation information D2. The analysis unit 125 is an example of an index acquisition means, an importance acquisition means, an index value acquisition means, and an analysis means.

[0029] The display device 2 displays the evaluation information D2 output by the analysis unit 125 on the display device 2. By looking at the displayed evaluation information, the user can evaluate multiple models from the perspective of the accuracy index and the compatibility index.

[0030] Furthermore, the user inputs correction information D3 for correcting the current model to the input device 3 as necessary. The correction information D3 is information related to the correction, such as information on explanatory variables used as input to the model, information on weights for each explanatory variable, information on weights set for each attribute of the category of explanatory variables, and information on weights for each sample constituting the input data. The model training unit 122 corrects the model by retraining the model using the input correction information D3.

[0031] [Example of evaluation information display] In the first embodiment, the analysis unit 125 ranks and displays a plurality of models corresponding to one task based on the importance designated by the user for a plurality of evaluation indexes.

[0032] Fig. 4 shows an example of a display of evaluation information according to the first embodiment. The display example 40 in Fig. 4 includes a model list 41 and a slide bar 42. The model list 41 displays a list of multiple models, and includes candidate models, accuracy indicators, and compatibility indicators. "Candidate models" indicates multiple models corresponding to one task. In the example in Fig. 4, four candidate models h1 to h4 corresponding to one task are displayed.

[0033] The "accuracy index" is an index that indicates the accuracy of each candidate model. Essentially, the accuracy index indicates the probability that the model output is correct. For example, in the case of a regression task, the accuracy index is R 2The coefficient of determination (COF), RMSE (Root Means Square Error), and MAE (Mean Absolute Error) can be used. However, for indicators where a smaller value indicates better accuracy, it can be multiplied by "-1" so that a larger value indicates better accuracy. Specifically, for RMSE and MSE, the value multiplied by "-1" can be used as the indicator. For classification tasks, accuracy indicators include accuracy, precision, recall, F-score (F1-score), and AUC (Area under the ROC curve). The "compatibility index" indicates the compatibility between multiple models, specifically, the degree to which another model hx can correctly interpret data that was correctly interpreted by a certain model h0.

[0034] 4, the analysis unit 125 uses a "compatibility index" and an "accuracy index" as evaluation indices. However, the evaluation indices used by the analysis unit 125 are not limited to these, and any evaluation index can be used. For example, the analysis unit 125 may analyze the performance of the model using a "fairness index" as an evaluation index.

[0035] The slide bar 42 is a bar for specifying the ratio of the importance (weight) of the accuracy index and the compatibility index, and is operated by the user. Specifically, when the cursor 42a is at the left end of the slide bar 42, the importance of the accuracy index is 100% and the importance of the compatibility index is 0%. On the other hand, when the cursor 42a is at the right end of the slide bar 42, the importance of the accuracy index is 0% and the importance of the compatibility index is 100%. Furthermore, when the cursor 42a is in the center of the slide bar 42, the importance of the accuracy index is 50% and the importance of the compatibility index is 50%. By specifying the position of the cursor 42a on the slide bar 42, the user can specify how much importance to assign to either the accuracy index or the compatibility index.

[0036] 4, it is assumed that the position of cursor 42a corresponds to an accuracy index importance of 80% and a compatibility index importance of 20%. Analysis unit 125 first calculates the accuracy index value and the compatibility index value for each of candidate models h1 to h4 using evaluation data, etc. Note that the larger the accuracy index value, the higher the accuracy, and the larger the compatibility index value, the higher the compatibility.

[0037] Next, the analysis unit 125 ranks the candidate models using a value obtained by aggregating the evaluation indices using the ratio of importance specified by the user (hereinafter also referred to as an "aggregated evaluation value"). This aggregated evaluation value is a value that indicates an overall evaluation of multiple evaluation indices based on the importance specified by the user. Therefore, by ranking multiple candidate models using the aggregated evaluation value, the user can select an appropriate candidate model under the conditions of the importance specified by the user.

[0038] In a preferred example, the analysis unit 125 aggregates the evaluation indexes using a linear function and uses a weighted sum as the aggregated evaluation value. In this case, the analysis unit 125 calculates the weighted sum of the accuracy index value and the compatibility index value for each candidate model using the ratio of the importance of the accuracy index to the compatibility index specified by the user (80%:20%=4:1) as a weight. For example, the analysis unit 125 calculates the weighted sum S for each candidate model as follows: S=4×(accuracy index)+1×(compatibility index) (1)

[0039] The analysis unit 125 calculates the weighted sum for all candidate models h1 to h4 using the above formula (1), ranks the candidate models in descending order of the weighted sum S, and displays them as a list. As a result, in the example of Fig. 4, the user can see that model h3 is the most appropriate in terms of the ratio of the importance of the accuracy index and the compatibility index (80%:20%) specified by the user.

[0040] In this way, the analysis unit 125 displays a list of multiple candidate models according to an aggregate evaluation value that aggregates the accuracy index and compatibility index using the importance level specified by the user, allowing the user to easily know which candidate model to use based on the importance level specified by the user.

[0041] FIG. 5 shows another example of displaying evaluation information according to the first embodiment. This example also shows accuracy indices and compatibility indices for multiple models corresponding to a single task. However, while in the example of FIG. 4 the user specifies the importance using a slide bar 42, in the example of FIG. 5 the user specifies the importance using a bar graph. The display example 50 of FIG. 5 includes a model list 51 and a bar graph 52. The model list 41, like the first example, displays a list of multiple models. However, in the example of FIG. 5, two accuracy indices and two compatibility indices selected by the user are used. Regarding the compatibility indices, for example, when there are two models, a new and an old, compatibility index 1 can be the percentage of data that the old model correctly answered that also correctly answered the new model. Furthermore, when there are two models, a new and an old, compatibility index 2 can be the percentage of data that the new model incorrectly answered that also correctly answered the old model.

[0042] The bar graph 52 includes a bar 52a for each evaluation index. The user can change the importance of each evaluation index by changing the length of the bar 52a corresponding to that evaluation index using a cursor C or the like. In the example of FIG. 5, the user changes the importance of accuracy index 1 using cursor C. If there is no room to extend the bar 52a due to the display area of ​​the display example 50, the user can press a renormalization button 52b to fit the length of each bar 52a in the display example 50 within the display area without changing the importance ratio between the multiple evaluation indexes. Furthermore, as shown in FIG. 5, the ratio of the length of each bar 52a to the total (total of four bars) is displayed below each bar 52a, allowing the user to fine-tune the importance ratio for each evaluation index.

[0043] The method by which the analysis unit 125 ranks the multiple candidate models is the same as in the first example. That is, the analysis unit 125 uses the importance ratio specified by the user to calculate the weighted sum S of each evaluation index (in this example, accuracy index 1, accuracy index 2, compatibility index 1, and compatibility index 2) as an aggregate evaluation value, and displays the multiple candidate models in that order in the model list 51. In this example as well, the user can easily know which candidate model is optimal for the importance ratio specified by the user.

[0044] [Model analysis processing] 6 is a flowchart showing the model analysis process according to the first embodiment. The model analysis process is a process for displaying a plurality of models corresponding to a task specified by a user, ranked according to the importance of the evaluation index specified by the user. This process is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating mainly as the model training unit 122 and the analysis unit 125 shown in FIG. 3.

[0045] First, the analysis unit 125 acquires the user's designation of a task and evaluation indexes (step S10). Fig. 4 shows an example in which the user designates a task and designates an accuracy index and a compatibility index as evaluation indexes. Fig. 5 shows an example in which the user designates a task and designates accuracy index 1, accuracy index 2, compatibility index 1, and compatibility index 2 as evaluation indexes.

[0046] Next, the analysis unit 125 acquires a plurality of candidate models corresponding to the task designated by the user (step S11). In the examples of Fig. 4 and Fig. 5, the analysis unit 125 acquires candidate models h1 to h4.

[0047] Next, the analysis unit 125 determines a display order based on one of the multiple evaluation indexes as a default display (step S12), and displays a list of multiple candidate models in the determined display order (step S13). Note that the one evaluation index in this case may be, for example, an evaluation index specified by the user from the multiple evaluation indexes, or the evaluation index specified first by the user may be used. As a result, in the default display, the multiple candidate models are listed in descending order of the one evaluation index.

[0048] Next, the analysis unit 125 determines whether the user has changed the importance (step S14). Specifically, in the example of FIG. 4, the analysis unit 125 determines whether the user has changed the position of the cursor 42a on the slide bar 42. Furthermore, in the example of FIG. 5, the analysis unit 125 determines whether the user has changed the length of the bar 52a of any evaluation index. If the user has changed the importance (step S14: Yes), the analysis unit 125 calculates the weighted sum S described above based on the changed importance, and determines the display order of multiple candidate models in the order of the weighted sum S (step S15). Then, the analysis unit 125 redisplays the model list in the determined display order (step S13). As a result, the model list 41 shown in FIG. 4 or the model list 51 shown in FIG. 5 is displayed.

[0049] On the other hand, if the user has not changed the importance (step S14: No), the analysis unit 125 determines whether or not the user has input an end instruction (step S16). If the end instruction has not been input (step S16: No), the process returns to step S14. On the other hand, if the end instruction has been input (step S16: Yes), the process ends.

[0050] In the above flowchart, the display order of multiple candidate models is determined based on any one evaluation index as a default display in step S12. Alternatively, the user may be prompted to input importance from the beginning, and the display order of multiple candidate models may be determined and displayed based on the input importance.

[0051] As described above, in the model analysis process of the first embodiment, an aggregate evaluation value for multiple evaluation indexes is calculated based on multiple evaluation indexes specified by the user and their importance, and multiple candidate models corresponding to the task specified by the user are ranked and displayed based on the aggregate evaluation value. Thus, the user can easily know the optimal candidate model for the importance level specified by the user.

[0052] [Variations] (Variation 1) In the above model analysis process, the analysis unit 125 aggregates multiple evaluation indexes using a linear function that indicates a weighted sum, but instead, it may also be possible to aggregate multiple evaluation indexes using other linear or nonlinear functions and calculate an aggregate evaluation value.

[0053] (Variation 2) In the above model analysis process, the importance ratio specified by the user may be associated with the task and learned. This makes it possible to generate a machine learning model that has learned the relationship between the task and the importance ratio that the user considers appropriate for that task. By using this machine learning model, the user can input a task and obtain an appropriate importance ratio for that task as output.

[0054] (Variation 3) In the first embodiment, the analysis unit 125 ranks and displays a list of multiple models corresponding to a task specified by the user. Alternatively, the analysis unit 125 may rank and display any multiple models specified by the user, regardless of the identity of the task.

[0055] (Variation 4) In the first embodiment, the analysis unit 125 acquires multiple models corresponding to the task selected by the user from the model DB 123 and calculates the evaluation index of each model using the evaluation data. Alternatively, a function for calculating the evaluation index of each model may be provided externally. For example, the analysis unit 125 may acquire the evaluation index of each model from a DB that stores the evaluation index of each model.

[0056] (Variation 5) In the above embodiment, the model generation device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured by a server and a terminal device. FIG. 7 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 7, the server 100x has the configuration of the model generation device 100 shown in FIG. 3. Furthermore, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG. 3.

[0057] Second Embodiment Next, a second embodiment will be described. The overall configuration of the model generation system 1 according to the second embodiment, and the hardware configuration and functional configuration of the model generation device 100 are basically the same as those of the first embodiment, and therefore description thereof will be omitted.

[0058] [Example of evaluation information display] In the second embodiment, the analysis unit 125 displays the relationship between multiple models corresponding to one task based on multiple evaluation indexes. Specifically, the analysis unit 125 embeds each model in a space based on the values ​​of multiple evaluation indexes specified by the user, and draws in the space an area of ​​models that can be considered to be at the same level in relation to the multiple evaluation indexes and the importance specified by the user.

[0059] FIG. 8 shows an example of displaying evaluation information according to the second embodiment. Display example 60 in FIG. 8 is obtained by embedding each model in a space (hereinafter also referred to as "evaluation index space") in which the compatibility index and accuracy index specified by the user are the X-axis and Y-axis, respectively. As in the first embodiment, the analysis unit 125 acquires multiple models corresponding to one task specified by the user, and calculates the values ​​of the compatibility index and accuracy index specified by the user for those models. Then, the analysis unit 125 embeds each model in the evaluation index space based on the values ​​of the compatibility index and accuracy index of each model. This results in display example 60.

[0060] Next, the analysis unit 125 draws line segments L1 to L5 in the evaluation index space where the two evaluation indexes are at the same level based on the aforementioned aggregated evaluation value, and creates a display example 61. Now, let us assume that the weighted sum S is used as the aggregated evaluation value. If the importance (weight) specified by the user for the accuracy index is w1 and the importance (weight) specified for the compatibility index is w2, then the weighted sum S is expressed by the following formula: S=w1×(accuracy)+w2×(compatibility) (2)

[0061] The analysis unit 125 sets the value of the weighted sum S to a predetermined value S1 and draws a line segment L1 in the evaluation index space. Similarly, the analysis unit 125 changes the value of the weighted sum S to S2, S3, ... by an arbitrary amount and draws line segments L2 to L5 in the evaluation index space. Furthermore, the analysis unit 125 displays areas A1 to A5 defined by line segments L1 to L5 in different colors or the like to distinguish them from one another.

[0062] Here, line segments L1 to L5 indicate the position of the model in the evaluation index space when the user-specified importance (w1 / w2) and the aggregate evaluation value S of multiple evaluation indexes are fixed, and the values ​​of the accuracy index and compatibility index are changed. Therefore, for example, if two models M1 and M2 are on line segment L1, they satisfy the ratio of the importance of the accuracy index and the compatibility index specified by the user, and are models with the same aggregate evaluation value S, which indicates the overall evaluation of multiple evaluation indexes, and can be considered to be models of the same level in terms of the importance specified by the user. The same is true for the other line segments L2 to L5. In other words, line segments L1 to L5 can be considered as contour lines indicating the position of models in the evaluation index space that satisfy the importance specified by the user and have the same overall evaluation.

[0063] Therefore, area A2 defined by line segments L1 and L2 in the evaluation index space can be considered an area indicating the location of models that largely satisfy the importance specified by the user and have roughly the same level of performance. The same applies to the other areas A1, A3 to A5. Therefore, the user can know that multiple models that belong to the same area in display example 61 are roughly at the same level in terms of the importance specified by the user. For example, the user can recognize that models h1 and h2 that belong to area A2 are roughly at the same level, and models h3 and h4 that belong to area A3 are roughly at the same level.

[0064] In this way, in the second embodiment, by displaying multiple models in the evaluation index space, the user can easily visually grasp models that are at the same level in terms of the importance of the evaluation index specified by the user. In particular, by specifying two evaluation indexes that are considered to have a large trade-off and performing the above display, the user can find a model that can exhibit appropriate performance by skillfully adjusting the two conflicting evaluation indexes.

[0065] [Model analysis processing] 9 is a flowchart showing the model analysis process according to the second embodiment. The model analysis process is a process for displaying multiple models corresponding to a task selected by a user in an evaluation index space according to the importance of the evaluation index specified by the user. This process is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating mainly as the model training unit 122 and the analysis unit 125 shown in FIG. 3.

[0066] First, the analysis unit 125 acquires a task by a user, multiple evaluation indices, and the importance of those evaluation indices (step S10). Fig. 8 shows an example in which a user specifies a task and specifies an accuracy index and a compatibility index as evaluation indices.

[0067] Next, the analysis unit 125 acquires multiple candidate models corresponding to the task specified by the user (step S11). In the example of FIG. 8, the analysis unit 125 acquires models h1 to h5. Next, the analysis unit 125 calculates the value of the evaluation index for each model and embeds it into an evaluation index space (step S22). In the example of FIG. 8, the analysis unit 125 calculates the accuracy index and compatibility index for models h1 to h5, and embeds each of models h1 to h5 in the evaluation index space as shown in display example 60.

[0068] Next, the analysis unit 125 draws contour lines L1 to L5 using the importance levels specified by the user and the values ​​of each evaluation index. In the example of Fig. 8, the analysis unit 125 draws contour lines L1 to L5 by fixing the ratio (w1 / w2) of the importance levels of each evaluation index specified by the user in the above formula (2), changing the value of the weighted sum S within a predetermined range, and varying the accuracy index and compatibility index. Then, the analysis unit 125 colors the areas A1 to A5 defined by the contour lines L1 to L5 in different colors, and draws areas of models that can be considered to be at the same level (step S23). Then, the processing ends.

[0069] As described above, in the model analysis process of the second embodiment, multiple models corresponding to a task specified by the user are displayed in the evaluation index space based on multiple evaluation indexes and their importance specified by the user. Also, based on the evaluation indexes and their importance specified by the user, a range of models that can be considered to be at the same level are displayed in the evaluation index space. Therefore, the user can easily know models that can be considered to be at the same level when the user specifies the importance, for example, models that can be used as alternatives.

[0070] [Variations] (Variation 1) In the above example, the analysis unit 125 aggregates two evaluation indexes using a linear function that indicates a weighted sum and plots contour lines on the evaluation index space. Alternatively, the analysis unit 125 may aggregate two evaluation indexes using other linear or nonlinear functions and plot contour lines on the evaluation index space.

[0071] (Variation 2) In the second embodiment, the analysis unit 125 also ranks and lists a plurality of models corresponding to the task specified by the user. Alternatively, the analysis unit 125 may display any plurality of models specified by the user in the evaluation index space, regardless of the identity of the task.

[0072] (Variation 3) In the second embodiment, similarly to the fourth modification of the first embodiment, the analysis unit 125 may obtain the evaluation index of each model from an external source.

[0073] (Variation 4) In the second embodiment, similarly to the fifth modification of the first embodiment, the model generating device may be configured by a server and a terminal device as shown in FIG.

[0074] <Third embodiment> 10 is a block diagram showing the functional configuration of a model analysis device 70 according to the third embodiment. The model analysis device 70 includes an index acquisition unit 71, an importance acquisition unit 72, an index value acquisition unit 73, and an analysis unit 74.

[0075] 11 is a flowchart of processing by the model analysis device 70 of the third embodiment. The index acquisition means 71 acquires the designation of multiple evaluation indexes related to a model (step S71). The importance acquisition means 72 acquires the importance for the multiple evaluation indexes (step S72). The index value acquisition means 73 acquires the values ​​of multiple evaluation indexes for multiple models (step S73). The analysis means 74 analyzes the ranking of the multiple models based on the acquired evaluation index values ​​and importance (step S74).

[0076] According to the model analysis device 70 of the third embodiment, it becomes possible to evaluate models and select appropriate models in consideration of multiple evaluation indexes.

[0077] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0078] (Appendix 1) index acquisition means for acquiring a designation of a plurality of evaluation indices for the model; an importance obtaining means for obtaining importance for the plurality of evaluation indexes; index value acquisition means for acquiring values ​​of the plurality of evaluation indexes for the plurality of models; an analysis means for analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance; A model analysis device comprising:

[0079] (Appendix 2) The model analysis device described in Appendix 1, wherein the analysis means aggregates the values ​​of the multiple evaluation indexes based on the importance to calculate an aggregate evaluation value for each of the models, and outputs information ranking the multiple models according to the aggregate evaluation value.

[0080] (Appendix 3) a task acquisition means for receiving a task specification; 3. The model analysis device according to claim 1, wherein the calculation means calculates values ​​of the evaluation indexes for a plurality of models corresponding to a specified task.

[0081] (Appendix 4) 3. The model analysis device according to claim 1, wherein the importance obtaining means obtains a ratio of importance for the plurality of evaluation indexes.

[0082] (Appendix 5) 3. The model analysis device according to claim 1, wherein the importance obtaining means obtains an importance value for each of the plurality of evaluation indexes.

[0083] (Appendix 6) 6. The model analysis device according to claim 1, wherein the output means calculates a weighted sum of the importance and values ​​of the plurality of evaluation indexes as the aggregate evaluation value.

[0084] (Appendix 7) Obtaining multiple evaluation metric specifications for the model; Obtaining the importance of the plurality of evaluation indexes; acquiring values ​​of the evaluation indexes for the plurality of models; A model analysis method that analyzes the ranking of the plurality of models based on the acquired evaluation index values ​​and importance.

[0085] (Appendix 8) Obtaining multiple evaluation metric specifications for the model; Obtaining the importance of the plurality of evaluation indexes; acquiring values ​​of the evaluation indexes for the plurality of models; A recording medium storing a program for causing a computer to execute a process for analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance.

[0086] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0087] 1. 1x Model Generation System 2, 2x display device 3. 3x input devices 7 Terminal Equipment 100 Model generation device 112 processors 121 Training Data DB 122 Model Training Department 123 Model DB 124 Evaluation Data DB 125 Analysis Department

Claims

1. index acquisition means for acquiring a designation of a plurality of evaluation indices for the model; an importance obtaining means for obtaining importance for the plurality of evaluation indexes; index value acquisition means for acquiring values ​​of the plurality of evaluation indexes for a plurality of models; an analysis means for analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance; a display means for displaying an evaluation index space for the plurality of models based on the values ​​of the plurality of evaluation indexes; Equipped with the evaluation index space has the plurality of evaluation indexes as its coordinate axes, each of the plurality of models is embedded in the evaluation index space at a position corresponding to the value of the evaluation index of the model; A model analysis device in which a plurality of line segments are drawn in the evaluation index space, so that the plurality of evaluation indexes are at the same level.

2. the analysis means aggregates the values ​​of the plurality of evaluation indexes based on the importance to calculate an aggregate evaluation value for each of the models, and outputs information ranking the plurality of models according to the aggregate evaluation value; The model analysis device according to claim 1 , wherein the analysis means calculates a weighted sum of the importance and the values ​​of the plurality of evaluation indexes as the aggregate evaluation value.

3. a task acquisition means for receiving a task specification; 3. The model analysis device according to claim 1, wherein the index value acquisition means calculates values ​​of the evaluation index for a plurality of models corresponding to a specified task.

4. The model analysis device according to claim 1 , wherein the importance obtaining means obtains a ratio of importance for the plurality of evaluation indexes.

5. The model analysis device according to claim 1 , wherein the importance obtaining means obtains an importance value for each of the plurality of evaluation indexes.

6. The evaluation indexes are two, The model analysis device according to claim 1 , wherein the evaluation index space has one of the evaluation indexes on the X axis and the other on the Y axis.

7. 1. A computer-implemented method for model analysis, comprising: Obtaining multiple evaluation metric specifications for the model; Obtaining the importance of the plurality of evaluation indexes; Obtaining values ​​of the evaluation indexes for a plurality of models; Analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance; displaying an evaluation index space for the plurality of models based on the values ​​of the plurality of evaluation indexes; the evaluation index space has the plurality of evaluation indexes as its coordinate axes, each of the plurality of models is embedded in the evaluation index space at a position corresponding to the value of the evaluation index of the model; A model analysis method in which a plurality of line segments are drawn in the evaluation index space, where the plurality of evaluation indexes are at the same level.

8. Obtaining multiple evaluation metric specifications for the model; Obtaining the importance of the plurality of evaluation indexes; Obtaining values ​​of the evaluation indexes for a plurality of models; Analyzing the ranking of the plurality of models based on the acquired evaluation index values ​​and importance; causing a computer to execute a process of displaying an evaluation index space for the plurality of models based on values ​​of the plurality of evaluation indexes; the evaluation index space has the plurality of evaluation indexes as its coordinate axes, each of the plurality of models is embedded in the evaluation index space at a position corresponding to the value of the evaluation index of the model; A program in which a plurality of line segments are drawn in the evaluation index space, where the plurality of evaluation indexes are at the same level.

Images