Model analysis device, model analysis method, and program

Abstract

Provided is a model analysis device wherein an acquisition means acquires designation of a plurality of evaluation indexes relating to a model and designation of degrees of importance for the plurality of evaluation indexes. An evaluation value acquisition means acquires the values of the plurality of evaluation indexes for a plurality of models. A display means displays the plurality of models on the basis of the evaluation index values in a space in which the plurality of evaluation indexes serve as the axes. A rendering means renders, in the space, a region in which model evaluation values determined on the basis of the evaluation indexes and degrees of importance are the same. An output means outputs information representing the space.

Description

[Technical Field]

[0001] The present disclosure relates to the analysis of machine learning models. [Background technology]

[0002] In recent years, predictive models obtained by machine learning have been used in various fields. Patent Document 1 relates to a system for learning and / or optimizing processes related to semiconductor manufacturing, and describes a method for ranking candidate process models based on quality values ​​and diversity values. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-504872 Summary of the Invention [Problem to be solved by the invention]

[0004] When selecting a predictive model to use in machine learning, it is desirable to evaluate the predictive model by comparing and considering multiple evaluation indicators depending on the task to which the predictive model will be applied, and select an appropriate predictive model.

[0005] One object of the present disclosure is to evaluate models and select an appropriate model by taking into account multiple evaluation indicators. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a model analysis apparatus includes: an acquisition means for acquiring designations of a plurality of evaluation indices related to a model and designations of importance for the plurality of evaluation indices; evaluation value acquisition means for acquiring values ​​of the plurality of evaluation indexes for a plurality of models; a model evaluation value calculation means for calculating a weighted sum of the values ​​of the plurality of evaluation indexes using a designated importance ratio as a weight as a model evaluation value; The aforementioned Model evaluation valuea display means for displaying the plurality of models in a space having the plurality of evaluation indexes as axes based on the above; a line segment drawing means for drawing a plurality of contour lines having the same model evaluation value in the space; The area defined by the adjacent contour lines is defined as an area indicating the location of models that satisfy the specified importance and have the same level of performance. Draw in the space region A drawing means; an output means for outputting information indicating the space; Equipped with.

[0007] In another aspect of the present disclosure, a computer-implemented model analysis method includes: acquiring a designation of a plurality of evaluation indices for the model and a designation of importance for the plurality of evaluation indices; Obtaining values ​​of the plurality of evaluation indexes for a plurality of models; Calculating a weighted sum of the values ​​of the plurality of evaluation indexes using the ratio of the designated importance as a weight as a model evaluation value; The aforementioned Model evaluation value displaying the plurality of models in a space with the plurality of evaluation indexes as axes based on the A plurality of contour lines having the same model evaluation value are drawn in the space; The area defined by the adjacent contour lines is defined as an area indicating the location of models that satisfy the specified importance and have the same level of performance. Drawing on the space, Information indicating the space is output.

[0008] In yet another aspect of the invention, a program includes: acquiring a designation of a plurality of evaluation indices for the model and a designation of importance for the plurality of evaluation indices; Obtaining values ​​of the evaluation indexes for a plurality of models; Calculating a weighted sum of the values ​​of the plurality of evaluation indexes using the ratio of the designated importance as a weight as a model evaluation value; The aforementioned Model evaluation value displaying the plurality of models in a space with the plurality of evaluation indexes as axes based on the A plurality of contour lines having the same model evaluation value are drawn in the space; The area defined by the adjacent contour lines is defined as an area indicating the location of models that satisfy the specified importance and have the same level of performance. Drawing on the space, The computer is caused to execute a process of outputting information indicating the space. [Effects of the Invention]

[0009] According to the present disclosure, it becomes possible to evaluate models and select appropriate models by taking multiple evaluation indicators into consideration. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the model generating device. [Figure 3] FIG. 1 is a block diagram showing a functional configuration of a model generation device according to a first embodiment. [Figure 4] 10 shows an example of displaying evaluation information according to the first embodiment. [Figure 5] 10 shows another example of displaying evaluation information according to the first embodiment. [Figure 6] 10 is a flowchart of a model analysis process according to the first embodiment. [Figure 7] FIG. 1 is a block diagram showing a schematic configuration of a model generation system using a server and a terminal device. [Figure 8] 10 shows an example of displaying evaluation information according to the second embodiment. [Figure 9] 10 is a flowchart of a process performed by a model analysis device according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing the functional configuration of a model analysis device according to a third embodiment. [Figure 11] 10 is a flowchart of a process performed by a model analysis device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Overall configuration] FIG. 1 is a block diagram showing the overall configuration of a model generation system according to the first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is an application of the model analysis device of the present disclosure, and is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by the user to give instructions and input necessary when modifying a model or viewing the evaluation information.

[0012] First, the operation of the model generation system 1 will be briefly described. The model generation device 100 generates a machine learning model (hereinafter simply referred to as a "model") using training data prepared in advance. The model generation device 100 also analyzes and evaluates the generated model. Specifically, the model generation device 100 performs model prediction using evaluation data and evaluates the model based on the prediction result. The model generation device 100 evaluates multiple models using multiple evaluation indices. The evaluation indices include various indices such as the prediction accuracy, compatibility, and fairness of the model. Note that prediction accuracy indicates the probability that the prediction result of a model is correct. Compatibility indicates the degree of agreement between the prediction results of one model and another model. Furthermore, fairness indicates the degree of difference or variation between the prediction results of models for data with different attributes. The model generation device 100 then presents the results of evaluating the multiple models in relation to the multiple evaluation indices to the user as evaluation information. The user can check the evaluation information and select an appropriate model from the multiple models, or operate the input device 3 to input correction information for correcting the model.

[0013] A "machine learning model" is information that represents the relationship between explanatory variables and a response variable. A machine learning model is, for example, a component that estimates the results of an estimation target by calculating a response variable based on explanatory variables. A machine learning model is generated by executing a learning algorithm using training data, for which the response variable value has already been obtained, and arbitrary parameters as input. A machine learning model may be represented, for example, by a function c that maps an input x to a correct answer y. A machine learning model may estimate a numerical value of an estimation target, or may estimate a label of an estimation target. A machine learning model may output a variable that describes the probability distribution of a response variable. A machine learning model may also be referred to as a "learning model," an "analysis model," an "AI (artificial intelligence) model," or a "prediction formula." An explanatory variable is a variable used as input in a machine learning model. An explanatory variable may also be referred to as a "feature" or "characteristic."

[0014] Furthermore, the learning algorithm for generating the machine learning model is not particularly limited and may be an existing learning algorithm. For example, the learning algorithm may be a random forest, a support vector machine, a naive Bayes, a piecewise linear model using FAB inference (Factorized Asymptotic Bayesian Inference), or a neural network. Note that a piecewise linear model technique using FAB inference is disclosed, for example, in U.S. Patent Publication US2014 / 0222741A1.

[0015] [Hardware configuration] 2 is a block diagram showing the hardware configuration of model generation device 100. As shown in the figure, model generation device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0016] The I / F 111 inputs and outputs data to and from external devices. Specifically, training data and evaluation data used in generating a model, as well as instructions and inputs entered by a user using the input device 3, are input to the model generation device 100 via the I / F 111. Furthermore, evaluation information for a model generated by the model generation device 100 is output to the display device 2 via the I / F 111.

[0017] The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generation device 100 by executing a prepared program. The processor 112 may be a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, or an FPGA (Field-Programmable Gate Array). The processor 112 executes the model analysis process described below.

[0018] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0019] The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 records various programs to be executed by the processor 112. When the model generation device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0020] DB 115 stores information about the model generated by model generation device 100 (hereinafter referred to as the "existing model") and the model corrected by retraining (hereinafter referred to as the "corrected model"). DB 115 also stores training data, evaluation data, correction information input by the user, etc., input via I / F 111 as necessary.

[0021] [Function Configuration] 3 is a block diagram showing the functional configuration of the model generation device 100 according to the first embodiment. Functionally, the model generation device 100 includes a training data DB 121, a model training unit 122, a model DB 123, an evaluation data DB 124, and an analysis unit 125.

[0022] The training data DB 121 stores training data used to generate a model. The training data D1 is input to the model training unit 122. The training data D1 is composed of multiple combinations of input data and correct answers (correct values ​​or correct labels) for the input data.

[0023] The model training unit 122 trains a model using the training data D1 and generates a model. The model training unit 122 outputs model data M corresponding to the generated model to the model DB 123 and the analysis unit 125. The model data M includes multiple pieces of parameter information constituting the model. The parameter information includes, for example, information on explanatory variables used as input to the model, information on weights for each explanatory variable, and information on weights for each sample constituting the input data.

[0024] The model DB 123 stores multiple models for each task. A "task" refers to a type of prediction or inference made by a machine learning model based on a problem or question and available data. Tasks include classification tasks, regression tasks, clustering tasks, object recognition tasks, and various other tasks. For example, a classification task involves assigning data to a category. A regression task involves predicting the value of a target variable based on the values ​​of other explanatory variables. A clustering task involves grouping data according to similarity. An object recognition task involves detecting objects in an image and outputting their location and class. The model DB 123 stores multiple models for each task. Note that the multiple models corresponding to a given task may use different algorithms, may be different models obtained using the same algorithm on different data, or may be different models obtained using the same algorithm with different settings (e.g., hyperparameters).

[0025] The model training unit 122 retrains the existing model to generate a modified model. In this case, the model training unit 122 modifies the parameters that configure the model based on modification information D3 input by the user using the input device 3, and retrains the model using training data for retraining as necessary. The model training unit 122 stores model data M of the modified model obtained by the retraining in the model DB 123 and outputs it to the analysis unit 125.

[0026] Furthermore, the method by which the model training unit 122 retrains the model is not particularly limited. For example, the model training unit 122 may use a method of retraining a completely new, modified model while ignoring the existing model, or a method of updating the existing model using new training data. Other methods by which the model training unit 122 retrains the existing model include a method of updating only the coefficients by retraining without changing the types of explanatory variables used in the model, and a method of retraining that also includes the selection of explanatory variables.

[0027] The evaluation data DB 124 stores evaluation data used to evaluate the generated model. The evaluation data may be, for example, a portion of collected data that was not used as training data, newly collected data, or validation data prepared for verification. Training data may also be used as evaluation data. The evaluation data is composed of multiple combinations of input data and correct answers (correct values ​​or correct labels) for the input data.

[0028] The analysis unit 125 analyzes the performance of multiple models using the evaluation data. Specifically, the analysis unit 125 acquires multiple models corresponding to a task selected by the user from the model DB 123 and calculates evaluation indexes for each model using the evaluation data. In the following description, the analysis unit 125 calculates an accuracy index and a compatibility index for each model as multiple evaluation indexes specified by the user. The analysis unit 125 then evaluates each model in relation to the accuracy index and the compatibility index, and outputs the result to the display device 2 as evaluation information D2.

[0029] The display device 2 displays the evaluation information D2 output by the analysis unit 125 on the display device 2. By looking at the displayed evaluation information, the user can evaluate multiple models from the perspective of the accuracy index and the compatibility index.

[0030] Furthermore, the user inputs correction information D3 for correcting the current model to the input device 3 as necessary. The correction information D3 is information related to the correction, such as information on explanatory variables used as input to the model, information on weights for each explanatory variable, information on weights set for each attribute of the category of explanatory variables, and information on weights for each sample constituting the input data. The model training unit 122 corrects the model by retraining the model using the input correction information D3.

[0031] [Example of evaluation information display] In the first embodiment, the analysis unit 125 ranks and displays a plurality of models corresponding to one task based on the importance designated by the user for a plurality of evaluation indexes.

[0032] Fig. 4 shows an example of a display of evaluation information according to the first embodiment. The display example 40 in Fig. 4 includes a model list 41 and a slide bar 42. The model list 41 displays a list of multiple models, and includes candidate models, accuracy indicators, and compatibility indicators. "Candidate models" indicates multiple models corresponding to one task. In the example in Fig. 4, four candidate models h1 to h4 corresponding to one task are displayed.

[0033] The "accuracy index" is an index that indicates the accuracy of each candidate model. Essentially, the accuracy index indicates the probability that the model output is correct. For example, in the case of a regression task, the accuracy index is R 2 The coefficient of determination (COF), RMSE (Root Means Square Error), and MAE (Mean Absolute Error) can be used. However, for indicators where a smaller value indicates better accuracy, it can be multiplied by "-1" so that a larger value indicates better accuracy. Specifically, for RMSE and MSE, the value multiplied by "-1" can be used as the indicator. For classification tasks, accuracy indicators include accuracy, precision, recall, F-score (F1-score), and AUC (Area under the ROC curve). The "compatibility index" indicates the compatibility between multiple models, specifically, the degree to which another model hx can correctly interpret data that was correctly interpreted by a certain model h0.

[0034] 4, the analysis unit 125 uses a "compatibility index" and an "accuracy index" as evaluation indices. However, the evaluation indices used by the analysis unit 125 are not limited to these, and any evaluation index can be used. For example, the analysis unit 125 may analyze the performance of the model using a "fairness index" as an evaluation index.

[0035] The slide bar 42 is a bar for specifying the ratio of the importance (weight) of the accuracy index and the compatibility index, and is operated by the user. Specifically, when the cursor 42a is at the left end of the slide bar 42, the importance of the accuracy index is 100% and the importance of the compatibility index is 0%. On the other hand, when the cursor 42a is at the right end of the slide bar 42, the importance of the accuracy index is 0% and the importance of the compatibility index is 100%. Furthermore, when the cursor 42a is in the center of the slide bar 42, the importance of the accuracy index is 50% and the importance of the compatibility index is 50%. By specifying the position of the cursor 42a on the slide bar 42, the user can specify how much importance to assign to either the accuracy index or the compatibility index.

[0036] 4, it is assumed that the position of cursor 42a corresponds to an accuracy index importance of 80% and a compatibility index importance of 20%. Analysis unit 125 first calculates the accuracy index value and the compatibility index value for each of candidate models h1 to h4 using evaluation data, etc. Note that the larger the accuracy index value, the higher the accuracy, and the larger the compatibility index value, the higher the compatibility.

[0037] Next, the analysis unit 125 ranks the candidate models using a value obtained by aggregating the evaluation indicators using the ratio of importance specified by the user (hereinafter also referred to as an "aggregated evaluation value"). This aggregated evaluation value is an example of a model evaluation value, and is a value that indicates an overall evaluation of multiple evaluation indicators based on the importance specified by the user. Therefore, by ranking multiple candidate models using the aggregated evaluation value, the user can select an appropriate candidate model under the conditions of the importance specified by the user.

[0038] In a preferred example, the analysis unit 125 aggregates the evaluation indexes using a linear function and uses a weighted sum as the aggregated evaluation value. In this case, the analysis unit 125 calculates the weighted sum of the accuracy index value and the compatibility index value for each candidate model using the ratio of the importance of the accuracy index to the compatibility index specified by the user (80%:20%=4:1) as a weight. For example, the analysis unit 125 calculates the weighted sum S for each candidate model as follows: S=4×(accuracy index)+1×(compatibility index) (1)

[0039] The analysis unit 125 calculates the weighted sum for all candidate models h1 to h4 using the above formula (1), ranks the candidate models in descending order of the weighted sum S, and displays them as a list. As a result, in the example of Fig. 4, the user can see that model h3 is the most appropriate in terms of the ratio of the importance of the accuracy index to the compatibility index (80%:20%) that the user has specified.

[0040] In this way, the analysis unit 125 displays a list of multiple candidate models according to an aggregate evaluation value that aggregates the accuracy index and compatibility index using the importance level specified by the user, allowing the user to easily know which candidate model to use based on the importance level specified by the user.

[0041] FIG. 5 shows another example of displaying evaluation information according to the first embodiment. This example also shows accuracy indices and compatibility indices for multiple models corresponding to a single task. However, while in the example of FIG. 4 the user specifies the importance using a slide bar 42, in the example of FIG. 5 the user specifies the importance using a bar graph. The display example 50 of FIG. 5 includes a model list 51 and a bar graph 52. The model list 41, like the first example, displays a list of multiple models. However, in the example of FIG. 5, two accuracy indices and two compatibility indices selected by the user are used. Regarding the compatibility indices, for example, when there are two models, a new and an old, compatibility index 1 can be the percentage of data that the old model correctly answered that also correctly answered the new model. Furthermore, when there are two models, a new and an old, compatibility index 2 can be the percentage of data that the new model incorrectly answered that also correctly answered the old model.

[0042] The bar graph 52 includes a bar 52a for each evaluation index. The user can change the importance of each evaluation index by changing the length of the bar 52a corresponding to that evaluation index using a cursor C or the like. In the example of FIG. 5, the user changes the importance of accuracy index 1 using cursor C. If there is no room to extend the bar 52a due to the display area of ​​the display example 50, the user can press a renormalization button 52b to fit the length of each bar 52a in the display example 50 within the display area without changing the importance ratio between the multiple evaluation indexes. Furthermore, as shown in FIG. 5, the ratio of the length of each bar 52a to the total (total of four bars) is displayed below each bar 52a, allowing the user to fine-tune the importance ratio for each evaluation index.

[0043] The method by which the analysis unit 125 ranks the multiple candidate models is the same as in the first example. That is, the analysis unit 125 uses the importance ratio specified by the user to calculate the weighted sum S of each evaluation index (in this example, accuracy index 1, accuracy index 2, compatibility index 1, and compatibility index 2) as an aggregate evaluation value, and displays the multiple candidate models in that order in the model list 51. In this example, too, the user can easily know which candidate model is optimal for the importance ratio specified by the user.

[0044] [Model analysis processing] 6 is a flowchart showing the model analysis process according to the first embodiment. The model analysis process is a process for displaying multiple models corresponding to a task specified by a user, in order of the importance of the evaluation index specified by the user. This process is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating mainly as the model training unit 122 and the analysis unit 125 shown in FIG. 3.

[0045] First, the analysis unit 125 acquires the user's designation of a task and evaluation indexes (step S10). Fig. 4 shows an example in which the user designates a task and designates an accuracy index and a compatibility index as evaluation indexes. Fig. 5 shows an example in which the user designates a task and designates accuracy index 1, accuracy index 2, compatibility index 1, and compatibility index 2 as evaluation indexes.

[0046] Next, the analysis unit 125 acquires a plurality of candidate models corresponding to the task designated by the user (step S11). In the examples of Fig. 4 and Fig. 5, the analysis unit 125 acquires candidate models h1 to h4.

[0047] Next, the analysis unit 125 determines a display order based on one of the multiple evaluation indexes as a default display (step S12), and displays a list of multiple candidate models in the determined display order (step S13). Note that the one evaluation index in this case may be, for example, an evaluation index specified by the user from the multiple evaluation indexes, or the evaluation index specified first by the user may be used. As a result, in the default display, the multiple candidate models are listed in descending order of the one evaluation index.

[0048] Next, the analysis unit 125 determines whether the user has changed the importance (step S14). Specifically, in the example of FIG. 4, the analysis unit 125 determines whether the user has changed the position of the cursor 42a on the slide bar 42. Furthermore, in the example of FIG. 5, the analysis unit 125 determines whether the user has changed the length of the bar 52a of any evaluation index. If the user has changed the importance (step S14: Yes), the analysis unit 125 calculates the weighted sum S described above based on the changed importance, and determines the display order of multiple candidate models in the order of the weighted sum S (step S15). Then, the analysis unit 125 redisplays the model list in the determined display order (step S13). As a result, the model list 41 shown in FIG. 4 or the model list 51 shown in FIG. 5 is displayed.

[0049] On the other hand, if the user has not changed the importance (step S14: No), the analysis unit 125 determines whether or not the user has input an end instruction (step S16). If the end instruction has not been input (step S16: No), the process returns to step S14. On the other hand, if the end instruction has been input (step S16: Yes), the process ends.

[0050] In the above flowchart, the display order of multiple candidate models is determined based on any one evaluation index as a default display in step S12. Alternatively, the user may be prompted to input importance from the beginning, and the display order of multiple candidate models may be determined and displayed based on the input importance.

[0051] As described above, in the model analysis process of the first embodiment, an aggregate evaluation value for multiple evaluation indexes is calculated based on multiple evaluation indexes specified by the user and their importance, and multiple candidate models corresponding to the task specified by the user are displayed in order based on the aggregate evaluation value. Thus, the user can easily know the optimal candidate model for the importance level specified by the user.

[0052] [Variations] (Variation 1) In the above model analysis process, the analysis unit 125 aggregates multiple evaluation indexes using a linear function that indicates a weighted sum, but instead, it may also be possible to aggregate multiple evaluation indexes using other linear or nonlinear functions and calculate an aggregate evaluation value.

[0053] (Variation 2) In the above model analysis process, the importance ratio specified by the user may be associated with the task and learned. This makes it possible to generate a machine learning model that has learned the relationship between the task and the importance ratio that the user considers appropriate for that task. By using this machine learning model, the user can input a task and obtain an appropriate importance ratio for that task as output.

[0054] (Variation 3) In the first embodiment, the analysis unit 125 displays a list of multiple models corresponding to a task specified by the user in an ordered manner. Alternatively, the analysis unit 125 may display a list of multiple models specified by the user in an ordered manner, regardless of the identity of the task.

[0055] (Variation 4) In the first embodiment, the analysis unit 125 acquires multiple models corresponding to the task selected by the user from the model DB 123 and calculates the evaluation index of each model using the evaluation data. Alternatively, a function for calculating the evaluation index of each model may be provided externally. For example, the analysis unit 125 may acquire the evaluation index of each model from a DB that stores the evaluation index of each model.

[0056] (Variation 5) In the above embodiment, the model generation device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured by a server and a terminal device. FIG. 7 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 7, the server 100x has the configuration of the model generation device 100 shown in FIG. 3. Furthermore, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG. 3.

[0057] Second Embodiment Next, a second embodiment will be described. The overall configuration of the model generation system 1 according to the second embodiment, and the hardware configuration and functional configuration of the model generation device 100 are basically the same as those of the first embodiment, and therefore description thereof will be omitted.

[0058] [Example of evaluation information display] In the second embodiment, the analysis unit 125 displays the relationship between multiple models corresponding to one task based on multiple evaluation indexes. Specifically, the analysis unit 125 embeds each model in a space based on the values ​​of multiple evaluation indexes specified by the user, and draws in the space an area of ​​models that can be considered to be at the same level in relation to the multiple evaluation indexes and the importance specified by the user.

[0059] FIG. 8 shows an example of displaying evaluation information according to the second embodiment. Display example 60 in FIG. 8 is obtained by embedding each model in a space (hereinafter also referred to as "evaluation index space") in which the compatibility index and accuracy index specified by the user are the X-axis and Y-axis, respectively. As in the first embodiment, the analysis unit 125 acquires multiple models corresponding to one task specified by the user, and calculates the values ​​of the compatibility index and accuracy index specified by the user for those models. Then, the analysis unit 125 embeds each model in the evaluation index space based on the values ​​of the compatibility index and accuracy index of each model. This results in display example 60.

[0060] Next, the analysis unit 125 draws line segments L1 to L5 in the evaluation index space where the two evaluation indexes are at the same level based on the aforementioned aggregated evaluation value, and creates a display example 61. Now, let us assume that the weighted sum S is used as the aggregated evaluation value. If the importance (weight) specified by the user for the accuracy index is w1 and the importance (weight) specified for the compatibility index is w2, then the weighted sum S is expressed by the following formula: S=w1×(accuracy)+w2×(compatibility) (2)

[0061] The analysis unit 125 sets the value of the weighted sum S to a predetermined value S1 and draws a line segment L1 in the evaluation index space. Similarly, the analysis unit 125 changes the value of the weighted sum S to S2, S3, ... by an arbitrary amount and draws line segments L2 to L5 in the evaluation index space. Furthermore, the analysis unit 125 displays areas A1 to A5 defined by line segments L1 to L5 in different colors or the like to distinguish them from one another.

[0062] Here, line segments L1 to L5 indicate the position of the model in the evaluation index space when the user-specified importance (w1 / w2) and the aggregate evaluation value S of multiple evaluation indexes are fixed, and the values ​​of the accuracy index and compatibility index are changed. Therefore, for example, if two models M1 and M2 are on line segment L1, they satisfy the ratio of the importance of the accuracy index and the compatibility index specified by the user, and are models with the same aggregate evaluation value S, which indicates the overall evaluation of multiple evaluation indexes, and can be considered to be models of the same level in terms of the importance specified by the user. The same is true for the other line segments L2 to L5. In other words, line segments L1 to L5 can be considered as contour lines indicating the position of models in the evaluation index space that satisfy the importance specified by the user and have the same overall evaluation.

[0063] Therefore, area A2 defined by line segments L1 and L2 in the evaluation index space can be considered an area indicating the location of models that largely satisfy the importance specified by the user and have roughly the same level of performance. The same applies to the other areas A1, A3 to A5. Therefore, the user can know that multiple models that belong to the same area in display example 61 are roughly at the same level in terms of the importance specified by the user. For example, the user can recognize that models h1 and h2 that belong to area A2 are roughly at the same level, and models h3 and h4 that belong to area A3 are roughly at the same level.

[0064] In this way, in the second embodiment, by displaying multiple models in the evaluation index space, the user can easily visually grasp models that are at the same level in terms of the importance of the evaluation index specified by the user. In particular, by specifying two evaluation indexes that are considered to have a large trade-off and performing the above display, the user can find a model that can exhibit appropriate performance by skillfully adjusting the two conflicting evaluation indexes.

[0065] [Model analysis processing] 9 is a flowchart showing the model analysis process according to the second embodiment. The model analysis process is a process for displaying multiple models corresponding to a task selected by a user in an evaluation index space according to the importance of the evaluation index specified by the user. This process is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating mainly as the model training unit 122 and the analysis unit 125 shown in FIG. 3.

[0066] First, the analysis unit 125 acquires a task by a user, multiple evaluation indices, and the importance of those evaluation indices (step S10). Fig. 8 shows an example in which a user specifies a task and specifies an accuracy index and a compatibility index as evaluation indices.

[0067] Next, the analysis unit 125 acquires multiple candidate models corresponding to the task specified by the user (step S11). In the example of FIG. 8, the analysis unit 125 acquires models h1 to h5. Next, the analysis unit 125 calculates the value of the evaluation index for each model and embeds it into an evaluation index space (step S22). In the example of FIG. 8, the analysis unit 125 calculates the accuracy index and compatibility index for models h1 to h5, and embeds each of models h1 to h5 in the evaluation index space as shown in display example 60.

[0068] Next, the analysis unit 125 draws contour lines L1 to L5 using the importance levels specified by the user and the values ​​of each evaluation index. In the example of Fig. 8, the analysis unit 125 draws contour lines L1 to L5 by fixing the ratio (w1 / w2) of the importance levels of each evaluation index specified by the user in the above formula (2), changing the value of the weighted sum S within a predetermined range, and varying the accuracy index and compatibility index. Then, the analysis unit 125 colors the areas A1 to A5 defined by the contour lines L1 to L5 in different colors, and draws areas of models that can be considered to be at the same level (step S23). Then, the processing ends.

[0069] As described above, in the model analysis process of the second embodiment, multiple models corresponding to a task specified by the user are displayed in the evaluation index space based on multiple evaluation indexes and their importance specified by the user. Also, based on the evaluation indexes and their importance specified by the user, a range of models that can be considered to be at the same level are displayed in the evaluation index space. Therefore, the user can easily know models that can be considered to be at the same level when the user specifies the importance, for example, models that can be used as alternatives.

[0070] [Variations] (Variation 1) In the above example, the analysis unit 125 aggregates two evaluation indexes using a linear function that indicates a weighted sum and plots contour lines on the evaluation index space. Alternatively, the analysis unit 125 may aggregate two evaluation indexes using other linear or nonlinear functions and plot contour lines on the evaluation index space.

[0071] (Variation 2) In the second embodiment, the analysis unit 125 also displays a list of multiple models corresponding to the task specified by the user in an ordered manner. Alternatively, the analysis unit 125 may display multiple arbitrary models specified by the user in the evaluation index space, regardless of the identity of the task.

[0072] (Variation 3) In the second embodiment, similarly to the fourth modification of the first embodiment, the analysis unit 125 may obtain the evaluation index of each model from an external source.

[0073] (Variation 4) In the second embodiment, similarly to the fifth modification of the first embodiment, the model generating device may be configured by a server and a terminal device as shown in FIG.

[0074] <Third embodiment> 10 is a block diagram showing the functional configuration of a model analysis device 70 according to the third embodiment. The model analysis device 70 includes an acquisition unit 71, an evaluation value acquisition unit 72, a display unit 73, a drawing unit 74, and an output unit 75.

[0075] 11 is a flowchart of processing by the model analysis device 70 of the third embodiment. The acquisition means 71 acquires the designation of multiple evaluation indexes for a model and the designation of importance for the multiple evaluation indexes (step S71). The evaluation value acquisition means 72 acquires the values ​​of multiple evaluation indexes for multiple models (step S72). The display means 73 displays multiple models in a space with the multiple evaluation indexes as axes based on the values ​​of the evaluation indexes (step S73). The drawing means 74 draws in the space an area where the model evaluation value determined based on the evaluation indexes and importance is the same (step S74). The output means 75 outputs information indicating the space (step S75).

[0076] According to the model analysis device 70 of the third embodiment, it becomes possible to evaluate models and select appropriate models in consideration of multiple evaluation indexes.

[0077] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0078] (Appendix 1) an acquisition means for acquiring designations of a plurality of evaluation indices related to a model and designations of importance for the plurality of evaluation indices; evaluation value acquisition means for acquiring values ​​of the plurality of evaluation indexes for a plurality of models; a display means for displaying the plurality of models in a space having the plurality of evaluation indexes as axes based on the values ​​of the evaluation indexes; a drawing means for drawing, in the space, an area in which the model evaluation value determined based on the evaluation index and the importance is the same; an output means for outputting information indicating the space; A model analysis device comprising:

[0079] (Appendix 2) 2. The model analysis device according to claim 1, wherein the drawing means draws a plurality of contour lines for which the model evaluation value obtained by aggregating the values ​​of the plurality of evaluation indexes is the same using the importance.

[0080] (Appendix 3) 3. The model analysis device according to claim 2, wherein the drawing means draws the contour lines using a function that calculates the model evaluation value using the values ​​of the plurality of evaluation indexes and the importance.

[0081] (Appendix 4) 3. The model analysis apparatus according to claim 2, wherein the drawing means draws the regions defined by the contour lines in different colors.

[0082] (Appendix 5) a task acquisition means for receiving a task specification; 2. The model analysis device according to claim 1, wherein the calculation means calculates values ​​of the evaluation indexes for a plurality of models corresponding to a specified task.

[0083] (Appendix 6) acquiring a designation of a plurality of evaluation indices for the model and a designation of importance for the plurality of evaluation indices; Obtaining values ​​of the evaluation indexes for a plurality of models; displaying the plurality of models in a space having the plurality of evaluation indexes as axes based on the values ​​of the evaluation indexes; drawing, in the space, a region in which the model evaluation value determined based on the evaluation index and the importance is the same; A model analysis method that outputs information indicating the space.

[0084] (Appendix 7) acquiring a designation of a plurality of evaluation indices for the model and a designation of importance for the plurality of evaluation indices; Obtaining values ​​of the evaluation indexes for a plurality of models; displaying the plurality of models in a space with the plurality of evaluation indexes as axes based on the values ​​of the evaluation indexes; drawing, in the space, a region in which the model evaluation value determined based on the evaluation index and the importance is the same; A recording medium on which a program for causing a computer to execute a process for outputting information indicating the space is recorded.

[0085] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0086] 1. 1x Model Generation System 2, 2x display device 3. 3x input devices 7 Terminal Equipment 100 Model generation device 112 processors 121 Training Data DB 122 Model Training Department 123 Model DB 124 Evaluation Data DB 125 Analysis Department

Claims

1. an acquisition means for acquiring designations of a plurality of evaluation indices related to a model and designations of importance for the plurality of evaluation indices; evaluation value acquisition means for acquiring values ​​of the plurality of evaluation indexes for a plurality of models; a model evaluation value calculation means for calculating a weighted sum of the values ​​of the plurality of evaluation indexes using a designated importance ratio as a weight as a model evaluation value; a display means for displaying the plurality of models in a space having the plurality of evaluation indexes as axes based on the model evaluation values; a line segment drawing means for drawing a plurality of contour lines having the same model evaluation value in the space; an area drawing means for drawing an area defined by adjacent contour lines in the space as an area indicating the location of models that satisfy a designated importance and have the same level of performance; an output means for outputting information indicating the space; A model analysis device comprising:

2. 2. The model analysis device according to claim 1, wherein the region drawing means draws the contour lines using a function that calculates the model evaluation value using the values ​​of the plurality of evaluation indexes and the importance.

3. 2. The model analysis device according to claim 1, wherein the region drawing means draws the regions defined by the contour lines in different colors.

4. a task acquisition means for receiving a task specification; The model analysis device according to claim 1 , wherein the evaluation value acquisition means calculates values ​​of the plurality of evaluation indexes for a plurality of models corresponding to a specified task.

5. 1. A computer-implemented method for model analysis, comprising: acquiring a designation of a plurality of evaluation indices for the model and a designation of importance for the plurality of evaluation indices; Obtaining values ​​of the evaluation indexes for a plurality of models; Calculating a weighted sum of the values ​​of the plurality of evaluation indexes using the ratio of the designated importance as a weight as a model evaluation value; displaying the plurality of models in a space with the plurality of evaluation indexes as axes based on the model evaluation values; A plurality of contour lines having the same model evaluation value are drawn in the space; A region defined by adjacent contour lines is drawn in the space as a region indicating the location of models that satisfy the designated importance and have the same level of performance; A model analysis method that outputs information indicating the space.

6. acquiring a designation of a plurality of evaluation indices for the model and a designation of importance for the plurality of evaluation indices; Obtaining values ​​of the evaluation indexes for a plurality of models; Calculating a weighted sum of the values ​​of the plurality of evaluation indexes using the ratio of the designated importance as a weight as a model evaluation value; displaying the plurality of models in a space with the plurality of evaluation indexes as axes based on the model evaluation values; A plurality of contour lines having the same model evaluation value are drawn in the space; A region defined by adjacent contour lines is drawn in the space as a region indicating the location of models that satisfy the designated importance and have the same level of performance; A program that causes a computer to execute a process of outputting information indicating the space.

Images