Model analysis device, model analysis method, and program

Abstract

Provided is a model analysis device wherein a model acquisition means acquires a model. A dataset acquisition means acquires a dataset. A performance calculation means calculates the performance of the model for each attribute corresponding to each category of the dataset. An output means outputs performance information indicating the calculated performance of the model for each attribute. A parameter changing means receives a change in a model parameter corresponding to an attribute. A training means retrains the model using the changed parameter. If the model is retrained, the performance calculation means calculates the performance of the model after retraining, and the output means outputs the calculated performance of the model.

Description

Technical Field

[0001] This disclosure relates to the analysis of machine learning models.

Background Art

[0002] In recent years, prediction models obtained by machine learning have been used in various fields. Patent Document 1 describes a method for visualizing the relationship between the number of learning data used for model learning and the discrimination accuracy of the model obtained by learning for each category of learning data used for learning the discrimination model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When making a prediction using a prediction model, there may be a difference in the performance of the model depending on the category and attributes of the input data set. Therefore, the user needs to check the difference in model performance and modify the model to reduce the difference in performance.

[0005] One object of this disclosure is to evaluate the performance of a model for each attribute of the category of the data set used for prediction and modify the model so as to reduce the difference in model performance.

Means for Solving the Problems

[0006] From one aspect of this disclosure, a model analysis device includes model acquisition means for acquiring a model, data set acquisition means for acquiring a data set, performance calculation means for calculating the performance of the model for each attribute corresponding to each category of the data set Output means for outputting performance information indicating the performance of the model for each calculated attribute; Parameter change means for receiving a change in the parameters of the model corresponding to the attribute; Training means for retraining the model using the changed parameters; comprising: When the model is retrained, the performance calculation means calculates the performance of the retrained model.

[0007] In another aspect of the present disclosure, Program executed by a computer A model analysis method obtains a model, obtains a dataset, calculates the performance of the model for each attribute corresponding to each category of the dataset, outputs performance information indicating the performance of the model for each calculated attribute, receives a change in the parameters of the model corresponding to the attribute, retrains the model using the changed parameters, and calculates the performance of the retrained model.

[0008] In still another aspect of the present invention, Program is obtains a model, obtains a dataset, calculates the performance of the model for each attribute corresponding to each category of the dataset, outputs performance information indicating the performance of the model for each calculated attribute, receives a change in the parameters of the model corresponding to the attribute, causes a computer to execute a process of retraining the model using the changed parameters and calculating the performance of the retrained model.

Advantages of the Invention

[0009] According to the present disclosure, the performance of the model can be evaluated for each attribute of the category of the dataset used for prediction, and the model can be corrected so as to reduce the difference in the performance of the model.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0011] Hereinafter, with reference to the drawings, preferred embodiments of the present disclosure will be described. <First Embodiment> [Overall Configuration] FIG. 1 is a block diagram showing the overall configuration of the model generation system according to the first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is an application of the model analysis device of the present disclosure and is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used for the user to give necessary instructions and inputs when modifying the model or viewing the evaluation information.

[0012] First, the operation of the model generation system 1 will be outlined. The model generation device 100 generates a machine learning model (hereinafter, also simply referred to as a "model") using pre-prepared training data. Further, the model generation device 100 analyzes and evaluates the generated model. Specifically, the model generation device 100 makes predictions using the model with evaluation data and analyzes the prediction performance of the model based on the prediction results. The model generation device 100 calculates a value indicating the performance of the model (hereinafter, also referred to as a "performance value") for each attribute of the category of the evaluation data. Then, the model generation device 100 presents information indicating the performance of the model for each attribute of the category to the user as evaluation information. The user can check the evaluation information and input correction information for correcting the model by operating the input device 3.

[0013] Note that the "machine learning model" is information representing the relationship between the explanatory variable and the objective variable. The machine learning model is, for example, a component for estimating the result of the estimation target by calculating the target variable based on the explanatory variable. The machine learning model is generated by executing a learning algorithm with the learning data for which the value of the objective variable has already been obtained and an arbitrary parameter as inputs. The machine learning model may be represented, for example, by a function c that maps the input x to the correct answer y. The machine learning model may estimate the numerical value of the estimation target or may estimate the label of the estimation target. The machine learning model may output a variable that describes the probability distribution of the objective variable. The machine learning model may also be described as a "learning model", "analysis model", "AI (Artificial Intelligence) model" or "prediction formula". Further, the explanatory variable is a variable used as an input in the machine learning model. The explanatory variable may be described as a "feature amount" or "feature".

[0014] In addition, the learning algorithm for generating the machine learning model is not particularly limited, and existing learning algorithms may be used. For example, the learning algorithm may be a random forest, a support vector machine, a naive Bayes, a piecewise linear model using FAB inference (Factorized Asymptotic Bayesian Inference), or a neural network. Note that the method of the piecewise linear model using FAB inference is disclosed in, for example, U.S. Patent Publication No. US2014 / 0222741A1.

[0015] [Hardware Configuration] FIG. 2 is a block diagram showing the hardware configuration of the model generation device 100. As shown in the figure, the model generation device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0016] The I / F 111 inputs and outputs data to and from an external device. Specifically, the training data, evaluation data, and instructions and inputs entered by the user using the input device 3 for generating the model are input to the model generation device 100 through the I / F 111. In addition, the evaluation information of the model generated by the model generation device 100 is output to the display device 2 through the I / F 111.

[0017] The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generation device 100 by executing a pre-prepared program. Note that the processor 112 may be a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, or an FPGA (Field-Programmable Gate Array). The processor 112 executes the model analysis process described later.

[0018] The memory 113 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory during the execution of various processes by the processor 112.

[0019] The recording medium 114 is a non-volatile and non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 stores various programs executed by the processor 112. When the model generation device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0020] The DB 115 stores information regarding the models generated by the model generation device 100 (hereinafter referred to as "existing models") and the models after correction by retraining (hereinafter referred to as "corrected models"). Further, the DB 115 stores, as necessary, training data, evaluation data, correction information input by the user, etc. input through the I / F 111.

[0021] (Functional configuration) FIG. 3 is a block diagram showing the functional configuration of the model generation device 100 according to the first embodiment. Functionally, the model generation device 100 includes a training data DB 121, a model training unit 122, a model DB 123, an evaluation data DB 124, and an analysis unit 125.

[0022] The training data DB 121 stores the training data used for generating the model. The training data D1 is input to the model training unit 122. Note that the training data D1 is composed of a plurality of combinations of input data and correct answers (correct values or correct labels) for the input data.

[0023] The model training unit 122 trains a model using the training data D1 and generates a model. The model training unit 122 outputs the model data M corresponding to the generated model to the model DB123 and the analysis unit 125. Note that the model data M includes a plurality of parameter information constituting the model. The parameter information includes, for example, information on explanatory variables used as inputs to the model, information on weights for each explanatory variable, information on weights for each sample constituting the input data, and the like. The weights for each explanatory variable may be set for each attribute of the category of each explanatory variable.

[0024] Also, the model training unit 122 retrains an existing model to generate a modified model. In this case, the model training unit 122 modifies the parameters constituting the model based on the modification information D3 input by the user using the input device 3, and retrains the model using the training data for retraining as necessary. The model training unit 122 stores the model data M of the modified model obtained by retraining in the model DB123 and outputs it to the analysis unit 125. The model training unit 122 is an example of a parameter change means and a training means.

[0025] Here, an example of the retraining method is shown. Examples of the retraining method by the model training unit 122 include a method of completely retraining a completely new modified model ignoring the existing model, and a method of updating the existing model using new training data. In addition, as a method of retraining the existing model, there are a method of updating only the coefficients by retraining without changing the types of explanatory variables used in the model, and a method of retraining including the selection of explanatory variables. However, the retraining method of the model is not limited to the methods described above.

[0026] The evaluation data DB124 stores evaluation data used for evaluating the generated model. The evaluation data is, for example, among the collected data, some data not used as training data, newly collected data, validation data prepared for verification, and the like. Note that the training data may be used as evaluation data. The evaluation data is composed of a plurality of combinations of input data and correct answers (correct values or correct labels) for the input data.

[0027] The evaluation data is stored and used in units of datasets. Each data included in the dataset includes a plurality of categories. For example, in the case of a prediction model for determining credit, data of a large number of persons to be determined is prepared as evaluation data, and each data includes categories such as age group, gender, residential area, family composition, income, and the like. Further, each category includes a plurality of attributes (groups).

[0028] The analysis unit 125 analyzes the performance of the model for each attribute corresponding to each category of the dataset using the evaluation data. For example, assume that a certain category "gender" includes attributes "male" and "female". In this case, the analysis unit 125 calculates performance values indicating the performance of the model for each of the attributes "male" and "female". The analysis unit 125 also calculates the performance value of the model for each attribute for other categories. Note that as performance indicators of the model, for example, various indicators that can be used to evaluate the performance of the model, such as the prediction accuracy and compatibility of the model, can be used. As other examples of performance indicators, an F1 score, precision, and recall may be used, and in the case of a regression task, a coefficient of determination or the like may be used. Further, as an error-based indicator for which a lower value is considered better, for example, a mean squared error or cross entropy may be used. Then, the analysis unit 125 generates information indicating the performance value of the model for each attribute for a plurality of categories (hereinafter, also referred to as "performance information by attribute"), and outputs it to the display device 2 as evaluation information D2. The analysis unit 125 is an example of a model acquisition means, a dataset acquisition means, a performance calculation means, and an output means.

[0029] The display device 2 displays the evaluation information D2 output by the analysis unit 125 on the display device 2. By viewing the displayed performance information by attribute, the user can know the differences in the performance of the models for each attribute in each category.

[0030] Also, the user inputs correction information D3 for correcting the current model into the input device 3 as needed. The correction information D3 is information related to corrections such as, for example, information on explanatory variables used as inputs to the model, information on weights for each explanatory variable, information on weights set for each attribute of the category of explanatory variables, and information on weights for each sample constituting the input data. The model training unit 122 corrects the model by retraining the model using the input correction information D3.

[0031] [Example of Displaying Performance Information by Attribute] Next, an example of displaying performance information by attribute will be described. FIG. 4(A) shows an example of displaying performance information by attribute. In the display example 40, the performance values of the model are displayed for each attribute for the entire dataset used for prediction. In the following example, the accuracy of the model is used as the performance value of the model.

[0032] Specifically, the display example 40 includes an overall performance value column 41, a category column 42, a performance value column 43, and a training button 44. The overall performance value column 41 shows the performance value of the model for the entire target dataset. In this example, "Overall accuracy 85%" is shown as the performance value of the model for the entire dataset. The category column 42 shows the categories for which the performance values of the model are to be displayed. In this example, the performance values of the model are displayed for the four categories of "gender", "age", "residence", and "annual income".

[0033] The performance value column 43 shows the performance values of the model for each attribute for each category. Fig. 4(B) shows the part of the performance value column 43 corresponding to the category "gender" indicated by reference numeral 45 in Fig. 4(A). The performance value column 43 includes, for each category, attribute frames 46a to 46b, parameter value display columns 47a to 47b, and operation buttons 48a to 48b, 49a to 49b. In Fig. 4(B), the attribute frame 46a corresponds to the attribute "male", and the attribute frame 46b corresponds to the attribute "female". Also, the parameter value display column 47a corresponds to the attribute "male", and the parameter value display column 47b corresponds to the attribute "female".

[0034] The parameter value display columns 47a to 47b show the current values of the parameters used by the model for prediction. In this example, for instance, it is assumed that the parameter is a weight indicating the importance of each attribute. In Fig. 4(B), the parameter value display column 47a shows that the weight of the model for the explanatory variable of the attribute "male" in the category "gender" is "0.45". Also, the parameter value display column 47b shows that the weight of the model for the explanatory variable of the attribute "female" in the category "gender" is "0.50". The operation buttons 48a to 48b, 49a to 49b are buttons for the user to increase or decrease the parameter values for each attribute. When the user wants to increase the parameter value, the user presses the operation buttons 48a to 48b, and when the user wants to decrease the parameter value, the user presses the operation buttons 49a to 49b.

[0035] In Fig. 4(A), for example, for the category "gender", the performance value for the attribute "male" is 95%, and the performance value for the attribute "female" is 60%. Note that the ratio of each attribute is shown within the parentheses of each attribute frame 46 in the performance value column 43. For example, in the example of Fig. 4(A), for the category "gender", the ratio of males in the entire dataset is 60%, and the ratio of females is 40%. Also, the size (horizontal length) of each attribute frame corresponds to the ratio of each attribute. For example, for the category "gender", the lengths of the attribute frame 46a for "male" and the attribute frame 46b for "female" are in the ratio of 3:2 (60:40).

[0036] Regarding the category "Age", the performance value for the attribute "Child" is 35%, the performance value for the attribute "Adult" is 94%, and the performance value for the attribute "Senior" is 95%. In the category "Age", the proportion of the attribute "Child" is 20%, the proportion of the attribute "Adult" is 60%, and the proportion of the attribute "Senior" is 20%.

[0037] Regarding the category "Residence", the performance value for the attribute "Kanto" is 84%, and the performance value for the attribute "Kansai" is 86%. In the category "Residence", the proportion of the attribute "Kanto" is 55%, and the proportion of the attribute "Kansai" is 45%.

[0038] Regarding the category "Annual Income", the performance value for the attribute "Less than 4 million" is 83%, and the performance value for the attribute "4 million or more" is 87%. In the category "Annual Income", the proportion of the attribute "Less than 4 million" is 48%, and the proportion of the attribute "4 million or more" is 52%.

[0039] The category column 42 and the performance value column 43 are shaded based on the fairness of the model's performance. In this specification, for the sake of illustration, hatching is used instead of color-coding for distinction. A small difference in the model's performance value between different attributes within the same category is said to be "the model's performance is fair", and a large difference in the model's performance value is said to be "the model's performance is unfair". The category column 42 and the performance value column 43 are shaded according to the type of hatching depending on whether the model's performance is fair or unfair.

[0040] Now, assume that the threshold for determining that the performance of the model is unfair is "5%". For the category "gender", the accuracy of the attribute "male" is 95% and the accuracy of the attribute "female" is 60%. Since the difference is equal to or greater than the threshold "5%", the performance of the model is determined to be unfair. Therefore, as shown in the legend of FIG. 4(A), the attribute frame of "male" with a high performance value is indicated by the hatching of pattern P1, and the attribute frame of "female" with a low performance value is indicated by the hatching of pattern P2. In actual display, for example, by color-coding, it is preferable to display the attribute frame with a lower performance value in "red" and the attribute frame with a higher performance value in "yellow" so that the attribute frame with a lower performance value is more prominent. Also, the category column 42 is indicated by the hatching of pattern P2, similar to the attribute frame with a lower performance value. As a result, even by simply looking at the category column 42, the user can easily recognize that the performance of the model for that category is unfair.

[0041] Regarding the category "age" as well, since the performance value of the attribute "child" is lower than the above-mentioned threshold compared to the performance values of the attributes "adult" and "elderly", the performance of the model is determined to be unfair. As a result, the category column 42 and the attribute frame of "child" are indicated by the hatching of pattern P2, and the attribute frames of "adult" and "elderly" are indicated by the hatching of pattern P1.

[0042] On the other hand, for the category "residence", since the difference between the performance value of the attribute "Kanto" and the performance value of the attribute "Kansai" is less than or equal to the above-mentioned threshold, the performance of the model is determined to be fair. Therefore, both the category column 42 and the performance value column 43 are indicated by the hatching of pattern P3, which indicates that the performance of the model is fair.

[0043] Regarding the category "annual income" as well, since the difference between the performance value of the attribute "less than 4 million" and the performance value of the attribute "4 million or more" is less than or equal to the above-mentioned threshold, the performance of the model is determined to be fair. Therefore, both the category column 42 and the performance value column 43 are indicated by the hatching of pattern P3, which indicates that the performance of the model is fair.

[0044] As described above, according to the display of performance information by attribute shown in FIG. 4(A), for each category, the performance values of the model are displayed for each attribute. Therefore, the user can know that even for the same model, the performance value of the model varies depending on the attributes of the category included in the dataset used for prediction. Also, for each category, it is determined whether the performance of the model is fair based on the performance values for each attribute, and the categories where the performance of the model is fair and those where it is unfair are displayed separately, so that the user can immediately know the fairness of the model for each category.

[0045] When the user thinks that the model needs to be modified after viewing the performance information by attribute, the user can operate the input device 3 to input the correction information D3 and instruct the retraining of the model. Specifically, the user checks the current parameter values by looking at the parameter value display column 47 for each attribute, and uses the cursor C to operate the operation buttons 48 and 49 to input the increase or decrease of the parameter values. Then, the user can instruct the retraining of the model by pressing the training button 44. When the training button 44 is pressed, the input device 3 transmits the instruction for retraining and the parameter values changed by the operation of the operation buttons 48 and 49 to the model training unit 122 of the model generation device 1 as the correction information D3.

[0046] The model training unit 122 retrains the model using the changed parameter values, and outputs the model data M after retraining to the model DB 123 and the analysis unit 125. Then, the analysis unit 125 calculates the performance information by attribute for the model after retraining and displays it on the display device 2. In this way, the performance information by attribute is displayed for the model after the adjustment of the parameter values by the user.

[0047] Note that the model training unit 122 may change other parameter values according to the parameter values of the attributes changed by the user. For example, when the user increases the weight of the attribute "male" in the category "gender" by 0.01, the model training unit 122 accordingly decreases the weight of the attribute "female" in the category "gender" by 0.01. That is, the model training unit 122 normalizes the parameter values of each attribute for each category using the parameter values after the change by the user, and retrains the model.

[0048] The user can retrain the model so that the performance of the model becomes fair by adjusting the parameter values for the categories where the performance of the model is particularly unfair after looking at the performance information by attribute shown in FIG. 4(A). For example, in the example of FIG. 4(A), regarding the category "gender", the performance value for gender "male" is high and the performance value for gender "female" is low. Basically, the performance value increases by increasing the weight value, and the performance value decreases by decreasing the weight value. Therefore, the user may operate the operation buttons 48 or 49 so as to decrease the weight of the attribute "male" or increase the weight of the attribute "female".

[0049] As described above, the user corrects the parameter values for each category where the performance of the model is unfair and performs retraining. FIG. 5 is an example of the display of the performance information by attribute after the model has been retrained. In this example, for each category, since the performance of the model has become fair, the hatching in the category column 42 and the performance value column 43 for each category is all pattern P3.

[0050] [Model Analysis Process] Next, the model analysis process by the model generation device 100 will be described. FIG. 6 is a flowchart of the model analysis process by the model generation device 100. The model analysis process is a process of calculating the performance of the model for each attribute of each category and displaying the performance information by attribute as described above. This process is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating mainly as the model training unit 122 and the analysis unit 125 shown in FIG. 3.

[0051] First, the analysis unit 125 acquires a target data set (step S10). Next, the analysis unit 125 divides the data set for each category to create partial data sets, and calculates the performance value of the model for each attribute for each category by performing prediction using the model (step S11). Then, the analysis unit 125 displays the performance information by attribute on the display device 2 as illustrated in FIGS. 4 and 5 (step S12).

[0052] Next, the model training unit 122 determines whether the user has changed the parameter value using the input device 3 (step S13). When the parameter value is changed (step S13: Yes), the model training unit 122 retrains the model using the changed parameter value (step S14). Then, the model analysis unit 125 repeats steps S11 to S12 for the retrained model. In this way, each time the user changes the parameter value, performance information by attribute is generated and displayed for the model retrained using the changed parameter value.

[0053] On the other hand, when the parameter value has not been changed (step S13: No), the analysis unit 125 determines whether an end instruction has been input by the user (step S15). When the end instruction has not been input (step S15: No), the process returns to step S13. On the other hand, when the end instruction has been input (step S15: Yes), the process ends.

[0054] When calculating the performance value of the model for each attribute in step S11, if the data included in the data set is a continuous variable such as numerical data, the analysis unit 125 generates an attribute by converting the continuous variable into a category. Specifically, the analysis unit 125 converts continuous variables such as age and annual income into ordered categories. For example, the analysis unit 125 equally divides the range between the minimum value and the maximum value taken by the continuous variable into two or more attributes (groups) and converts them into categories.

[0055] In addition, when the analysis unit 125 can prepare a split point based on business knowledge, it may convert a continuous variable into a category according to the split point. For example, in the case of age, the analysis unit 125 creates three attributes such as "child", "adult", and "elderly" based on business knowledge. Further, the analysis unit 125 may determine a split point according to the percentile so that the number of data samples for each attribute is the same, and convert the continuous variable into a category. In addition, the analysis unit 125 may create a plurality of attributes by clustering the data (variables) included in the data set. Note that when the variables included in the data set are originally categories such as gender and place of residence, the above processing is unnecessary.

[0056] [Modification Example] (Modification Example 1) In a state where the performance information by attribute is displayed as shown in FIG. 4(A), the analysis unit 125 may present guide information regarding adjustment of parameter values to the user. For example, in the example of FIG. 4(A), the performance of the model is unfair for the category "gender", and as described above, the performance of the model approaches fairness by reducing the weight of the attribute "male". Therefore, the analysis unit 125 may guide the user to reduce the weight of the attribute "male" by making the operation button 49b corresponding to the attribute "male" a different color (in this example, a white button) from the other operation buttons 48a to 48b and 49a, as shown in FIGS. 4(A) and 4(B). Instead, the analysis unit 125 may display a message such as "Regarding gender, the performance of the model approaches fairness by reducing the weight of males or increasing the weight of females."

[0057] (Modification Example 2) In the above-described embodiment, the model generation device 100 is configured as an independent device such as a PC. Instead, the model generation device may be configured by a server and a terminal device. FIG. 7 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 7, the server 100x has the configuration of the model generation device 100 shown in FIG. 3. Also, the display device 2x and the input device 3x of the terminal device 7 used by the user are used as the display device 2 and the input device 3 shown in FIG. 3.

[0058] <Second Embodiment> FIG. 8 is a block diagram showing a functional configuration of the model analysis device 70 according to the second embodiment. The model analysis device 70 includes a model acquisition unit 71, a dataset acquisition unit 72, a performance calculation unit 73, an output unit 74, a parameter change unit 75, and a training unit 76.

[0059] FIG. 9 is a flowchart of the processing by the model analysis device 70 according to the second embodiment. The model acquisition unit 71 acquires a model (step S71). The dataset acquisition unit 72 acquires a dataset (step S72). The performance calculation unit 73 calculates the performance of the model for each attribute corresponding to each category of the dataset (step S73). The output unit 74 outputs performance information indicating the performance of the model for each calculated attribute (step S74). The parameter change unit 75 receives a change in the parameters of the model corresponding to the attribute (step S75). The training unit 76 retrains the model using the changed parameters (step S76). When the model is retrained, the performance calculation unit 73 calculates the performance of the retrained model, and the output unit 74 outputs the calculated performance of the model (step S77).

[0060] According to the model analysis device 70 of the second embodiment, it is possible to evaluate the performance of the model for each attribute of the category of the dataset used for prediction and correct the model so as to reduce the difference in the performance of the model.

[0061] Some or all of the above embodiments may be described as follows in the following appendices, but are not limited thereto.

[0062] (Appendix 1) Model acquisition means for acquiring a model, Dataset acquisition means for acquiring a dataset, Performance calculation means for calculating the performance of the model for each attribute corresponding to each category of the dataset, Output means for outputting performance information indicating the performance of the model for each calculated attribute, Parameter change means for receiving a change in the parameters of the model corresponding to the attribute, Training means for retraining the model using the changed parameters, comprising The performance calculation means is a model analysis device that calculates the performance of the retrained model when the model is retrained.

[0063] (Appendix 2) The model analysis device according to Appendix 1, wherein the performance information includes the values of the current parameters of the model for each attribute.

[0064] (Appendix 3) The model analysis device according to Appendix 1 or 2, wherein the performance information includes an operation button for changing the value of the parameter for each attribute and an operation button for instructing training of the model.

[0065] (Appendix 4) The model analysis device according to any one of Appendices 1 to 3, wherein the performance information separately indicates a category in which the difference in the performance of the model for each attribute is equal to or greater than a predetermined value and a category in which the difference is less than the predetermined value.

[0066] (Appendix 5) The model analysis device according to any one of Appendices 1 to 3, wherein the performance information includes information indicating the ratio of the number of data for each attribute corresponding to each category.

[0067] (Appendix 6) Obtain a model, Obtain a dataset, For each attribute corresponding to each category of the dataset, calculate the performance of the model, Output performance information indicating the performance of the model for each calculated attribute, Receive a change in the parameters of the model corresponding to the attribute, Retrain the model using the changed parameters and calculate the performance of the retrained model. A model analysis method.

[0068] (Appendix 7) Obtain a model, Obtain a dataset, For each attribute corresponding to each category of the dataset, calculate the performance of the model, Output performance information indicating the performance of the model for each calculated attribute, Receive a change in the parameters of the model corresponding to the attribute, A recording medium recording a program for causing a computer to execute a process of retraining the model using the changed parameters and calculating the performance of the retrained model.

[0069] The present disclosure has been described above with reference to embodiments and examples, but the present disclosure is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

Explanation of Reference Numerals

[0070] 1, 1x Model generation system 2, 2x Display device 3, 3x Input device 7 Terminal device 100 Model generation device 112 Processor 121 Training data DB 122 Model training unit 123 Model DB 124 Evaluation data DB 125 Analysis unit

Claims

1. Model acquisition means for acquiring a model, Dataset acquisition means for acquiring a dataset, Performance calculation means for calculating the performance of the model for each attribute corresponding to each category of the dataset, Output means for outputting performance information indicating the performance of the model for each calculated attribute, Parameter change means for receiving a change in the parameters of the model corresponding to the attribute, Training means for retraining the model using the changed parameters, comprising, The performance calculation means calculates the performance of the retrained model when the model is retrained. A model analysis device.

2. The model analysis device according to claim 1, wherein the performance information includes the value of the current parameters of the model for each attribute.

3. The model analysis device according to claim 1 or 2, wherein the performance information includes an operation button for changing the value of the parameter for each attribute and an operation button for instructing training of the model.

4. The model analysis device according to any one of claims 1 to 3, wherein the performance information distinguishes and shows a category in which the difference in the performance of the model for each attribute is equal to or greater than a predetermined value and a category in which the difference is less than the predetermined value.

5. The model analysis device according to any one of claims 1 to 3, wherein the performance information includes information indicating the ratio of the number of data for each attribute corresponding to each category.

6. A model analysis method executed by a computer, comprising: acquiring a model, acquiring a dataset, calculating the performance of the model for each attribute corresponding to each category of the dataset, outputting performance information indicating the performance of the model for each calculated attribute, receiving a change in the parameters of the model corresponding to the attribute, retraining the model using the changed parameters, and calculating the performance of the retrained model. A model analysis method.

7. acquiring a model, acquiring a dataset, calculating the performance of the model for each attribute corresponding to each category of the dataset, outputting performance information indicating the performance of the model for each calculated attribute, receiving a change in the parameters of the model corresponding to the attribute, A program that causes a computer to execute a process of retraining the model using the changed parameters and calculating the performance of the retrained model.

Images