Model analysis device, model analysis method, and program

Abstract

In this model analysis device, a predicted value acquisition means acquires a predicted value of a model with respect to input data. An output means outputs assessment information including: a graph that indicates predicted values and actual measured values; and a display area for displaying a determination criterion for prediction mistakes. A criterion acquisition means acquires the determination criterion. An extraction means extracts a predicted value corresponding to a prediction mistake and indicates the predicted value on the graph, on the basis of the determination criterion.

Description

[Technical Field]

[0001] The present disclosure relates to the analysis of machine learning models. [Background technology]

[0002] In recent years, prediction models obtained by machine learning have been used in various fields. Patent Document 1 describes a method for predicting real estate prices using a prediction model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2020 / 004049 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 describes a method for displaying pairs of data samples in which the way predictions are made by the model (predicted value - actual value) is reversed, i.e., the positive and negative prediction errors are displayed. However, the evaluation of a model changes depending on how the error is defined in the first place.

[0005] One object of the present disclosure is to provide a model analysis device that can define an error of a prediction model and appropriately evaluate the model based on the defined error. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a model analysis apparatus includes: a predicted value acquisition means for acquiring a predicted value of a model for input data; an output means for outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction error has occurred; a criterion acquisition means for acquiring the judgment criterion; an extracting means for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying the extracted predicted values ​​on the graph; Equipped with 、 The judgment criteria include an error index that specifies the type of error between the predicted value and the actual measurement value, and information that defines a threshold value for judging the predicted value to be a prediction error. . In another aspect of the present disclosure, a model analysis apparatus includes: a predicted value acquisition means for acquiring a predicted value of a model for input data; an output means for outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction error has occurred; a criterion acquisition means for acquiring the judgment criterion; an extracting means for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying the extracted predicted values ​​on the graph; Equipped with The extracting means changes the judgment criteria when a specific predicted value on the graph is designated by a user so as to judge the designated predicted value as a prediction error, and extracts predicted values ​​corresponding to a prediction error based on the changed judgment criteria. . In yet another aspect of the present disclosure, a model analysis apparatus includes: a predicted value acquisition means for acquiring a predicted value of a model for input data; an output means for outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction error has occurred; a criterion acquisition means for acquiring the judgment criterion; an extracting means for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying the extracted predicted values ​​on the graph; Equipped with the evaluation information is displayed on the graph of the predicted values ​​and includes a mark indicating the predicted value corresponding to the prediction error; the display area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values ​​corresponding to prediction errors; The evaluation information includes a mark indicating a predicted value selected according to a rule selected by a user. .

[0007] of the present disclosure moreover In another aspect, the computer-implemented model analysis method comprises: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; Based on the judgment criteria, predicted values ​​corresponding to prediction errors are extracted and displayed on the graph. death, The judgment criteria include an error index that specifies the type of error between the predicted value and the actual measurement value, and information that defines a threshold value for judging the predicted value to be a prediction error. . In yet another aspect of the present disclosure, a computer-implemented model analysis method includes: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying them on the graph; When a specific predicted value on the graph is designated by a user, the extraction of the predicted value is performed by changing the judgment criteria so that the designated predicted value is determined to be a prediction error, and extracting a predicted value corresponding to a prediction error based on the changed judgment criteria. . In yet another aspect of the present disclosure, a computer-implemented model analysis method includes: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying them on the graph; the evaluation information is displayed on the graph of the predicted values ​​and includes a mark indicating the predicted value corresponding to the prediction error; the display area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values ​​corresponding to prediction errors; The evaluation information includes a mark indicating a predicted value selected according to a rule selected by a user. .

[0008] In yet another aspect of the disclosure, a program includes: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; Based on the judgment criteria, a predicted value corresponding to a prediction error is extracted, and the computer is caused to execute the process shown on the graph. 、 The judgment criteria include an error index that specifies the type of error between the predicted value and the actual measurement value, and information that defines a threshold value for judging the predicted value to be a prediction error. . In yet another aspect of the disclosure, a program includes: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and causing a computer to execute processing shown on the graph; When a specific predicted value on the graph is designated by a user, the extraction of the predicted value is performed by changing the judgment criteria so that the designated predicted value is determined to be a prediction error, and extracting a predicted value corresponding to a prediction error based on the changed judgment criteria. . In yet another aspect of the disclosure, a program includes: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria, and causing a computer to execute the processing shown on the graph; the evaluation information is displayed on the graph of the predicted values ​​and includes a mark indicating the predicted value corresponding to the prediction error; the display area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values ​​corresponding to prediction errors; The evaluation information includes a mark indicating a predicted value selected according to a rule selected by a user. . [Effects of the Invention]

[0009] According to the present disclosure, it is possible to set criteria for determining errors in a prediction model and appropriately evaluate the model using errors based on the set criteria. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the model generating device. [Figure 3] FIG. 1 is a block diagram showing a functional configuration of a model generation device according to a first embodiment. [Figure 4] 10 shows a first display example of evaluation information. [Figure 5] 10 shows a second display example of evaluation information. [Figure 6] 10 shows a third display example of evaluation information. [Figure 7] Another example of setting the threshold value will be shown below. [Figure 8] 10 is a flowchart of a model analysis process performed by the model generating device. [Figure 9] FIG. 1 is a block diagram showing a schematic configuration of a model generation system using a server and a terminal device. [Figure 10] FIG. 10 is a block diagram showing the functional configuration of a model analysis device according to a second embodiment. [Figure 11] 10 is a flowchart of a process performed by a model analysis device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Overall configuration] FIG. 1 is a block diagram showing the overall configuration of a model generation system according to the first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is an application of the model analysis device of the present disclosure, and is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by the user to give instructions and input necessary when modifying a model or viewing the evaluation information.

[0012] First, an outline of the operation of the model generation system 1 will be described. The model generation device 100 generates a machine learning model (hereinafter simply referred to as a "model") using training data prepared in advance. The model generation device 100 also evaluates the generated model. Specifically, the model generation device 100 performs prediction using the model using evaluation data, etc., detects prediction errors of the model based on the prediction results, and presents the results to the user as evaluation information. The user can check the prediction errors of the model and input correction information for correcting the model by operating the input device 3. Particularly in this embodiment, the user can input criteria for determining prediction errors and further change them as needed. Therefore, the user can set criteria for determining prediction errors from a perspective that the user deems appropriate and appropriately evaluate the model by viewing evaluation information based on the criteria.

[0013] Here, a "model" is information that represents the relationship between explanatory variables and a response variable. A model is, for example, a component that estimates the result of an estimation target by calculating a response variable based on explanatory variables. A model is generated by executing a learning algorithm using training data, for which the value of the response variable has already been obtained, and arbitrary parameters as input. A model may be represented, for example, by a function c that maps an input x to a correct answer y. A model may estimate a numerical value of an estimation target, or may estimate a label of an estimation target. A model may output a variable that describes the probability distribution of a response variable. A model may also be referred to as a "learning model," an "analysis model," an "AI (Artificial Intelligence) model," or a "prediction formula," etc.

[0014] [Hardware configuration] 2 is a block diagram showing the hardware configuration of model generation device 100. As shown in the figure, model generation device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0015] The I / F 111 inputs and outputs data to and from external devices. Specifically, training data and evaluation data used in generating a model, as well as instructions and inputs entered by a user using the input device 3, are input to the model generation device 100 via the I / F 111. Furthermore, evaluation information for a model generated by the model generation device 100 is output to the display device 2 via the I / F 111.

[0016] The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generation device 100 by executing a prepared program. The processor 112 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The processor 112 executes the model analysis process described below.

[0017] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0018] The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 records various programs to be executed by the processor 112. When the model generation device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0019] DB 115 stores information about the model generated by model generation device 100 (hereinafter referred to as the "existing model") and the model corrected by retraining (hereinafter referred to as the "corrected model"). DB 115 also stores, as necessary, training data input via I / F 111, evaluation data, correction information input by the user, a history of prediction error criteria input by the user, and the like.

[0020] (Functional configuration) 3 is a block diagram showing the functional configuration of the model generation device 100 according to the first embodiment. Functionally, the model generation device 100 includes a training data DB 121, a model training unit 122, a model DB 123, an evaluation data DB 124, a prediction error analysis unit 125, and an evaluation information output unit 126.

[0021] The training data DB 121 stores training data used to generate a model. The training data D1 is input to the model training unit 122. The training data D1 is composed of multiple combinations of input data and correct labels (teacher labels) for the input data.

[0022] The model training unit 122 trains a model using the training data D1 and generates a model. The model training unit 122 outputs model data M corresponding to the generated model to the model DB 123 and the prediction error analysis unit 125. The model data M includes information on multiple parameters that constitute the model. The parameter information includes, for example, information on explanatory variables (or feature amounts) used as input to the model, information on weights for each explanatory variable, and information on weights for each sample that constitutes the input data.

[0023] Furthermore, the model training unit 122 retrains the existing model to generate a corrected model. In this case, the model training unit 122 corrects the parameters constituting the model based on correction information D3 input by the user using the input device 3, and retrains the model using training data for retraining as necessary. The model training unit 122 stores model data M of the corrected model obtained by retraining in the model DB 123 and outputs it to the prediction error analysis unit 125.

[0024] The evaluation data DB 124 stores evaluation data used to evaluate the generated model. The evaluation data includes various types of data that can be used to evaluate the model. The evaluation data is basically composed of multiple combinations of input data and correct labels (teacher labels) for the input data. Examples of evaluation data include the following: (1) Validation data or test data, which are data that were not used to generate the model. In this case, the evaluation data is basically a set of input data and correct labels. (2) "Data newly collected after the model is generated," such as operational data If labeling is not performed immediately, the evaluation data may be input data only. (3) "Data that is generated in some way and is unknown to the model" For example, if the features in the input data are (day of the week, public holidays, weather), pseudo-future data can be created using calendar information and weather forecasts. (4) "Data identical to the training data" The training data used to generate the model can be used as the evaluation data, in which case the same data as the training data may be stored in the evaluation data DB 124 as the evaluation data.

[0025] The prediction error analysis unit 125 analyzes prediction errors of an existing model using evaluation data. Specifically, the prediction error analysis unit 125 inputs the input data of the evaluation data into the existing model, performs prediction, and obtains the prediction result. Then, based on the evaluation data used and the prediction result, the prediction error analysis unit 125 extracts prediction errors made by the existing model from the prediction result of the model.

[0026] Here, the definition of a prediction error, i.e., the criteria for judging a prediction error, is set by the user. The prediction error analysis unit 125 analyzes the prediction results of the model based on the criteria set by the user. Information D4 of the criterion set by the user is sent from the input device 3 to the prediction error analysis unit 125. The prediction error analysis unit 125 extracts prediction errors included in the prediction results according to the acquired criteria. The prediction error analysis unit 125 then outputs the prediction results of the model and the extracted prediction errors to the evaluation information output unit 126. Note that, if necessary, the prediction error analysis unit 125 also outputs the used evaluation data to the evaluation information output unit 126. Note that a method for setting the criteria for judging a prediction error will be described in detail later. The prediction error analysis unit 125 is an example of a prediction value acquisition means, a criterion acquisition means, and an extraction means.

[0027] The evaluation information output unit 126 generates evaluation information D2 for evaluating the existing model based on the information input from the prediction error analysis unit 125. Specifically, the evaluation information D2 includes information indicating the relationship between the actual measurement value and the prediction result (predicted value) by the existing model, and the detected prediction error. Then, the evaluation information output unit 126 outputs the generated evaluation information D2 to the display device 2. The evaluation information output unit 126 is an example of an output means.

[0028] The display device 2 displays the evaluation information D2 output by the evaluation information output unit 126 on the display device 2. This allows the user to evaluate the performance of the existing model by referring to the relationship between the actual measurement value and the predicted value by the existing model and information indicating the prediction error contained in the predicted value by the model. Examples of information indicating the prediction error include information indicating a sample of the predicted value corresponding to the prediction error (hereinafter referred to as a "prediction error sample"). If necessary, the user inputs correction information D3 to the input device 3 to correct the model so that the prediction error does not occur. The correction information D3 is information regarding corrections such as information on explanatory variables used as input to the model, information on weights for each explanatory variable, and information on weights for each sample constituting the input data. The model training unit 122 corrects the model by retraining the model using the input correction information D3.

[0029] [Example of evaluation information display] Next, a display example of the evaluation information displayed on the display device 2 will be described. (First display example) FIG. 4 shows a first display example of evaluation information. In this example, the prediction model is a model for predicting sales of a certain product. Note that FIG. 4 shows a display example after the user has already set criteria for judging prediction errors. The first display example 40 includes graphs 41a to 41c and an input area 42. Graph 41a shows predicted values ​​by the model, and graph 41b shows actual measured values. The horizontal axes of graphs 41a and 41b indicate dates in a certain month, and the vertical axes indicate sales. Marks 41x indicating prediction error samples are displayed on the predicted value graph 41a.

[0030] Graph 41c is a graph showing an error index for evaluating the error between the model-predicted value and the actual measured value. In the example of FIG. 4, graph 41c is a bar graph showing the absolute error between the model-predicted value and the actual measured value. The horizontal axis of graph 41c indicates the date, and the vertical axis indicates the absolute error. A threshold value 41d is shown on graph 41c. The threshold value is used to extract misprediction samples based on the absolute error specified as the error index.

[0031] The input area 42 is an area where the user sets the criteria for misprediction. That is, the user sets the criteria for misprediction that he or she wants to extract by inputting the necessary information into the input area 42. In the example of FIG. 4, an error index and a threshold value are set as the criteria. In this case, the above-mentioned prediction error analysis unit 125 extracts, as misprediction samples, samples for which the error between the predicted value based on the error index set by the user and the actual measured value is greater than the threshold value set by the user.

[0032] The "error index" may be an error such as absolute error or squared error. The user operates the input area 43 to set the error index they wish to use. The "threshold" is defined by a reference threshold value and a threshold adjustment parameter. In this example, the threshold adjustment parameter is a scaling factor indicating how many times the threshold is larger than the reference value. The reference threshold value is defined by the type of data to be used and the average error corresponding to the error specified as the error index. The data to be used may include, for example, training data, validation data, training data for a predetermined period, and validation data for a predetermined period. The user operates the input area 44 to set the reference threshold value they wish to use. For example, if the user specifies absolute error as the error index and uses validation data, they select "Validation Data MAE" as shown in Figure 4. "MAE" stands for Mean Absolute Error (MAE). If squared error is used as the error index, mean squared error (MSE) is usually used as the error type.

[0033] The user also operates the input area 45 to set an arbitrary magnification as a threshold adjustment parameter. The threshold value is the product of the threshold reference value set in the input area 44 and the magnification set in the input area 45. In the example of FIG. 4, the user sets the validation data MAE in the input area 44 and the magnification "2" in the input area 45, so the threshold is Threshold = 2 × MAE_va "MAE_va" is the calculated value of the mean absolute error (MAE) of the validation data.

[0034] An OK button 49 is displayed in the input area 42. The OK button 49 is a button for the user to indicate that the setting of the judgment criteria has been completed in the input area 42. Furthermore, the input area 42 displays the number of extracted mispredicted samples as an extraction result 48.

[0035] 4 is a display example after the user has already set the criteria for determining prediction errors. In the initial state, i.e., before the user sets the criteria for determining prediction errors, the graph 41a of predicted values ​​and the graph 41b of actual values ​​are displayed, but the graph 41c and the mark 41x indicating a prediction error are not displayed, and each of the input areas 43 to 45 in the input area 42 is empty.

[0036] When the user sets a misprediction criterion and presses the OK button 49, information D4 about the set criterion, specifically, the information entered in the input areas 43 to 45, is transmitted from the input device 3 to the misprediction analysis unit 125. Based on the received criterion information D4, the misprediction analysis unit 125 extracts samples that meet the criterion from the model's predicted values ​​as misprediction samples, and outputs the samples to the evaluation information output unit 126. The evaluation information output unit 126 transmits the received information about the misprediction samples to the display device 2, displays a mark 41x indicating the misprediction sample on the graph 41a, and displays the misprediction sample extraction result 48 in the input area 42. In this way, misprediction samples are extracted according to the criterion set by the user and displayed on the display device 2. As a result, a display such as that shown in FIG. 4 is produced.

[0037] Although the case where the user sets the judgment criteria has been described here, the first display example is not limited to this. For example, the display device 2 may display a predetermined value as the judgment criteria. Alternatively, the display device 2 may display the judgment criteria set by the user in a previous operation. Alternatively, the display device 2 may display the judgment criteria recommended for each user using a machine learning model learned from the input history of the judgment criteria.

[0038] (Second display example) 5 shows a second display example of the evaluation information. The second display example 40a differs from the first display example 40 in that the user can set or modify the criteria for determining prediction errors by specifying a sample on the graph 41a.

[0039] The user looks at the graph of predicted values ​​shown in graph 41a and specifies samples that appear to be mispredictions. Specifically, in the example of FIG. 5, the user determines that the values ​​of the samples on the "8th" and "11th" correspond to mispredictions on the graph of predicted values ​​in graph 41a, and specifies these two samples by clicking them, as indicated by marks 46. The user then presses the OK button 49 in the input area 42. This sends information specifying the two samples indicated by marks 46 to the misprediction analysis unit 125. The misprediction analysis unit 125 modifies the criteria so that the two specified samples are extracted as mispredictions, and outputs information about the mispredictions extracted based on the modified criteria to the evaluation information output unit 126. The evaluation information output unit 126 sends evaluation information including the modified misprediction information to the display device 2 for display.

[0040] 5, the magnification value in the input area 45 is changed to "1," and a modified threshold value 41d and a pre-modification threshold value 41e are displayed. That is, in this example, the magnification value is changed from "2" to "1" to decrease the threshold value so that the two samples specified by the user are determined to be prediction errors, and the threshold value graph is changed accordingly.

[0041] In the above example, the judgment criteria are corrected by specifying a sample on the graph 41a of predicted values, but the judgment criteria may be input in this manner from the beginning. In this case, in the initial state where only the graphs 41a and 41b are displayed, the user inputs data into the input areas 43 and 44, and then, without inputting data into the input area 45, specifies the sample to be judged as a misprediction on the graph 41a of predicted values ​​and presses the OK button 49.

[0042] Thus, according to the second display example, when the user specifies a sample on the graph of predicted values, the misprediction analysis unit 125 sets or modifies the judgment criteria so that the specified sample is judged to be a misprediction. Therefore, even a user who lacks knowledge or experience and finds it difficult to set the input area 42, especially the threshold value, can appropriately set or modify the judgment criteria.

[0043] (Third display example) FIG. 6 shows a third display example of evaluation information. The third display example 40b differs from the first display example 40 in that an option field 47 is provided in the input area 42. The option field 47 is an item for the user to specify a display rule for mispredicted samples. In the example of FIG. 6, display rule R1 "display all mispredicted samples," display rule R2 "display only consecutive mispredicted samples," and display rule R3 "display only the last day if consecutive." are provided.

[0044] Specifically, when the user selects display rule R1, the misprediction analysis unit 125 extracts and displays all misprediction samples that meet the criteria. A graph 41a in Fig. 6 shows an example of this case.

[0045] When the user selects display rule R2, the misprediction analysis unit 125 extracts and displays only consecutive misprediction samples from among multiple misprediction samples that meet the criteria. Therefore, in this case, misprediction samples that are not consecutive along the horizontal axis (date) of the graph 41a are not displayed. For example, in the graph 41a of the display example 40b, consecutive misprediction samples for the 18th and 19th are displayed, but if there are no consecutive misprediction samples before or after them, such as on the 9th or 16th, those misprediction samples are not displayed.

[0046] Furthermore, when the user selects display rule R3, the misprediction analysis unit 125 extracts only the last day of consecutive misprediction samples from among the multiple misprediction samples that meet the criteria. Therefore, for example, if there are two consecutive misprediction samples, only the misprediction sample for the second day will be displayed. Note that the display rules for misprediction samples are not limited to the above three and can be set arbitrarily.

[0047] According to the third display example, the user can select a rule for displaying misprediction samples on the display device 2 from the viewpoint of the purpose of model evaluation, ease of viewing the displayed content, and the like.

[0048] (Another example of threshold settings) In the first to third display examples described above, the threshold is set as "threshold = magnification * reference value," but the threshold setting method is not limited to this, and various other methods can be used. FIG. 7 shows another example of an input area included in the display example. The input area 42x shown in FIG. 7 has input areas 51 and 52, a histogram 53, and a threshold bar 54. The input area 51 is used by the user to specify a dataset, and the input area 52 is used by the user to specify an error index. The histogram 53 shows the error calculated for each sample included in the dataset based on the user's specification. The threshold bar 54 is a bar that the user moves to arbitrarily set the threshold value.

[0049] In the example of FIG. 7, the user first operates input area 51 to specify a dataset, and then operates input area 52 to specify an error index. Here, examples of error indexes that the user can specify include the following: "y" indicates an actual value, and "y_pred" indicates a predicted value. (Example 1) Error y-y_pred (Example 2) Absolute error |y-y_pred| (Example 3) Squared error (y-y_pred) 2 (Example 4) Error rate (y-y_pred) / y (Example 5) Absolute error rate |y-y_pred| / y

[0050] The misprediction analysis unit 125 calculates the error index specified by the user for each sample in a dataset specified by the user, and displays the resulting errors for each sample as a histogram 53. The user can determine the threshold value based on their own intuition by moving a threshold bar 54 while viewing the displayed histogram 53. Note that instead of the user setting the threshold value, the misprediction analysis unit 125 may automatically set the threshold value so that a predetermined percentage (e.g., 20%) of the samples in the dataset are mispredicted samples.

[0051] [Model analysis processing] Next, a description will be given of the model analysis processing performed by the model generation device 100. Fig. 8 is a flowchart of the model analysis processing performed by the model generation device 100. The model analysis processing is processing for extracting prediction errors of an existing model generated by the model training unit 122 and displaying them on the display device 2. This processing is realized by the processor 112 shown in Fig. 2 executing a prepared program and operating as the elements shown in Fig. 3.

[0052] First, the model generation device 100 inputs the evaluation data into an existing model and obtains predicted values ​​from the existing model (step S10). Next, the model generation device 100 generates a graph showing the actual measured values ​​included in the evaluation data and the predicted values ​​from the existing model (step S11). The generated graph is displayed on the display device 2.

[0053] The user looks at a graph of actual values ​​and predicted values ​​displayed, for example, as shown in FIG. 4, and sets a criterion for determining mispredicted samples in the input area 42. The model generating device 100 acquires the set criterion for determining mispredicted samples (step S12). Next, the model generating device 100 extracts and outputs samples of predicted values ​​that meet the acquired criterion as mispredicted samples (step S13). The extracted mispredicted samples are indicated by marks 41x on the graph of predicted values ​​displayed on the display device 2, as shown in FIG.

[0054] Next, the model generating device 100 determines whether or not the user has input an instruction to modify the judgment criterion (step S14). If the instruction to modify the judgment criterion has been input (step S14: Yes), the process returns to step S12, where the model generating device 100 acquires the modified judgment criterion, and extracts and outputs misprediction samples according to the modified judgment criterion (step S13). In this way, the user can evaluate the model while repeatedly modifying the judgment criterion as necessary.

[0055] On the other hand, if an instruction to modify the judgment criteria has not been input (step S14: No), the model generation device 100 determines whether or not an instruction to end the process has been input by the user (step S15). If an instruction to end the process has not been input (step S15: No), the process returns to step S14. On the other hand, if an instruction to end the process has been input (step S15: Yes), the model analysis process ends.

[0056] Note that the user may set or modify the judgment criteria by specifying a sample on the graph of predicted values, as in the second display example shown in Fig. 5. In this case, the model generating device 100 acquires an input specifying a sample on the graph of predicted values ​​in step S12. Then, in step S13, the model generating device 100 modifies the judgment criteria for mispredicted samples so that the sample is determined to be a misprediction, and then extracts the mispredicted sample. Furthermore, as in the third display example shown in Fig. 6, if the user specifies a display rule for mispredicted samples in the option field 47, the model generating device 100 selects the mispredicted samples in accordance with the specified display rule and displays them on the display device 2 in step S13.

[0057] [Variations] In the above embodiment, the model generation device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured by a server and a terminal device. FIG. 9 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 9, the server 100x has the configuration of the model generation device 100 shown in FIG. 3. Furthermore, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG. 3.

[0058] Second Embodiment 10 is a block diagram showing the functional configuration of a model analysis device 70 according to the second embodiment. The model analysis device 70 includes a predicted value acquisition unit 71, an output unit 72, a reference acquisition unit 73, and an extraction unit 74.

[0059] 11 is a flowchart of processing by the model analysis device 70 of the second embodiment. The predicted value acquisition means 71 acquires predicted values ​​of the model for input data (step S71). The output means 72 outputs evaluation information including a graph showing predicted values ​​and actual measured values ​​and a display area for displaying the criterion for judging prediction errors (step S72). The criterion acquisition means 73 acquires the criterion (step S73). The extraction means 74 extracts predicted values ​​corresponding to prediction errors based on the criterion and displays them on a graph (step S74).

[0060] According to the model analysis device 70 of the second embodiment, it is possible to set a criterion for determining an error in a prediction model and to appropriately evaluate the model using the error based on the set criterion.

[0061] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0062] (Appendix 1) a predicted value acquisition means for acquiring a predicted value of a model for input data; an output means for outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction error has occurred; a criterion acquisition means for acquiring the judgment criterion; an extracting means for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying the extracted predicted values ​​on the graph; A model analysis device comprising:

[0063] (Appendix 2) The model analysis device according to claim 1, wherein the evaluation information is displayed on a graph of the predicted values ​​and includes a mark indicating the predicted value corresponding to the prediction error.

[0064] (Appendix 3) 3. The model analysis device according to claim 1, wherein the judgment criteria include an error index that specifies the type of error between the predicted value and the actual measured value, and information that specifies a threshold value for determining that the predicted value is a prediction error.

[0065] (Appendix 4) 4. The model analysis device according to claim 3, wherein the information defining the threshold value includes information indicating a reference value of the threshold value and a parameter for adjusting the threshold value.

[0066] (Appendix 5) 5. The model analysis device according to claim 3, wherein the evaluation information includes a graph showing the error between the predicted value and the actual measurement value using a specified error index, and the threshold value.

[0067] (Appendix 6) The model analysis device according to any one of appendices 1 to 5, wherein the extraction means, when a user specifies a specific predicted value on the graph, changes the judgment criteria so that the specified predicted value is judged to be a prediction error, and extracts predicted values ​​that fall under the prediction error based on the changed judgment criteria.

[0068] (Appendix 7) the input area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values ​​corresponding to prediction errors; 3. The model analysis device according to claim 2, wherein the evaluation information includes a mark indicating a predicted value selected according to a rule selected by a user.

[0069] (Appendix 8) Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; A model analysis method that extracts predicted values ​​corresponding to prediction errors based on the judgment criteria and displays them on the graph.

[0070] (Appendix 9) Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; A recording medium having recorded thereon a program for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and causing a computer to execute processing shown on the graph.

[0071] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0072] 1. 1x Model Generation System 2, 2x display device 3. 3x input devices 7 Terminal Equipment 100 Model generation device 112 processors 121 Training Data DB 122 Model Training Department 123 Model DB 124 Evaluation Data DB 125 Prediction Error Analysis Department 126 Evaluation information output unit

Claims

1. a predicted value acquisition means for acquiring a predicted value of a model for input data; an output means for outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction error has occurred; a criterion acquisition means for acquiring the judgment criterion; an extracting means for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying the extracted predicted values ​​on the graph; Equipped with The judgment criteria include an error index that specifies the type of error between the predicted value and the actual measurement value, and information that defines a threshold value for determining that the predicted value is a prediction error.

2. a predicted value acquisition means for acquiring a predicted value of a model for input data; an output means for outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction error has occurred; a criterion acquisition means for acquiring the judgment criterion; an extracting means for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying the extracted predicted values ​​on the graph; Equipped with The extraction means is a model analysis device that, when a user specifies a specific predicted value on the graph, changes the judgment criteria so that the predicted value is judged to be a prediction error, and extracts predicted values ​​that fall under the prediction error based on the changed judgment criteria.

3. a predicted value acquisition means for acquiring a predicted value of a model for input data; an output means for outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction error has occurred; a criterion acquisition means for acquiring the judgment criterion; an extracting means for extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying the extracted predicted values ​​on the graph; Equipped with the evaluation information is displayed on the graph of the predicted values ​​and includes a mark indicating the predicted value corresponding to the prediction error; the display area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values ​​corresponding to prediction errors; A model analysis device, wherein the evaluation information includes a mark indicating a predicted value selected according to a rule selected by a user.

4. 1. A computer-implemented method for model analysis, comprising: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying them on the graph; A model analysis method in which the judgment criteria include an error index that specifies the type of error between the predicted value and the actual measurement value, and information that defines a threshold for determining that the predicted value is a prediction error.

5. 1. A computer-implemented method for model analysis, comprising: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying them on the graph; The extraction of the predicted value is a model analysis method in which, when a specific predicted value on the graph is specified by a user, the judgment criteria are changed so that the predicted value is judged to be a prediction error, and a predicted value corresponding to a prediction error is extracted based on the changed judgment criteria.

6. 1. A computer-implemented method for model analysis, comprising: Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and displaying them on the graph, and the evaluation information is displayed on the graph of the predicted values ​​and includes a mark indicating the predicted value corresponding to the prediction error, the display area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values ​​corresponding to prediction errors; A model analysis method, wherein the evaluation information includes a mark indicating a predicted value selected according to a rule selected by a user.

7. Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria, and causing a computer to execute the processing shown on the graph; The judgment criteria include an error index that specifies the type of error between the predicted value and the actual measurement value, and information that defines a threshold value for determining that the predicted value is a prediction error.

8. Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria and causing a computer to execute processing shown on the graph; The program extracts the predicted value by changing the judgment criteria so that, when a specific predicted value on the graph is specified by the user, the predicted value is judged to be a prediction error, and extracts the predicted value that corresponds to a prediction error based on the changed judgment criteria.

9. Obtain the model's predictions for the input data, outputting evaluation information including a graph showing the predicted values ​​and the actual measured values ​​and a display area for displaying a criterion for determining whether a prediction is incorrect; Obtaining the criteria; extracting predicted values ​​corresponding to prediction errors based on the judgment criteria, and causing a computer to execute the processing shown on the graph; the evaluation information is displayed on the graph of the predicted values ​​and includes a mark indicating the predicted value corresponding to the prediction error; the display area includes information on a plurality of rules for selecting a predicted value to be displayed from predicted values ​​corresponding to prediction errors; The evaluation information includes a mark indicating a predicted value selected according to a rule selected by a user.

Images