Model analysis apparatus, model analysis method, and program

Abstract

An acquisition means of a model analysis device, according to the present invention, acquires a model prediction result of a model for input data. A determination means uses input data and the prediction result to determine a prediction error factor for the model. An extraction means extracts, on the basis of at least one of input data and the prediction error factor, a plurality of comparison time periods from a target time period of a prediction by the model. A factor output means outputs a distribution for the prediction error factor in the extracted plurality of comparison time periods.

Description

Technical Field

[0001] The present disclosure relates to the analysis of machine learning models.

Background Art

[0002] In recent years, prediction models obtained by machine learning have been used in various fields. Patent Document 1 describes a method for predicting power demand using a prediction model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the accuracy of the created prediction model is not sufficient, or when a certain amount of time has passed since the initial model creation and the trend of the data to be used has changed, etc., retraining of the prediction model is required. When a prediction error occurs in the prediction model, it is necessary to analyze the cause of the prediction error and take countermeasures.

[0005] One object of the present disclosure is to provide a model analysis apparatus capable of analyzing and visualizing the cause of a prediction error in a prediction model.

Means for Solving the Problems

[0006] From one aspect of the present disclosure, the model analysis apparatus an acquisition means for acquiring a prediction result of the model for the input data, a determination means for determining the cause of the prediction error of the model using the input data and the prediction result, an extraction means for extracting a plurality of comparison periods from the target period of the prediction by the model based on at least one of the input data and the cause of the prediction error, Factor output means for outputting the distribution of the prediction error factors in a plurality of extracted comparison periods is provided with.

[0007] In another aspect of the present disclosure, Program executed by a computer the model analysis method obtains the prediction result of the model for the input data, determines the prediction error factors of the model using the input data and the prediction result, extracts a plurality of comparison periods from the target period of the prediction by the model based on at least one of the input data and the prediction error factors, and outputs the distribution of the prediction error factors in the plurality of extracted comparison periods.

[0008] In still another aspect of the present invention, Program is to obtain the prediction result of the model for the input data, determine the prediction error factors of the model using the input data and the prediction result, extract a plurality of comparison periods from the target period of the prediction by the model based on at least one of the input data and the prediction error factors, and cause a computer to execute a process of outputting the distribution of the prediction error factors in the plurality of extracted comparison periods.

Advantages of the Invention

[0009] According to the present disclosure, it becomes possible to analyze and visualize the prediction error factors of the prediction model.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0011] Hereinafter, with reference to the drawings, preferred embodiments of the present disclosure will be described. <First Embodiment> [Overall Configuration] FIG. 1 is a block diagram showing the overall configuration of the model generation system according to the first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is an application of the model analysis device of the present disclosure and is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used for the user to give necessary instructions and inputs when modifying the model or displaying the evaluation information.

[0012] First, the operation of the model generation system 1 will be outlined. The model generation device 100 generates a machine learning model (hereinafter simply referred to as the "model") using pre-prepared training data. Further, the model generation device 100 evaluates the generated model. Specifically, predictions are made using the model with evaluation data and the like, and the factors causing prediction errors in the model are analyzed based on the prediction results. Then, based on the obtained factors causing prediction errors, the model generation device 100 extracts a plurality of comparison periods from the target period of prediction, creates a distribution of the factors causing prediction errors for each comparison period, and displays it on the display device 2 as evaluation information. As a result, the user can view the distribution of the factors causing prediction errors for each period extracted based on the characteristics of the factors causing prediction errors, and can consider countermeasures against prediction errors. Note that the user may operate the input device 3 to specify a comparison period for displaying the distribution of the factors causing prediction errors. Further, the user operates the input device 3 to input correction information for correcting the model.

[0013] Here, the "model" is information representing the relationship between the explanatory variable and the target variable. The model is, for example, a component for estimating the result of the estimation target by calculating the target variable based on the explanatory variable. The model is generated by executing a learning algorithm with the learning data for which the value of the target variable has already been obtained and any parameters as inputs. The model may be represented, for example, by a function c that maps the input x to the correct answer y. The model may estimate a numerical value to be estimated or may estimate a label to be estimated. The model may output a variable that describes the probability distribution of the target variable. The model may also be described as a "learning model", "analysis model", "AI (Artificial Intelligence) model" or "prediction formula".

[0014] [Hardware Configuration] FIG. 2 is a block diagram showing the hardware configuration of the model generation device 100. As shown in the figure, the model generation device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0015] I / F111 performs data input / output with an external device. Specifically, training data, evaluation data, and instructions and inputs entered by the user using the input device 3 for model generation are input into the model generation device 100 through I / F111. Also, the evaluation information of the model generated by the model generation device 100 is output to the display device 2 through I / F111.

[0016] The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generation device 100 by executing a pre-prepared program. Note that the processor 112 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The processor 112 executes the model analysis process described later.

[0017] The memory 113 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory during the execution of various processes by the processor 112.

[0018] The recording medium 114 is a non-volatile and non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 stores various programs executed by the processor 112. When the model generation device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0019] DB115 stores information regarding the model generated by the model generation device 100 (hereinafter referred to as the "existing model") and the model after modification by retraining (hereinafter referred to as the "modified model"). Further, DB115 stores, as necessary, training data, evaluation data, modification information input by the user, evaluation information regarding the cause of prediction errors, etc. input through I / F111.

[0020] (Functional Configuration) Figure 3 is a block diagram showing the functional configuration of the model generation device 100 according to the first embodiment. Functionally, the model generation device 100 includes a training data DB121, a model training unit 122, a model DB123, an evaluation data DB124, a prediction error analysis unit 125, and an evaluation information output unit 126.

[0021] The training data DB121 stores the training data used for generating the model. The training data D1 is input to the model training unit 122. Note that the training data D1 is composed of a plurality of combinations of input data and correct labels (teacher labels) for the input data.

[0022] The model training unit 122 trains the model using the training data D1 and generates the model. The model training unit 122 outputs the model data M corresponding to the generated model to the model DB123 and the prediction error analysis unit 125. Note that the model data M includes a plurality of parameter information constituting the model. The parameter information includes, for example, information on explanatory variables (or features) used as inputs to the model, information on weights for each explanatory variable, information on weights for each sample constituting the input data, etc.

[0023] In addition, the model training unit 122 retrains the existing model to generate a corrected model. In this case, based on the correction information D3 input by the user using the input device 3, the model training unit 122 corrects the parameters constituting the model and retrains the model using training data for retraining as necessary. The model training unit 122 stores the model data M of the corrected model obtained by retraining in the model DB123 and outputs it to the prediction error analysis unit 125.

[0024] The evaluation data DB124 stores evaluation data used for evaluating the generated model. The evaluation data includes various types of data that can be used for evaluating the model. Basically, the evaluation data is composed of a plurality of combinations of input data and correct labels (teacher labels) for the input data. Examples of the evaluation data are as follows. (1) "Data not used for generating the model" called validation data or test data In this case, the evaluation data is basically a set of input data and correct labels. (2) "Data newly collected after generating the model" such as operation data Note that when labeling is not performed immediately, the evaluation data may be only input data. (3) "Data generated by some method and unknown to the model" For example, when the feature amounts in the input data are (day of the week, holiday, weather), pseudo-future data can be created using calendar information and weather forecasts. (4) "Data identical to the training data" The training data used for generating the model can be used as evaluation data. In this case, data identical to the training data may be stored in the evaluation data DB124 as evaluation data.

[0025] The prediction error analysis unit 125 analyzes the prediction errors of the existing model using the evaluation data. Specifically, the prediction error analysis unit 125 inputs the input data of the evaluation data into the existing model to make a prediction and obtains the prediction result. Then, the prediction error analysis unit 125 analyzes the factors of the prediction errors (hereinafter referred to as "prediction error factors") made by the existing model based on the used evaluation data and the prediction result. Specifically, the prediction error analysis unit 125 estimates the degree to which the existing model corresponds to a plurality of predetermined prediction error factors, and outputs it to the evaluation information output unit 126 as the analysis result of the prediction error factors. The method for analyzing the prediction error factors will be described in detail later. The prediction error analysis unit 125 is an example of an acquisition means and a determination means.

[0026] The evaluation information output unit 126 generates evaluation information D2 for evaluating the existing model based on the analysis result of the prediction error factors. The evaluation information D2 includes the relationship between the prediction result (predicted value) by the existing model and the measured value, and the distribution of the prediction error factors in a predetermined comparison period. Then, the evaluation information output unit 126 outputs the generated evaluation information D2 to the display device 2. The evaluation information output unit 126 is an example of an extraction means and a factor output means.

[0027] The display device 2 displays the evaluation information D2 output by the evaluation information output unit 126 on the display device 2. Thereby, the user can refer to the relationship between the predicted value and the measured value by the existing model, and the distribution of the prediction error factors in the comparison period, and evaluate the performance of the existing model. Also, the user inputs the correction information D3 into the input device 3 as needed. The model training unit 122 corrects the model by retraining the model using the input correction information D3.

[0028] [Example of display of evaluation information] Next, an example of the display of the evaluation information displayed on the display device 2 will be described. (First display example) Figure 4 shows a first display example of evaluation information. The first display example 40 includes a graph G and distribution diagrams R1 and R2 of prediction error factors (hereinafter, also simply referred to as "factors"). The graph G is a graph showing the relationship between the predicted values by the existing model and the measured values. In the example of FIG. 4, the existing model is a model for predicting the sales of a product. The horizontal axis of the graph G indicates the number of days from a predetermined reference date, and the vertical axis indicates the sales. The graph G shows the measured values of the sales and the predicted values by the existing model.

[0029] As the evaluation information, first, the graph G is displayed. The user designates a predetermined period in the graph G as a comparison period. The "comparison period" is a period for displaying the distribution of the prediction error factors for the user's comparison. In the example of FIG. 4, the user operates the input device 3 to designate the comparison periods T1 and T2. The designation of the comparison period by the user is sent from the input device 3 to the evaluation information output unit 126. On the other hand, the prediction error analysis unit 125 analyzes the prediction error factors in the comparison period T1 designated by the user and outputs the analysis result to the evaluation information output unit 126. The evaluation information output unit 126 creates a distribution diagram R1 of the prediction error factors based on the analysis result and displays it on the display device 2. The distribution diagram R1 shows the distribution of six prediction error factors A to F. Specifically, the distribution diagram R1 includes a bar graph 51 showing the degree of the prediction error factors A to F and a radar chart 52. The user can compare the magnitudes of the respective prediction error factors by the bar graph 51 and can view the balance of a plurality of prediction error factors by the radar chart 52.

[0030] Similarly, the prediction error analysis unit 125 analyzes the prediction error factors in the comparison period T2 designated by the user and outputs the analysis result to the evaluation information output unit 126. The evaluation information output unit 126 creates a distribution diagram R2 of the prediction error factors based on the analysis result and displays it on the display device 2. In the first display example, the user can display the distribution diagrams R1 and R2 of the arbitrarily designated comparison periods T1 and T2 side by side.

[0031] (Second display example) Figure 5 shows a second display example of the evaluation information. The second display example 41 includes a graph G and distribution diagrams R3 and R4 of the prediction error factors. Similar to the first display example, the graph G is a graph showing the relationship between the predicted values by the existing model and the measured values.

[0032] In the first display example, the user designates the comparison period. In contrast, in the second display example, the evaluation information output unit 126 detects the change points of the distribution of the prediction error factors, sets and displays the comparison period with the change points as the boundary. Specifically, the evaluation information output unit 126 detects the points where changes occur in the distributions of factors A to F as the change points. In the example of Figure 5, the evaluation information output unit 126 detects the points where the distributions of the prediction error factors A to F change as the change point P1, and sets and displays the comparison periods T3 and T4. Specifically, the evaluation information output unit 126 sets the period in which the distributions of the prediction error factors are common before the change point P1 as the comparison period T3, and the period in which the distributions of the prediction error factors are common after the change point P1 as the comparison period T4.

[0033] Furthermore, the evaluation information output unit 126 creates distribution diagrams showing the distribution of the prediction error factors for each set comparison period. In the example of Figure 5, the evaluation information output unit 126 creates the distribution diagram R3 of the prediction error factors for the comparison period T3 and the distribution diagram R4 of the prediction error factors for the comparison period T4. Note that the fact that the distribution diagrams R3 and R4 include the bar graph 51 and the radar chart 52 is the same as in the first display example. Then, the evaluation information output unit 126 displays the evaluation information including the graph G including the comparison periods T3 and T4 and the distribution diagrams R3 and R4 on the display device 2.

[0034] In the second display example, the user can view the distribution of the prediction error factors for the comparison periods before and after the change with the change points of the distribution of the prediction error factors as the boundary.

[0035] (The third display example) Figure 6 shows a third display example of the evaluation information. The third display example 42 includes a graph G and distribution diagrams R5 to R7 of the prediction error factors. Similar to the first display example, the graph G is a graph showing the relationship between the predicted values by the existing model and the measured values.

[0036] In the third display example, the evaluation information output unit 126 detects the change points of the main prediction error factors, sets and displays a comparison period with the change points as boundaries. Specifically, the evaluation information output unit 126 detects the point at which the largest factor among factors A to F changes as the change point. In the example of FIG. 6, the evaluation information output unit 126 detects the point at which the largest factor among the prediction error factors A to F changes from factor A to factor C as the change point P2, and detects the point at which the largest factor changes from factor C to factor E as the change point P3. Then, the evaluation information output unit 126 sets the comparison period T5 before the change point P2, sets the period between the change points P2 and P3 as the comparison period T6, and sets the comparison period T7 after the change point P. Note that, instead of the point at which the largest factor changes, a point at which a combination of a plurality of factors (for example, 80% or more) among all factors changes may be used as the change point.

[0037] Furthermore, the evaluation information output unit 126 creates a distribution diagram showing the distribution of the prediction error factors for each set comparison period. In the example of FIG. 6, the evaluation information output unit 126 creates distribution diagrams R5 to R7 of the prediction error factors for the comparison periods T5 to T7. That is, the distribution diagram R5 corresponds to the comparison period T5 with many factor A, the distribution diagram R6 corresponds to the comparison period T6 with many factor C, and the distribution diagram R7 corresponds to the comparison period T7 with many factor E. Note that the fact that the distribution diagrams R5 to R7 include the bar graph 51 and the radar chart 52 is the same as in the first display example. Then, the evaluation information output unit 126 displays the graph G including the comparison periods T5 to T7 and the evaluation information including the distribution diagrams R5 to 7 on the display device 2.

[0038] In the third display example, the user can view the distribution of the prediction error factors for the comparison periods before and after the change with the point at which the main prediction error factor changes as the boundary.

[0039] (Fourth display example) FIG. 7 shows a fourth display example of the evaluation information. The fourth display example 43 includes the graph G and the distribution diagrams R8 and R9 of the prediction error factors. Similar to the first display example, the graph G is a graph showing the relationship between the predicted value by the existing model and the measured value.

[0040] In the fourth display example, the evaluation information output unit 126 detects a change point in the input data input to the existing model, specifically, a change point in the distribution of the explanatory variables, and sets a comparison period with the change point as the boundary. Specifically, the evaluation information output unit 126 analyzes the distributions of a plurality of explanatory variables included in the input data, and detects the point at which the distribution of the explanatory variables has changed as the change point. In the example of FIG. 7, the evaluation information output unit 126 detects that the distributions of a plurality of explanatory variables included in the input data have changed at the change point P4, and sets comparison periods T8 and T9 before and after the change point P4. For example, assume that the existing model predicts sales with the gender of customers as an explanatory variable. In this case, the evaluation information output unit 126 analyzes the distribution of the gender of customers, that is, the ratio of men and women, and when, for example, there were more female customers up to a certain point, but more male customers after that point, it detects that point as the change point.

[0041] Furthermore, the evaluation information output unit 126 creates a distribution diagram showing the distribution of prediction error factors for each set comparison period. In the example of FIG. 7, the evaluation information output unit 126 creates distribution diagrams R8 and R9 of prediction error factors for the comparison periods T8 and T9. Note that the fact that the distribution diagrams R8 and R9 include the bar graph 51 and the radar chart 52 is the same as in the first display example. Then, the evaluation information output unit 126 displays evaluation information including the graph G including the comparison periods T8 and T9 and the distribution diagrams R8 and R9 on the display device 2.

[0042] In the fourth display example, the user can view the distribution of prediction error factors for the comparison periods before and after the change, with the boundary being the point in time when the distribution of the explanatory variables, i.e., the trend of the input data, changes. In the above example, the evaluation information output unit 126 sets the comparison period using the change point of the distribution of the explanatory variables in the input data. Instead, the comparison period may be set using the change point of the target variable. For example, the evaluation information output unit 126 may set the comparison period before and after the day when the actual sales value suddenly doubles as the change point. Also, the evaluation information output unit 126 may detect the change point based on both the explanatory variable and the target variable. In the above example, for instance, the day when the male-female ratio of customers as the explanatory variable changes and the sales amount changes by a predetermined value or more may be detected as the change point.

[0043] Note that in the above first to fourth display examples, the graph G showing the relationship between the predicted value and the actual value by the existing model, and the distribution diagrams R1 to R9 including bar graphs and radar charts are all examples, and evaluation information may be illustrated using various other graphs and charts.

[0044] [Model Analysis Process] Next, the model analysis process by the model generation device 100 will be described. FIG. 8 is a flowchart of the model analysis process by the model generation device 100. The model analysis process is a process of analyzing the prediction error factors of the existing model generated by the model training unit 122 and displaying them on the display device 2. This process is realized by the processor 112 shown in FIG. 2 executing a program prepared in advance and operating as an element shown in FIG. 3.

[0045] First, the prediction error analysis unit 125 inputs the evaluation data into the existing model and obtains the predicted value by the existing model (step S10). Next, the prediction error analysis unit 125 analyzes the prediction error factors of the existing model using the actual value included in the evaluation data and the predicted value by the existing model (step S11). The prediction error analysis unit 125 outputs the analysis result of the prediction error factors to the evaluation information output unit 126.

[0046] Next, the evaluation information output unit 126 sets a comparison period (step S13). Specifically, in the case of the above-described first display example, the evaluation information output unit 126 sets the comparison period according to the user's input. On the other hand, in the case of the second to fourth display examples, the evaluation information output unit 126 sets the comparison period based on a change point in the distribution of prediction error factors, a change point in the maximum prediction error factor, or a change point in the distribution of input data.

[0047] Next, for each set comparison period, the evaluation information output unit 126 creates a distribution diagram of prediction error factors (step S13). As a result, distribution diagrams R1 to R8 exemplified in FIGS. 4 to 7 are created. Then, the evaluation information output unit 126 generates evaluation information including a graph showing the relationship between the measured value and the predicted value of the existing model and the distribution diagram of prediction error factors for each comparison period (step S14), and outputs it to the display device 2 (step S15). In this way, as exemplified in FIGS. 4 to 7, the evaluation information is displayed on the display device 2. Then, the process ends.

[0048] [Method for Analyzing Prediction Error Factors] Next, the method for analyzing prediction error factors will be described in detail. FIG. 9 shows the functional configuration of the prediction error analysis unit 125. As shown in the figure, the prediction error analysis unit 125 includes an index evaluation unit 131 and a factor identification unit 132.

[0049] Generally speaking, the index evaluation unit 131 calculates multiple types of indexes for the prediction model, the data of the explanatory variables used in the prediction model, or the data of the objective variable used in the prediction model. Next, the index evaluation unit 131 evaluates each of the calculated multiple types of indexes. Then, the factor identification unit 132 identifies the factors of the prediction error by the prediction model according to the combination of the evaluation results of each of the multiple types of indexes by the index evaluation unit 131. The factor identification unit 132 identifies the factors using, for example, a predetermined rule that associates the combination of evaluation results with the factors.

[0050] Specifically, the index evaluation unit 131 calculates a plurality of indices necessary for the analysis of prediction error factors and determines the calculation results of the indices. For example, the index evaluation unit 131 calculates the abnormality degree of the explanatory variables of the prediction error samples for the training data or the evaluation data, and evaluates the calculated abnormality degree. In this case, the index evaluation unit 131 evaluates the index by determining whether the value of the calculated abnormality degree is a value that is recognized as a sample with an abnormal prediction error sample. That is, in this case, the index evaluation unit 131 determines whether the prediction error sample is an abnormal sample using the calculated abnormality degree. As another example, the index evaluation unit 131 calculates the distance between distributions (hereinafter, also referred to as the "data distribution change amount") between the training data and the operation data, and evaluates the calculated distance between distributions. In this case, the index evaluation unit 131 evaluates the index by determining whether the value of the calculated distance between distributions is a value that is recognized as a change in the data distribution between the training time and the operation time. That is, in this case, the index evaluation unit 131 determines whether a change in the data distribution has occurred between the training time and the operation time using the calculated distance between distributions. Note that these are only examples, and the index evaluation unit 131 can calculate and evaluate various types of indices. In this way, the index evaluation unit 131 makes a predetermined determination on the index as an evaluation of the index. The determination for each index is made, for example, using a threshold value that is determined in advance and stored. Note that instead of the threshold value itself, a parameter for specifying the threshold value may be stored.

[0051] The factor identification unit 132 identifies the prediction error factor according to the combination of the evaluation results of the plurality of types of indices by the index evaluation unit 131. The factor identification unit 132 identifies the prediction error factor according to the combination of the determination results of the predetermined determination for each index. Specifically, the factor identification unit 132 identifies the prediction error factor by using a predetermined rule (hereinafter, referred to as the "factor determination rule") that associates the prediction error factor with the combination of the plurality of determination results. Note that the content of the factor determination rule used by the factor identification unit 132 is arbitrary. The factor determination rule is stored in a storage unit or the like in advance.

[0052] FIG. 10 shows an example of a tabular factor determination rule used by the factor identification unit 132. In this example, the index evaluation unit 131 generates Yes or No determination results for three questions Q1, Q2, and Q3 corresponding to three different indices. In question Q1, it is determined whether the prediction error sample 25 is a normal sample based on the abnormality degree of the explanatory variables of the prediction error sample with respect to the training data. In question Q2, by calculating evaluation indices such as the mean squared error using the neighboring training samples and the prediction model, it is determined how well the existing model fits in the neighboring region of the training data. Here, the neighboring training samples refer to the samples in the training data located within the neighboring region. Also, the neighboring region refers to the range of the values of the explanatory variables that are judged to be close to the values of the explanatory variables of the prediction error sample. At this time, the specific method of defining the neighboring region is arbitrary. For example, a region where the distance (such as Euclidean distance) from the prediction error sample calculated using the values of the explanatory variables is equal to or less than a predetermined distance may be used as the neighboring region. In question Q3, it is determined whether the data distribution has changed between training and operation using the data distribution change amount between the distribution of the explanatory variables of the training data and the distribution of the explanatory variables of the operation data.

[0053] The factor identification unit 132 identifies the prediction error factors using the determination results by the index evaluation unit 131 and the factor determination rule in FIG. 10. There are eight combinations of the three types of determination results, and in the tabular factor determination rule, a prediction error factor is assigned to each of these eight types. In the case of FIG. 10, the eight combinations are assigned to four prediction error factors. In the example of FIG. 10, as the prediction error factors, "errors other than the prediction model and data", "local errors", "changes in data distribution", and "abnormality of explanatory variables" are obtained.

[0054] The above method for analyzing the prediction error factors is described in International Application PCT / JP2021 / 007191, and the entire description thereof is incorporated herein by reference. Note that the method for analyzing the prediction error factors in the present embodiment is not limited to the above, and other methods can also be adopted.

[0055] [Modification Example] (Modification Example 1) In the above-described embodiment, the evaluation information output unit 126 outputs evaluation information including a graph showing the relationship between the measured value and the predicted value of the existing model, and a distribution diagram of the prediction error factors for each comparison period. In addition to this, the evaluation information output unit 126 may output countermeasures for the prediction error factors.

[0056] FIG. 11 is a diagram conceptually showing the method of Modification 1. As described above, when the prediction error factors are presented for each comparison period, the user will consider countermeasures for the prediction error factors. For example, when "bias in labels in the dataset" is determined to be a prediction error factor, and the user performs "undersampling" as a countermeasure and the prediction error is improved. In this case, the prediction error factor "bias in labels in the dataset" and its countermeasure "undersampling" are associated and accumulated as history data. By collecting the countermeasures taken by a large number of users for various prediction error factors, information on effective countermeasures for various prediction error factors can be obtained. Note that, as the information on the countermeasures, the correction information input by the user using the input device 3 may be used.

[0057] Then, using the collected history data, a countermeasure prediction model is created that predicts effective countermeasures for the prediction error factors. Note that the countermeasure prediction model may initially be rule-based, or may be created as a learning model when a certain number of data are accumulated.

[0058] Then, the evaluation information output unit 126 predicts countermeasures for the prediction error factors included in the evaluation information using the countermeasure prediction model, and outputs evaluation information including recommended countermeasures. Thereby, in addition to the prediction error factors for each comparison period, the user can receive the presentation of recommended countermeasures for the prediction error factors.

[0059] (Modification 2) In the above embodiment, the model generation device 100 is configured as an independent device such as a PC. Instead, the model generation device may be configured by a server and a terminal device. FIG. 12 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 12, the server 100x has the configuration of the model generation device 100 shown in FIG. 3. Also, the display device 2x and the input device 3x of the terminal device 7 used by the user are used as the display device 2 and the input device 3 shown in FIG. 3. With this configuration, it becomes possible to easily collect and share countermeasures and the like input by a plurality of users in the server 100x.

[0060] <Second Embodiment> FIG. 13 is a block diagram showing a functional configuration of the model analysis device according to the second embodiment. The model analysis device 70 includes an acquisition unit 71, a determination unit 72, an extraction unit 73, and a factor output unit 74.

[0061] FIG. 14 is a flowchart of the processing by the model analysis device according to the second embodiment. First, the acquisition unit 71 acquires the prediction result of the model for the input data (step S71). The determination unit 72 determines the cause of the prediction error of the model using the input data and the prediction result (step S72). The extraction unit 73 extracts a plurality of comparison periods from the target period of the prediction by the model based on at least one of the input data and the cause of the prediction error (step S73). The factor output unit 74 outputs the distribution of the cause of the prediction error in the plurality of extracted comparison periods (step S74).

[0062] According to the model analysis device 70 of the second embodiment, it is possible to analyze the cause of the prediction error of the prediction model and visualize it for each of a plurality of periods.

[0063] Some or all of the above embodiments may be described as follows in the following supplementary notes, but are not limited thereto.

[0064] (Supplementary Note 1) An acquisition unit that acquires the prediction result of the model for the input data, Determination means for determining the factors causing prediction errors of the model using the input data and the prediction results; Extraction means for extracting a plurality of comparison periods from the target period of prediction by the model based on at least one of the input data and the factors causing prediction errors; Factor output means for outputting the distribution of the factors causing prediction errors in the plurality of extracted comparison periods; A model analysis device comprising:

[0065] (Appendix 2) The model analysis device according to Appendix 1, wherein the extraction means extracts the comparison period based on the distribution of a plurality of factors causing prediction errors.

[0066] (Appendix 3) The model analysis device according to Appendix 2, wherein the extraction means extracts the comparison period based on the change points of the distribution of the plurality of factors causing prediction errors.

[0067] (Appendix 4) The model analysis device according to Appendix 2, wherein the extraction means extracts the comparison period based on the change points of the main factors causing prediction errors among the plurality of factors causing prediction errors.

[0068] (Appendix 5) The model analysis device according to Appendix 1, wherein the extraction means extracts the comparison period based on the change points of the distribution of the input data.

[0069] (Appendix 6) The model analysis device according to any one of Appendices 1 to 5, wherein the factor output means outputs countermeasures against the factors causing prediction errors.

[0070] (Appendix 7) The model analysis device according to Appendix 6, wherein the factor output means outputs the countermeasures using a countermeasure prediction model that has learned the relationship between a plurality of factors causing prediction errors and the countermeasures against each factor causing prediction errors.

[0071] (Appendix 8) The model analysis device according to Supplementary Note 6, comprising storage means for associating and storing a plurality of prediction error factors and countermeasures for each prediction error factor.

[0072] (Supplementary Note 9) Obtain the prediction result of the model for the input data, Using the input data and the prediction result, determine the prediction error factors of the model, Based on at least one of the input data and the prediction error factors, extract a plurality of comparison periods from the target period of the prediction by the model, A model analysis method for outputting the distribution of the prediction error factors in the extracted plurality of comparison periods.

[0073] (Supplementary Note 10) Obtain the prediction result of the model for the input data, Using the input data and the prediction result, determine the prediction error factors of the model, Based on at least one of the input data and the prediction error factors, extract a plurality of comparison periods from the target period of the prediction by the model, A recording medium recording a program for causing a computer to execute a process of outputting the distribution of the prediction error factors in the extracted plurality of comparison periods.

[0074] Although the present disclosure has been described with reference to the embodiments and examples above, the present disclosure is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

Explanation of Signs

[0075] 1, 1x Model generation system 2, 2x Display device 3, 3x Input device 7 Terminal device 100 Model generation device 112 Processor 121 Training data DB 122 Model training unit 123 Model DB 124 Evaluation Data DB 125 Prediction Error Analysis Unit 126 Evaluation Information Output Unit

Claims

1. An acquisition means for acquiring a prediction result of a model for input data; A determination means for determining a prediction error factor of the model using the input data and the prediction result; An extraction means for extracting a plurality of comparison periods from a target period of prediction by the model based on at least one of the input data and the prediction error factor; A factor output means for outputting a distribution of the prediction error factor in the plurality of extracted comparison periods; A model analysis device comprising:

2. The model analysis device according to claim 1, wherein the extraction means extracts the comparison period based on a distribution of a plurality of prediction error factors.

3. The model analysis device according to claim 2, wherein the extraction means extracts the comparison period based on a change point of the distribution of the plurality of prediction error factors.

4. The model analysis device according to claim 2, wherein the extraction means extracts the comparison period based on a change point of a main prediction error factor among the plurality of prediction error factors.

5. The model analysis device according to claim 1, wherein the extraction means extracts the comparison period based on a change point of the distribution of the input data.

6. The model analysis device according to any one of claims 1 to 5, wherein the factor output means outputs a countermeasure against the prediction error factor.

7. The model analysis device according to claim 6, wherein the factor output means outputs the countermeasure using a countermeasure prediction model that has learned a relationship between a plurality of prediction error factors and countermeasures for each prediction error factor.

8. The model analysis device according to claim 6, further comprising a storage means for associating and storing a plurality of prediction error factors and countermeasures for each prediction error factor.

9. A model analysis method executed by a computer, comprising: acquiring a prediction result of a model for input data; determining a prediction error factor of the model using the input data and the prediction result; extracting a plurality of comparison periods from a target period of prediction by the model based on at least one of the input data and the prediction error factor; outputting a distribution of the prediction error factor in the plurality of extracted comparison periods.

10. acquiring a prediction result of a model for input data; determining a prediction error factor of the model using the input data and the prediction result; extracting a plurality of comparison periods from a target period of prediction by the model based on at least one of the input data and the prediction error factor; A program that causes a computer to execute a process of outputting the distribution of the prediction error factors in a plurality of extracted comparison periods.

Images