Model processing device, model processing method, and program

Abstract

Provided is a model processing device in which an existing model acquisition means acquires an existing model. A first output means outputs first evaluation information indicating a relationship between data for evaluation and output of the existing model for the data for evaluation. A correction acquisition means acquires correction information inputted for the first evaluation information. A corrected model acquisition means acquires a corrected model that has been corrected on the basis of the correction information inputted for the first evaluation information. A second output means outputs second evaluation information indicating a relationship in output between the existing model and the corrected model for the data for evaluation.

Description

[Technical Field]

[0001] The present disclosure relates to evaluating and correcting machine learning models. [Background technology]

[0002] In recent years, predictive models obtained by machine learning have been used in various fields. When the accuracy of the created predictive model is insufficient, or when time has passed since the initial model was created and the trends of the data used have changed, it becomes necessary to retrain the predictive model. Patent Document 1 describes a method for adjusting the model learning process based on evaluation information for the created model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2021 / 161896 Summary of the Invention [Problem to be solved by the invention]

[0004] When determining whether a model currently in operation needs to be retrained or what corrections to make, it is desirable to have sufficient information about the data used to create the model and the output of the model currently in operation.

[0005] One object of the present disclosure is to provide a model processing device that enables appropriate modification of a model by presenting sufficient information regarding the data used to create the model and the output of the model in operation. [Means for solving the problem]

[0006] In one aspect of the present disclosure, the model processing device includes: an existing model acquisition means for acquiring an existing model; a first output means for outputting first evaluation information indicating a relationship between evaluation data and an output of the existing model with respect to the evaluation data; a correction acquisition means for acquiring correction information input for the first evaluation information; a corrected model acquisition means for acquiring a corrected model that has been corrected based on correction information input for the first evaluation information; a second output means for outputting second evaluation information indicating a relationship between the outputs of the existing model and the corrected model with respect to the evaluation data; Equipped with 、 The correction information includes designation of problem locations and problem levels in the output of the existing model. .

[0007] In another aspect of the present disclosure, a computer-implemented method for model processing comprises: Get an existing model, outputting first evaluation information indicating a relationship between the evaluation data and an output of the existing model for the evaluation data; Acquire correction information input for the first evaluation information; obtaining a corrected model corrected based on correction information input for the first evaluation information; Outputting second evaluation information indicating the relationship between the outputs of the existing model and the corrected model with respect to the evaluation data. death, The correction information includes designation of problem locations and problem levels in the output of the existing model. .

[0008] In yet another aspect of the disclosure, a program includes: Get an existing model, outputting first evaluation information indicating a relationship between the evaluation data and an output of the existing model for the evaluation data; Acquire correction information input for the first evaluation information; obtaining a corrected model corrected based on correction information input for the first evaluation information; and causing a computer to execute a process of outputting second evaluation information indicating a relationship between the outputs of the existing model and the corrected model with respect to the evaluation data. 、 The correction information includes designation of problem locations and problem levels in the output of the existing model. . [Effects of the Invention]

[0009] According to the present disclosure, by providing sufficient information regarding the data used to create the model and the output of the model during operation, it is possible to appropriately modify the model. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the model generating device. [Figure 3] FIG. 1 is a block diagram showing a functional configuration of a model generation device according to a first embodiment. [Figure 4] 10 shows an example of displaying evaluation information in the case of prediction of one-dimensional time series data. [Figure 5] 10 shows an example of displaying evaluation information in the case of predicting multidimensional time series data. [Figure 6] 10 shows an example of a modification UI for inputting parameter modification information. [Figure 7] An example of a fix UI for entering issue fix information is shown below. [Figure 8] 10 shows an example of a model visualization UI and a correction UI for regression analysis. [Figure 9] 10 shows an example of a model comparison UI and a correction UI for regression analysis. [Figure 10] The method for inputting correction information is shown below. [Figure 11] 10 shows an example of a model visualization UI and a correction UI for a classification model. [Figure 12] 10 shows an example of a model comparison UI and a correction UI for a classification model. [Figure 13] An example of the UI for modifying explanatory variables is shown below. [Figure 14] 10 is a flowchart of a model correction process performed by the model generating device. [Figure 15] FIG. 1 is a block diagram showing a schematic configuration of a model generation system using a server and a terminal device. [Figure 16] FIG. 10 is a block diagram showing the functional configuration of a model processing device according to a second embodiment. [Figure 17] 10 is a flowchart of processing by a model processing device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Concept explanation] First, the basic concept of this embodiment will be described. In this embodiment, an interactive UI (User Interface) is used to present evaluation information regarding a prediction model (hereinafter also simply referred to as a "model") to a user, and model modification information is obtained from the user, and necessary model modifications are performed. Note that a "user" refers to a person who modifies the model, such as a model developer or operator. Furthermore, the tasks of the prediction model include various tasks such as regression of time series data, regression of data other than time series data, discrimination, and classification, and are not limited to a specific task.

[0012] Here, a "model" is information that represents the relationship between explanatory variables and a response variable. A model is, for example, a component that estimates the result of an estimation target by calculating a response variable based on explanatory variables. A model is generated by executing a learning algorithm using training data, for which the value of the response variable has already been obtained, and arbitrary parameters as input. A model may be represented, for example, by a function c that maps an input x to a correct answer y. A model may estimate a numerical value of an estimation target, or may estimate a label of an estimation target. A model may output a variable that describes the probability distribution of a response variable. A model may also be referred to as a "learning model," an "analysis model," an "AI (Artificial Intelligence) model," or a "prediction formula," etc.

[0013] Specifically, the interactive UI includes a model visualization UI, a correction UI, and a model comparison UI. The "model visualization UI" is a UI for presenting to a user information indicating the relationship between training data used to generate a model and the output of the generated model. The "correction UI" is a UI for a user to input information for correcting a model (hereinafter referred to as "correction information"). The "model comparison UI" is a UI for presenting to a user information indicating the relationship between a model before being corrected based on the correction information (also referred to as an "existing model") and a model after being corrected (also referred to as a "corrected model"). The correction UI can be used in combination with the model visualization UI or the model comparison UI. In this way, in this embodiment, by using an interactive UI, it is possible to correct a model by appropriately reflecting the user's domain knowledge.

[0014] [Overall configuration] FIG. 1 is a block diagram showing the overall configuration of a model generation system according to a first embodiment. The model generation system 1 includes a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by the user to give instructions and input necessary when modifying a model.

[0015] First, the operation of the model generation system 1 will be outlined. The model generation device 100 generates a machine learning model (hereinafter simply referred to as a "model") using training data prepared in advance. The model generation device 100 also displays evaluation information indicating the relationship between the training data used to generate the model and the output of the generated model on the display device 2. This evaluation information is provided to the user using the model visualization UI described above. This allows the user to evaluate the performance of the generated model in relation to the training data. The generated model is operated in a planned environment.

[0016] If the performance of the existing model in operation becomes insufficient due to a change in the trend of the data to be processed during model operation, the user inputs model modification information using the input device 3. This input is performed using the modification UI described above. The model generation device 100 modifies the existing model based on the input modification information and generates a modified model. The model generation device 100 also displays evaluation information indicating the relationship between the output of the existing model before modification and the output of the modified model on the display device 2. This evaluation information is presented to the user using the model comparison UI described above. This allows the user to evaluate the performance of the modified model in relation to the existing model before modification. In this way, the user can appropriately modify the model by referring to the evaluation information indicating the relationship between the training data and the created model or the relationship between the models before and after modification.

[0017] The model visualization UI and model comparison UI may also present the relationship between the created model and data other than the training data, i.e., data that the model has not learned. Data that the model has not learned includes, for example, validation data (test data) and operational data. Displaying the prediction results of the model for data that the model has not learned is important when examining model issues or comparing models.

[0018] [Hardware configuration] 2 is a block diagram showing the hardware configuration of model generation device 100. As shown in the figure, model generation device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0019] The I / F 111 inputs and outputs data to and from an external device. Specifically, training data used to generate a model and modification information input by a user using the input device 3 are input to the model generation device 100 via the I / F 111. In addition, evaluation information regarding the relationship between the training data and the output of an existing model, evaluation information regarding the relationship between the output of the existing model and the output of a modified model, and the like are output to the display device 2 via the I / F 111.

[0020] The processor 112 is a computer such as a CPU (Central Processing Unit), and controls the entire model generating device 100 by executing a prepared program. The processor 112 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The processor 112 executes a model correction process, which will be described later.

[0021] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0022] The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 records various programs to be executed by the processor 112. When the model generation device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0023] DB 115 stores information about the existing model and the modified model generated by model generation device 100. DB 115 also stores, as necessary, training data input via I / F 111, modification information input by the user, evaluation information regarding the relationship between the training data and the output of the existing model, evaluation information regarding the relationship between the output of the existing model and the output of the modified model, and the like.

[0024] (Functional configuration) 3 is a block diagram showing the functional configuration of the model generation device 100 according to the first embodiment. Functionally, the model generation device 100 includes a training data DB 121, a model training unit 122, a model DB 123, an evaluation data DB 124, and an evaluation information output unit 125.

[0025] The training data DB 121 stores training data used to generate a model. The training data D1 is input to the model training unit 122. The training data D1 is composed of multiple combinations of input data and correct labels (teacher labels) for the input data.

[0026] The model training unit 122 trains the model using training data and generates a model. The model training unit 122 outputs model data M corresponding to the generated model to the model DB 123 and the evaluation information output unit 125. The model data M includes information on a plurality of parameters that constitute the model. The parameter information includes, for example, information on explanatory variables used as input to the model, information on weights for each explanatory variable, and information on weights for each sample that constitutes the input data.

[0027] Furthermore, the model training unit 122 retrains the existing model to generate a modified model. In this case, the model training unit 122 modifies the parameters constituting the model based on modification information D3 input by the user using the input device 3, and retrains the model using training data for retraining as necessary. The model training unit 122 stores model data M of the modified model obtained by the retraining in the model DB 123 and outputs it to the evaluation information output unit 125. The model training unit 122 is an example of a modification acquisition means.

[0028] The evaluation data DB 124 stores evaluation data used to evaluate the generated model. The evaluation data includes various types of data that can be used to evaluate the model. Examples of evaluation data include the following: (1) Validation data or test data, which are data that were not used to generate the model. In this case, the evaluation data is basically a set of input data and correct labels. (2) "Data newly collected after the model is generated," such as operational data If labeling is not performed immediately, the evaluation data may be input data only. (3) "Data that is generated in some way and is unknown to the model" For example, if the features in the input data are (day of the week, public holidays, weather), pseudo-future data can be created using calendar information and weather forecasts. (4) Training data The training data may be used as evaluation data to evaluate the model.

[0029] The evaluation information output unit 125 generates evaluation information D2 for evaluating each model based on the model data M, and outputs it to the display device 2. The evaluation information D2 includes information on the relationship between the evaluation data and the output of the generated model, and information on the relationship between the output of the existing model and the output of the modified model for the evaluation data. The evaluation information output unit 125 outputs data for displaying the evaluation information using the model visualization UI or model comparison UI described above to the display device 2. The evaluation information output unit 125 is an example of an existing model acquisition means, a modified model acquisition means, a first output means, and a second output means.

[0030] The display device 2 displays the evaluation information D2 output by the evaluation information output unit 125 on the display device 2. As a result, the display device 2 presents the evaluation information to the user through the model visualization UI or the model comparison UI. In addition, the input device 3 outputs the correction information D3 input by the user through the above-mentioned correction UI to the model training unit 122.

[0031] In this way, the user evaluates the performance of the existing model and the modified model by referring to the evaluation information D2 displayed on the display device 2 by the model visualization UI and the model comparison UI. Then, the user inputs modification information D3 into the input device 3 by the modification UI as needed. The model training unit 122 modifies the model by retraining the model using the input modification information D3. In this way, the user can modify the model at an appropriate time and with an appropriate policy.

[0032] [Interactive UI example] Next, examples of the aforementioned interactive UIs will be described. As mentioned above, the interactive UIs include a model visualization UI, a modification UI, and a model comparison UI.

[0033] (Time series data forecasting) First, a case where the task of the model is to predict time-series data will be described. (1) Prediction of one-dimensional time series data Figure 4 shows an example of displaying evaluation information when the model's task is to predict one-dimensional time-series data. Specifically, display example G1 shows training data and model output for a model that predicts daily sales of a certain product. In display example G1, the horizontal axis represents date and the vertical axis represents sales. Graph 11 shows the training data used to generate the model. Graph 12 shows the output of an existing model generated using the training data shown in graph 11 and future input data. Note that since graph 12 is a predicted value by the model, correct label data corresponding to future input data is not necessarily required. Display example G1 is an example of display using a model visualization UI that displays evaluation information showing the relationship between the training data, future input data, and the output of the existing model.

[0034] When operating a general regression model like a time series model, to output a future prediction, input data for a future time point is prepared and input into the model to output a prediction for that future time point. Similarly, in the case of non-time series regression or discrimination, to output a predicted value of a future dependent variable that is not in the training data, input data for a future time point is prepared and input into the model to output a predicted value for that future time point. As an exception, in a time series model without explanatory variables, future predicted values ​​are obtained by extending the training data, so future input data is not required. Note that a time series model without explanatory variables is a model in which the latest predicted value is regressed from past values ​​of the dependent variable, i.e., an autoregressive model.

[0035] If the user determines from viewing the display example G1 that there is a problem with the existing model, the user inputs correction information. Specifically, the user inputs correction information to the input device 3 using the correction UI. In this embodiment, the correction information includes parameter correction information related to the training data and problem correction information related to the output of the existing model. The parameter correction information is information that specifies the location and amount of adjustment of the weighting that the model assigns to the training data. In other words, the user can specify a specific period of training data and adjust the weighting for the training data for that period.

[0036] In the example of Figure 4, the user specifies the range (period) indicated by line segment 21 of the training data shown in graph 11, and inputs correction information indicating that the weighting for the training data belonging to that period will be increased by 1.5 times. Similarly, the user can specify a specific period of training data and input an input to reduce the weighting for the training data for that period. In the example of Figure 4, the user specifies the range (period) indicated by line segment 22 of the training data shown in graph 11, and inputs correction information to ignore the training data belonging to that period, i.e., to set the weighting for the training data belonging to that period to "0."

[0037] On the other hand, the problem correction information is information that specifies the amount of adjustment for problem areas and the degree of problem (hereinafter also referred to as "problem level") in the model output. The problem level indicates the extent of the problem in the model output. The problem level can be specified using multiple levels (for example, 10 levels). The user can specify a specific period of the model output as a problem area and input the problem level of that problem area. In the example of Figure 4, the user specifies the range (period) indicated by lines 23 and 24 in the model output shown in graph 12, and inputs correction information that sets the problem level of the model output for that period to "4" and "5", respectively.

[0038] When the modification information is input, the model training unit 122 retrains the existing model based on the modification information to generate a modified model. The evaluation information output unit 125 generates evaluation information for the modified model and transmits it to the display device 2. FIG. 4 shows a display example G2 of the evaluation information for the modified model. In display example G2, a graph 11 of the training data is displayed, similar to display example G1. In addition, in display example G2, a graph 12 showing the output of the existing model before modification is displayed by a dashed line, and a graph 13 showing the output of the modified model is further displayed superimposed thereon. This allows the user to easily compare the output of the model before and after modification.

[0039] Furthermore, if the user determines that further corrections are necessary after viewing display example G2, they can input correction information as in display example G1 and further correct the corrected model. As described above, display examples G1 and G2 are display examples that simultaneously use a model visualization UI and a model comparison UI. Note that in the example of Figure 4, the model comparison UI compares two models before and after correction, but multiple models generated during the trial and error process of model correction may also be visualized simultaneously. In this case, graph G2 in Figure 4 simultaneously displays the outputs of multiple existing models, such as graph 12.

[0040] (2) Forecasting multidimensional time series data Next, we will explain the case where the model task is to predict multidimensional time series data. Predicting multidimensional time series data means that the model task first predicts multiple time series data and then predicts the final target variable using those prediction results. In this case, too, the method for predicting one-dimensional time series data described above can basically be applied to each dimension.

[0041] FIG. 5 shows an example of a display of evaluation information for predicting multidimensional time series data. In this example, the model first predicts the two-dimensional explanatory variables "humidity" and "temperature," and then predicts the one-dimensional objective variable "sales" from these prediction results. Specifically, display example G3 is an example of a display of evaluation information related to a task of predicting daily humidity. Display example G3 simultaneously displays a graph 14 showing training data, a graph 15 showing the output of the existing model, and a graph 16 showing the output of the modified model. In display example G3, the user inputs correction information using line segment 21 to correct the weighting for the training data.

[0042] Display example G4 is an example of displaying evaluation information related to a task of predicting daily temperatures. Display example G4 simultaneously displays graph 17 showing training data, graph 18 showing the output of the existing model, and graph 19 showing the output of the modified model. In display example G4, the user has entered modification information using line segment 22 to modify the weights for the training data.

[0043] Display example G5 is a display example of evaluation information related to a task of predicting daily sales from predicted humidity and temperature. Similar to FIG. 4, display example G5 simultaneously displays a graph 11 showing training data, a graph 12 showing the output of the existing model, and a graph 13 showing the output of the corrected model. Similarly to FIG. 4, display example G5 also allows the user to input correction information.

[0044] (3) How to enter correction information Next, we will explain in detail how to input correction information using the correction UI. As mentioned above, the correction information includes parameter correction information related to the training data and problem correction information related to the output of the existing model. The parameter correction information is information that specifies the location and amount of adjustment of the weights that the model assigns to the training data.

[0045] FIG. 6 shows an example of a correction UI for inputting parameter correction information related to training data. The user specifies a specific period of training data and inputs the weight adjustment for the training data for that period. FIG. 6(A) shows an example of specifying a range with a cursor. Specifically, in FIG. 6(A), the user uses the cursor to specify the adjustment area in the displayed training data. In the example of FIG. 6(A), the user uses the cursor to specify a rectangular range 31 as the weight adjustment area. After specifying the range, the user inputs the weight adjustment amount. The weight adjustment amount can be, for example, a multiplication factor for the current weight value. For example, if the user inputs "1.5" as the weight adjustment amount, the model training unit 122 increases the weight for the training data belonging to range 31 by 1.5 times the current value. The weight can be reduced by inputting a value less than "1" as the weight adjustment amount. This method is effective when uniformly adjusting weights for a certain range of training data.

[0046] Figure 6(B) shows an example of inputting the weight adjustment location and adjustment amount by clicking with a mouse. Specifically, the user selects either the weight increase mode or the weight decrease mode, and then clicks on the data point on the training data graph that corresponds to the date they want to adjust. In the weight increase mode, the weight increases according to the number of clicks by the user. In the weight decrease mode, the weight decreases according to the number of clicks by the user. This method is effective when the weight adjustment amount differs for each date.

[0047] FIG. 6(C) shows an example of inputting the weight adjustment location and adjustment amount by specifying the length of the bar indicating the weight. In FIG. 6(C), the weight value set by the existing model for the training data for each date is shown by the weight bar. The user can adjust the weight for the training data for that day by changing the length of the weight bar for the day they want to adjust. In other words, if the user wants to increase the weight, they simply increase the length of the weight bar. As shown in FIG. 6(C), the weight bar 32a indicating the weight of the existing model and the portion 32b whose length has been changed by the user can be displayed in different colors or other ways to distinguish them, making it possible to easily see the amount of weight adjustment.

[0048] FIG. 7 shows an example of a correction UI for inputting problem correction information related to the output of an existing model. The user designates a specific period in the output of the existing model as a problem location and inputs the problem level of the model output for that problem location. When the problem location and problem level are input as correction information, the model training unit 122 corrects the existing model so that the problem at the specified problem location is resolved. For example, the model training unit 122 corrects the existing model so that the difference between the model output and the actual measured value at the specified problem location is reduced. Furthermore, the model training unit 122 increases the amount of correction to the existing model as the input problem level increases.

[0049] The model training unit 122 can also modify the model parameters simply by inputting the task modification information. In this case, input of the parameter modification information is not required. The method for modifying the model parameters based on the input of task modification information is as follows. First, when task modification information is specified for training data (e.g., graph 11 in display example G1 of Figure 4), the model training unit 122 changes the weight of the sample specified as the task location based on the task level in accordance with a predetermined rule. Also, when task modification information is specified for future data (e.g., graph 12 in display example G1 of Figure 4), the model training unit 122 changes the weight of the sample in the training data that is similar to the sample specified as the task location in accordance with a predetermined rule based on the task level. Here, the predetermined rule can be a "function or model indicating the relationship between the task level and the amount of weight adjustment" that is prepared regardless of past input logs.

[0050] FIG. 7(A) shows an example of specifying a range with a cursor. Specifically, in FIG. 7(A), the user uses the cursor to specify a range to be adjusted in the displayed output of an existing model. In the example of FIG. 7(A), the user uses the cursor to specify a rectangular range 33 as the problem area. After specifying the range, the user inputs the problem level of the problem area. The problem level can be expressed as, for example, a 10-point scale. The larger the problem level value, the greater the problem. For example, if the user inputs "3" as the problem level, the model training unit 122 sets the problem level of the output of the existing model belonging to range 33 to "3" out of 10. This method is effective when uniformly adjusting the problem level for a certain range of existing model outputs.

[0051] Figure 7(B) shows an example of inputting the problem areas and problem levels of the existing model output by clicking with a mouse. Specifically, the user selects either the mode for increasing the problem level or the mode for decreasing the problem level, and then clicks on the data point on the graph of the existing model that corresponds to the date they wish to adjust. In the mode for increasing the problem level, the problem level increases according to the number of clicks by the user. In the mode for decreasing the problem level, the problem level decreases according to the number of clicks by the user. This method is effective when the amount of adjustment of the problem level differs for each date.

[0052] FIG. 7(C) shows an example of inputting problem areas and the amount of problem level adjustment by specifying the length of a bar indicating the problem level (hereinafter referred to as the "problem level bar"). In FIG. 7(C), the problem level bar indicates the problem level of the output of the existing model on each date. Note that the problem level bar for each date is displayed by calculating the problem level based on, for example, the difference between the output of the existing model on each date and the actual measured value on that day. Note that, when the actual measured value of the objective variable has already been obtained, it is preferable to display the actual measured value in the model visualization UI and model comparison UI. For example, the actual measured value of the objective variable is displayed on graph 12 of display examples G1 and G2 in FIG. 14. This makes it easier for the user to find problem areas.

[0053] The user can adjust the degree of challenge of the existing model output for that day by changing the length of the challenge degree bar for the day they want to adjust. The amount of adjustment of the degree of challenge can be specified by the amount of change in the length of the challenge degree bar. In other words, if the user wants to increase the degree of challenge, they can simply increase the length of the challenge degree bar. As shown in Figure 7(C), the challenge degree bar 34a, which indicates the degree of challenge of the existing model output, and the portion 34b whose length has been changed by the user can be displayed in different colors or other ways to make the amount of adjustment of the degree of challenge easier to see.

[0054] (Regression analysis) Next, a case where the model task is regression analysis will be described. When the model task is regression analysis of data other than time series data, the evaluation information is displayed as a two-dimensional or three-dimensional scatter plot.

[0055] (1) Model visualization UI and modification UI Figure 8(A) shows a display example G11 of the model visualization UI and correction UI for regression analysis. Display example G11 is an example of a model predicting hamburger sales. The horizontal axis indicates the season from summer to winter, and the vertical axis indicates the time from daytime to nighttime. Each data point in the figure represents a predicted sales value. The larger the sales value, the darker (closer to black) the data point is, and the smaller the sales value, the lighter (closer to white) the data point is. In display example G11, the predicted value of the model for the training data used to train the existing model is shown as a circle, and the predicted value of the model for the validation data used to evaluate and verify the existing model is shown as a square. Note that the vertical and horizontal axes in Figure 8(A) are each examples of features. However, if there are three or more features, two features may be randomly selected from the three or more features, or multiple features may be combined and converted into two features. This also applies to Figure 8(B), which will be described later.

[0056] As in display example G11, by displaying the distribution of the model's predicted values ​​for the training data and validation data, the user can understand the trends of the existing model. For example, in display example G11, the predicted values ​​for the training data and the predicted values ​​for the validation data in the region "winter night" differ, indicating that the prediction accuracy of the existing model is low in the region "winter night." In this case, the user can modify the existing model by inputting parameter modification information in display example G11, for example, to increase the weight for data point 37 that corresponds to the condition "winter night."

[0057] FIG. 8(B) shows another display example G12. Display example G12 simultaneously displays the training data used to train the existing model and the model's predicted values ​​for the operation data used in actual operation. The training data shows the actual measured values ​​of sales. On the other hand, for the operation data, since actual measured values ​​have not been obtained, the predicted values ​​that are the results of making predictions on the operation data using the existing model are shown. Specifically, in display example G12, the data points of the training data are shown with a solid line around them, and the data points of the operation data are shown with a dotted line around them. Note that the vertical axis, horizontal axis, and colors indicating the sales values ​​are the same as those in FIG. 8(A).

[0058] As in display example G12, by displaying the distribution of training data and predicted values ​​for operational data, the user can understand the trends of the existing model. For example, in display example G12, for the condition "winter night," the values ​​of the training data are small, while the predicted values ​​for operational data are large. Therefore, the user can determine that the prediction accuracy of the existing model for the condition "winter night" is insufficient and that correction is necessary. In this case, too, the user can correct the existing model by inputting parameter correction information in display example G12 that increases the weight for data point 37 that corresponds to the condition "winter night."

[0059] (2) Model comparison UI and modification UI FIG. 9 shows a display example G13 of the model comparison UI and correction UI. Display example G13 plots the error between the training data and the output of an existing model in a two-dimensional feature space. The vertical and horizontal axes indicate some feature that defines the feature space. That is, each of the vertical and horizontal axes indicates one feature. Note that if the features used by the model are three-dimensional or more, the evaluation information output unit may randomly select two feature values ​​from three or more feature values, or may combine multiple feature values ​​and convert them into two feature values.

[0060] The color of each displayed data point indicates the error between the training data and the output of the existing model; the darker the color, the greater the error. Looking at display example G13, the user determines that the points located in the lower right region have a large error and inputs correction information for those points. For example, the user may use a stylus pen or the like to draw line segment 35 to enclose the region with a large error, thereby designating the points within the region surrounded by line segment 35 as problem areas. The user also inputs the problem level for the points within the region surrounded by line segment 35. Display example G13 thus allows the user to input problem correction information including the problem areas and the problem level. Note that line segment 35 may also be used to input parameter correction amounts, such as sample weight correction information.

[0061] In the display example G13, the color of each data point indicates the error between the training data and the output of the existing model, but instead, actual values ​​or predicted values ​​may be colored as shown in Figure 8.

[0062] The model training unit 122 modifies the existing model based on the input modification information to generate a modified model. The evaluation information output unit 125 generates a new display example G14 shown in FIG. 9 based on the training data and the output of the modified model, and displays it on the display device 2. In display example G14, it can be seen that the error at the points belonging to the lower right area has decreased, and the performance of the model has improved. Note that display example G14 shows the error between the training data and the output of the modified model, and it is difficult to compare the existing model with the modified model using display example G14 alone. Therefore, in addition to display example G14, it is recommended to simultaneously display display example G13, which shows the error between the training data and the output of the existing model.

[0063] The above display examples G13 and G14 display the error between the training data and the model output as evaluation information, but instead, the error between the validation data and the model output may be displayed, or actual values ​​and predicted values ​​may be displayed in different colors.

[0064] (3) How to enter correction information Next, we will explain in detail how to input correction information in the correction UI. As for the correction UI, there are two cases: one where parameter correction information is input in a display using a model visualization UI and a correction UI, as shown in Figures 8(A) and (B), and one where issue correction information is input in a display using a model comparison UI and a correction UI, as shown in Figure 9. However, since the input method by the user is the same, we will explain both together.

[0065] Figure 10(A) shows a method for inputting correction information by clicking on each point in the display of evaluation information. When inputting parameter correction information, the user can specify the weight adjustment location by clicking on the data point of the displayed training data whose weight he / she wants to change, and can specify the amount of weight adjustment by the number of clicks. When inputting problem correction information, the user can specify the problem location by clicking on the data point with a problem among the displayed data points such as errors, and can specify the degree of the problem by the number of clicks.

[0066] Figure 10(B) shows a method for inputting correction information by surrounding data points in the display of evaluation information with a line segment or the like, as in display example G13 in Figure 9. When inputting parameter correction information, the user simply specifies the weight adjustment location by surrounding the data points of the displayed training data whose weights the user wants to change with a line segment, and then separately inputs or specifies the amount of weight adjustment. Furthermore, when inputting problem correction information, the user simply specifies the problem location by surrounding the data points with a line segment that have a problem with the displayed data points, and then separately inputs or specifies the problem level.

[0067] FIG. 10(C) shows a method for inputting correction information by surrounding data points in the evaluation information display with lines, as in FIG. 10(B). However, while in FIG. 10(B) the same weight adjustment amount or challenge level is specified for all data points belonging to the area surrounded by the lines, in the example of FIG. 10(C) different weight adjustment amounts or challenge levels can be set for data points belonging to the area surrounded by the lines using gradations. Specifically, when inputting parameter correction information, the user simply specifies the weight adjustment location by surrounding the displayed training data points whose weights the user wants to change with lines, and then specifies different weight adjustment amounts using the gradation display. Furthermore, when inputting challenge correction information, the user simply specifies the challenge location by surrounding the displayed data points with challenges with lines, and then specifies different challenge levels using the gradation display.

[0068] (classification) Next, we will explain the case where the model task is classification. When the model task is classification, the evaluation information is displayed as a two-dimensional or three-dimensional scatter plot, just like in the case of regression analysis.

[0069] (1) Model visualization UI and modification UI Display example G13 shown in Figure 11 is a display example of a model visualization UI and correction UI for a classification model. The existing model is, for example, a binary classification model that classifies an image into two categories, "A" and "B." Display example G13, like display example G11 in Figure 8, shows a feature space with two specific feature values ​​on the vertical and horizontal axes. Each data point is shown in a color corresponding to the misclassification rate calculated based on the training data and the output of the existing model, with darker colors indicating a higher misclassification rate.

[0070] By displaying the misclassification rate, as in display example G13, the user can understand the tendency of the existing model. For example, display example G13 shows that the misclassification rate is high in the lower right region of the feature space. Therefore, the user can specify the weight adjustment location by inputting line segment 41 that surrounds the data points that need to be corrected, and can input correction information by separately specifying the weight adjustment amount.

[0071] Note that any known method can be used to calculate the misclassification rate. For example, in the case of binary classification, if the labels of the predicted value by the model and the actual measured value do not match, the misclassification rate can be set to "1", and if they match, the misclassification rate can be set to "0". In this case, in the display example G13 illustrated in FIG. 11, each data point has only two colors. As another example, when the classification model outputs a classification score, the difference between the actual measured value (correct value) and the classification score can be used as the misclassification rate. In other words, if the actual measured value (correct value) is set to "0" and "1", (Misclassification rate) = (Actual value) - (Classification score) This can be done as follows.

[0072] In the display example G13 of Figure 11, correction information is entered by surrounding the data point with a line segment 41, but in the case of a classification task, any of the methods shown in Figures 10(A) to (C) may be used.

[0073] (2) Model comparison UI and modification UI FIG. 12(A) shows a display example G14 of the model comparison UI and the correction UI for a classification model. Display example G14 shows the misclassification rate of the corrected model corrected based on the correction information entered by the user in display example G13. In this case, the misclassification rate is calculated based on the training data and the output of the corrected model. As can be seen by comparing with display example G13 in FIG. 11, the output of the corrected model shows an improved misclassification rate for data points in the lower right region of the feature space. The model comparison UI can distinguish and display data points whose classification results have changed between the existing model and the corrected model, for example, by highlighting the contours of data point 42, thereby showing the relationship between the outputs of the new and old models. Note that in display example G14, further correction information may be entered and the model corrected using any of the methods shown in FIGS. 10(A) to 10(C).

[0074] FIG. 12(B) shows another display example G15 of the model comparison UI. In display example G15, the color of each data point indicates the absolute value of the difference in classification score between the existing model and the modified model. In other words, the greater the difference in classification results between the existing model and the modified model, the darker the color of the data point. This also makes it possible to show the relationship between the outputs of the new and old models. Note that even in display example G15, further correction information may be input and the model may be corrected using any of the methods shown in FIGS. 10(A) to 10(C).

[0075] (Modified UI for explanatory variables) In the above example, the weight for each data point is adjusted as parameter modification information. However, instead, the weight may be adjusted for each explanatory variable. FIG. 13(A) shows an example of a modification UI for modifying the weight for each explanatory variable. This modification UI displays a list 51 of explanatory variables used by the existing model and a weight bar 52 set for each explanatory variable. In the list 51 of explanatory variables, explanatory variables with checked checkboxes are explanatory variables used by the existing model. The user can increase or decrease the weight for each explanatory variable by using cursor 52x to change the length of the weight bar 52 corresponding to each explanatory variable.

[0076] Figure 13(B) shows an example of a modification UI for changing the explanatory variables used by an existing model. This modification UI displays a list 53 of explanatory variables used by the existing model. If the user wants to delete an explanatory variable currently in use, they can simply uncheck the checkbox in list 53. On the other hand, if the user wants to add an explanatory variable, they click the "Add" tab in list 53, and an addition list 54 is displayed. The addition list 54 is a list of commonly used explanatory variables. The user can add a new explanatory variable by referring to addition list 54, checking the checkbox of the explanatory variable they want to add, and pressing the "OK" button.

[0077] [Model correction process] Next, the model correction processing by the model generation device 100 will be described. Fig. 14 is a flowchart of the model correction processing by the model generation device 100. The model correction processing is processing for correcting an existing model, and is executed at an appropriate timing, for example, while the model is in operation. This processing is realized by the processor 112 shown in Fig. 2 executing a prepared program and operating as the elements shown in Fig. 3.

[0078] First, the model generation device 100 acquires a target existing model (step S10). Next, the model generation device 100 displays the training data and the output of the existing model using a model visualization UI (step S11). The user looks at the relationship between the displayed training data and the output of the existing model and determines whether or not the existing model needs to be modified. If it is determined that the existing model needs to be modified, the user inputs modification information using the modification UI. The model generation device 100 determines whether or not the user has input modification information (step S12). If no modification information has been input (step S12: No), the process ends.

[0079] On the other hand, if correction information has been input (step S12: Yes), the model generation device 100 acquires the correction information. Specifically, if the user finds a problem in the output of an existing model and inputs problem correction information, the model generation device 100 acquires the problem correction information (step S13). Also, if the user determines that parameters such as weights and explanatory variables need to be corrected and inputs parameter correction information, the model generation device 100 acquires the parameter correction information (step S14).

[0080] Next, the model generation device 100 modifies the parameters of the existing model based on the input problem modification information and parameter modification information to generate a modified model (step S15). Then, the model generation device 100 displays the output of the existing model and the output of the modified model using a model comparison UI (step S16). Then, the process returns to step S12. In this way, steps S13 to S16 are repeated as long as the user determines that model modification is necessary, and when it is determined that model modification is unnecessary, the process ends.

[0081] In this way, the user can determine whether to modify the model by looking at the relationship between the training data and the output of the existing model displayed by the model visualization UI, and the relationship between the output of the existing model and the output of the modified model displayed by the model comparison UI. If the user determines that model modification is necessary, the user can input modification information using the modification UI and modify the model.

[0082] In the above model correction process, the input of problem correction information in step S13 and the input of parameter correction information in step S14 are performed separately. This allows a single user to input problem correction information and parameter correction information at different times. It is also possible for different users to input problem correction information and parameter correction information. For example, the model operator (AI operator) may input problem correction information, and the model creator (AI creator) may input parameter correction information to correct the model. In this case, smooth communication between the model creator and model operator allows for appropriate model correction.

[0083] In the above model correction process, the model generation device 100 acquires parameter correction information input by the user in step S14. However, if the model generation device 100 acquires task correction information input by the user in step S13, the model generation device 100 may function as a weight adjustment amount calculation unit. Specifically, a function or model indicating the relationship between task degrees and weight adjustment amounts is created in advance using task degrees included in task correction information previously input by the user and weight adjustment amounts included in the parameter correction information. When the model generation device 100 acquires task correction information in step S13, it uses the above function or model to present a recommended weight adjustment amount to the user. The user may accept the presented recommended value, change the presented recommended value, or enter the weight adjustment amount themselves. This can streamline the input of parameter correction information in step S14. Furthermore, when a user with little parameter experience modifies a model, they can refer to the weight adjustment amounts made by many past users.

[0084] [Variations] (Variation 1) In the above embodiment, the model generation device 100 generates the initial model based on training data, but this is not essential. For example, the model generation device 100 may acquire an existing model from an external source and retrain the existing model to generate a new modified model.

[0085] (Variation 2) The function of the model training unit 122 in the model generation device 100 may be provided externally. For example, a model training device having the function of the model training unit 122 in Fig. 3 and a model evaluation device having the functions of the evaluation data DB 124 and the evaluation information output unit 125 may be provided separately. In this case, the model evaluation device inputs the correction information D3 input from the input device 3 to the model training device and obtains the corrected model generated by the model training device from the model training device.

[0086] (Variation 3) In the above embodiment, the model generation device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured by a server and a terminal device. FIG. 15 is a block diagram showing a schematic configuration of a model generation system 1x using a server and a terminal device. In FIG. 15, the server 100x has the configuration of the model generation device 100 shown in FIG. 3. Furthermore, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG. 3. With this configuration, correction information entered by multiple users can be easily collected in the server 100x and shared.

[0087] Second Embodiment 16 is a block diagram showing the functional configuration of a model processing device according to the second embodiment. The information processing device 70 includes an existing model acquisition means 71, a first output means 72, a correction acquisition means 73, a corrected model acquisition means 74, and a second output means 75.

[0088] 17 is a flowchart of processing by the model processing device of the second embodiment. Existing model acquisition means 71 acquires an existing model (step S71). First output means 72 outputs first evaluation information indicating the relationship between evaluation data and the output of the existing model for the evaluation data (step S72). Correction acquisition means 73 acquires correction information input for the first evaluation information (step S73). Corrected model acquisition means 74 acquires a corrected model corrected based on the correction information input for the first evaluation information (step S74). Second output means 75 outputs second evaluation information indicating the relationship between the evaluation data and the output of the existing model and the corrected model (step S75).

[0089] According to the information processing device 70 of the second embodiment, by presenting sufficient information regarding the data used to create the model and the output of the model currently in operation, it becomes possible to appropriately correct the model.

[0090] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0091] (Appendix 1) an existing model acquisition means for acquiring an existing model; a first output means for outputting first evaluation information indicating a relationship between evaluation data and an output of the existing model with respect to the evaluation data; a correction acquisition means for acquiring correction information input for the first evaluation information; a corrected model acquisition means for acquiring a corrected model that has been corrected based on correction information input for the first evaluation information; a second output means for outputting second evaluation information indicating a relationship between the outputs of the existing model and the corrected model with respect to the evaluation data; A model processing device comprising:

[0092] (Appendix 2) the correction acquisition means acquires correction information input for the second evaluation information, 2. The model processing device according to claim 1, wherein the modified model acquisition means acquires a new modified model modified based on modification information input for the second evaluation information.

[0093] (Appendix 3) the first evaluation information includes the evaluation data and an output of the existing model; 2. The model processing device according to claim 1, wherein the second evaluation information includes an output of the existing model and an output of the modified model.

[0094] (Appendix 4) the first evaluation information indicates a difference between the training data and an output of the existing model; 2. The model processing device according to claim 1, wherein the second evaluation information indicates a difference in output between the existing model and the modified model.

[0095] (Appendix 5) the model is a model for predicting a time series value, the first evaluation information simultaneously indicates the evaluation data and an output of the model; 2. The model processing device according to claim 1, wherein the second evaluation information simultaneously indicates an output of the existing model and an output of the modified model.

[0096] (Appendix 6) the model is a regression model, the first evaluation information indicates the evaluation data and an output of the existing model, 2. The model processing device according to claim 1, wherein the second evaluation information is a graph showing errors between the evaluation data and outputs of the existing model and the modified model.

[0097] (Appendix 7) the model is a classification model; the first evaluation information indicates a misclassification rate of a classification result by the existing model, 2. The model processing device according to claim 1, wherein the second evaluation information indicates a degree of misclassification of the classification result by the modified model or a difference between the classification score of the existing model and the classification score of the modified model.

[0098] (Appendix 8) 8. The model processing device according to claim 1, wherein the correction information includes a designation of problem locations and problem levels in the output of the existing model.

[0099] (Appendix 9) 9. The model processing device according to claim 8, wherein the correction information includes a specification of the location and amount of adjustment of the weighting that the model assigns to the evaluation data.

[0100] (Appendix 10) 10. A model processing device according to claim 8 or 9, comprising a calculation means for calculating an adjustment amount of a weight that the model assigns to the evaluation data based on the problem level.

[0101] (Appendix 11) Get an existing model, outputting first evaluation information indicating a relationship between the evaluation data and an output of the existing model for the evaluation data; Acquire correction information input for the first evaluation information; obtaining a corrected model corrected based on correction information input for the first evaluation information; A model processing method that outputs second evaluation information indicating the relationship between the outputs of the existing model and the corrected model with respect to evaluation data.

[0102] (Appendix 12) Get an existing model, outputting first evaluation information indicating a relationship between the evaluation data and an output of the existing model for the evaluation data; Acquire correction information input for the first evaluation information; obtaining a corrected model corrected based on correction information input for the first evaluation information; A recording medium having recorded thereon a program for causing a computer to execute a process of outputting second evaluation information indicating the relationship between the outputs of the existing model and the modified model with respect to evaluation data.

[0103] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0104] 1. 1x Model Generation System 2, 2x display device 3. 3x input devices 7 Terminal Equipment 100 Model generation device 112 processors 121 Training Data DB 122 Model Training Department 123 Model DB 124 Evaluation information output unit

Claims

1. an existing model acquisition means for acquiring an existing model; a first output means for outputting first evaluation information indicating a relationship between evaluation data and an output of the existing model with respect to the evaluation data; a correction acquisition means for acquiring correction information input for the first evaluation information; a corrected model acquisition means for acquiring a corrected model that has been corrected based on correction information input for the first evaluation information; a second output means for outputting second evaluation information indicating a relationship between the outputs of the existing model and the corrected model with respect to the evaluation data; Equipped with A model processing device in which the correction information includes designation of problem locations and problem levels in the output of the existing model.

2. the correction acquisition means acquires correction information input for the second evaluation information, 2. The model processing apparatus according to claim 1, wherein the modified model acquisition means acquires a new modified model modified based on modification information input for the second evaluation information.

3. the model is a model for predicting a time series value, the first evaluation information simultaneously indicates the evaluation data and an output of the model; The model processing apparatus according to claim 1 , wherein the second evaluation information simultaneously indicates the output of the existing model and the output of the corrected model.

4. the model is a regression model; the first evaluation information indicates the evaluation data and an output of the existing model, The model processing apparatus according to claim 1 , wherein the second evaluation information is a graph showing errors between the evaluation data and the outputs of the existing model and the corrected model.

5. the model is a classification model; the first evaluation information indicates a misclassification rate of a classification result by the existing model, The model processing device according to claim 1 , wherein the second evaluation information indicates a degree of misclassification of the classification result obtained by the corrected model, or a difference between the classification score of the existing model and the classification score of the corrected model.

6. The model processing apparatus according to claim 1 , wherein the correction information includes a designation of the location and amount of adjustment of the weighting given by the model to the evaluation data.

7. The model processing device according to claim 1 or 6, further comprising a calculation means for calculating an adjustment amount of a weight given by the model to the evaluation data based on the level of challenge.

8. 1. A computer-implemented method of model processing, comprising: Get an existing model, outputting first evaluation information indicating a relationship between the evaluation data and an output of the existing model for the evaluation data; Acquire correction information input for the first evaluation information; obtaining a corrected model corrected based on correction information input for the first evaluation information; outputting second evaluation information indicating a relationship between the outputs of the existing model and the corrected model with respect to the evaluation data; A model processing method in which the correction information includes designation of problem locations and problem levels in the output of the existing model.

9. Get an existing model, outputting first evaluation information indicating a relationship between the evaluation data and an output of the existing model for the evaluation data; Acquire correction information input for the first evaluation information; obtaining a corrected model corrected based on correction information input for the first evaluation information; causing a computer to execute a process of outputting second evaluation information indicating a relationship between outputs of the existing model and the modified model with respect to evaluation data; The correction information is a program that includes designation of problem areas and problem levels in the output of the existing model.

Images