Estimation device, estimation method, and program

Abstract

An estimation device 100 according to the present invention comprises: a data selection means 121 for selecting, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, first explanatory variable data and first objective variable data corresponding to the first explanatory variable data, on the basis of second explanatory variable data that is not associated with an objective variable; and a performance estimation means 122 for estimating the performance of a prediction model's prediction about the second explanatory variable data, on the basis of the comparison between prediction data obtained by inputting the first explanatory variable data selected according to the prediction model and the first objective variable data corresponding to the first explanatory variable data.

Description

Estimation device, estimation method, and program

[0001] The present disclosure relates to an estimation device, an estimation method, and a program.

[0002] A method has been studied for creating a prediction model that predicts dependent variable data from explanatory variable data based on explanatory variable data and dependent variable data corresponding to the explanatory variable data, and then putting the created prediction model into practical use. For example, Patent Literature 1 discloses a technology for creating multiple prediction model candidates from explanatory variable data and new data generated from the explanatory variable data, and selecting a prediction model by evaluating the prediction accuracy of the multiple prediction model candidates.

[0003] WO 2021 / 229648

[0004] However, depending on the operating conditions of a prediction model, it may take time to obtain data on the dependent variable that is the target of the prediction model. In such cases, a problem occurs in that the value of the performance index for the prediction of the prediction model cannot be calculated until the dependent variable is obtained.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an estimation device, an estimation method, and a program that can estimate the predictive performance of a predictive model without using data on dependent variables.

[0006] An estimation device according to one aspect of the present invention includes: a data selection means for selecting, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated; and a performance estimation means for estimating prediction performance of the prediction model for the second explanatory variable data, based on a comparison between prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data.

[0007] Moreover, an estimation method according to one aspect of the present invention has the following configuration: from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated, select first explanatory variable data and first objective variable data corresponding to the first explanatory variable data; and estimate prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the selected first explanatory variable data into the prediction model and the first objective variable data corresponding to the first explanatory variable data.

[0008] Furthermore, a program as one aspect of the present invention is configured to cause a computer to execute the following processes: select, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated; and estimate prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the selected first explanatory variable data into the prediction model and the first objective variable data corresponding to the first explanatory variable data.

[0009] According to the present disclosure, it is possible to estimate the predictive performance of a predictive model during operation without using data on dependent variables.

[0010] FIG. 1 is a block diagram showing an example of a hardware configuration of an estimation device according to a first embodiment of the present disclosure. FIG. 2 is a block diagram showing an example of a configuration of an estimation device according to the first embodiment of the present disclosure. FIG. 3 is a flowchart showing an example of an operation of the estimation device according to the first embodiment of the present disclosure. FIG. 4 is a block diagram showing an example of a configuration of an estimation device according to a second embodiment of the present disclosure. FIG. 5 is a block diagram showing an example of an operation of the estimation device according to the second embodiment of the present disclosure. FIG. 6 is a diagram showing an example of an estimation process performed by the estimation device according to the second embodiment of the present disclosure.

[0011] Preferred embodiments of the present disclosure will be described in detail with reference to the drawings.

[0012] First Embodiment First, an example of a first embodiment of the present disclosure will be described. Fig. 1 is a block diagram showing an example of the hardware configuration of an estimation device according to this embodiment, and Fig. 2 is a block diagram showing an example of the configuration of the estimation device. Fig. 3 is a flowchart showing an example of the operation of the estimation device. Note that this embodiment shows an outline of the configuration of an estimation device and an estimation method described in a second embodiment, which will be described later.

[0013] First, the hardware configuration of an estimation device 100 according to this embodiment will be described with reference to Fig. 1. The estimation device 100 is configured as a general information processing device, and is equipped with the following hardware configuration, for example: - CPU (Central Processing Unit) 101 (arithmetic unit) - ROM (Read Only Memory) 102 (storage device) - RAM (Random Access Memory) 103 (storage device) - Programs 104 loaded into the RAM 103 - Storage device 105 storing the programs 104 - Drive device 106 for reading and writing data from and to a storage medium 110 external to the information processing device - Communication interface 107 for connecting to a communication network 111 external to the information processing device - Input / output interface 108 for inputting and outputting data - Bus 109 for connecting the various components

[0014] 1 shows an example of the hardware configuration of the information processing device that is the estimation device 100, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with a part of the above-described configuration, such as not including the drive device 106. Furthermore, instead of the above-described CPU, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

[0015] The estimation device 100 can be equipped with the data selection means 121 and performance estimation means 122 shown in FIG. 2 by having the CPU 101 acquire and execute the program group 104. The program group 104 is, for example, stored in advance in the storage device 105 or the ROM 102, and is loaded into the RAM 103 and executed by the CPU 101 as needed. The program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, and the drive device 106 may read out the programs and supply them to the CPU 101. However, the data selection means 121 and performance estimation means 122 described above may be constructed using dedicated electronic circuits for realizing such means.

[0016] First, the estimation device 100 has a function of acquiring a prediction model, first explanatory variable data, first objective variable data corresponding to the first explanatory variable data, and second explanatory variable data. The prediction model, when executed within the estimation device 100, has a function of outputting a prediction result for the corresponding objective variable data when explanatory variable data is input. In this case, the output data is referred to as prediction data. The first explanatory variable data and first objective variable data are data prepared according to the prediction model, and include, for example, first explanatory variable data and first objective variable data as training data for generating the prediction model by machine learning, and first explanatory variable data and first objective variable data for verifying the prediction model. Furthermore, the second explanatory variable data is explanatory variable data acquired when the prediction model is used in another task, and there is no objective variable data for the second explanatory variable data, and the second explanatory variable data is not associated with the second explanatory variable data.

[0017] The estimation device 100 may acquire the prediction model and each piece of data from outside the estimation device 100. For example, if the estimation device 100 is connected to other devices via a network, the prediction model and each piece of data may be acquired from another device that exists on the same network as the estimation device 100. In this case, the estimation device 100 may include a communication device for communicating with other devices on the network. Furthermore, if the estimation device 100 includes a storage device (not shown), the prediction model and each piece of data may be acquired from the storage device.

[0018] The data selection means 121 selects, based on the second explanatory variable data, some or all of the first explanatory variable data and the corresponding first objective variable data from the first explanatory variable data and the first objective variable data corresponding to the first explanatory variable data acquired by the estimation device 100. At this time, the data selection means 121 selects, based on the content of the second explanatory variable data, all or part of the first explanatory variable data that is more suitable for estimating the prediction performance of the prediction model for the second explanatory variable data. More specifically, the data selection means 121 selects all or part of the first explanatory variable data based on the content of the second explanatory variable data, with the aim of making the estimated value of the prediction performance index for the second explanatory variable data of the prediction model by the performance estimation means 122 (described later) closer to the actual value. For example, in estimating the prediction performance of a prediction model, it is generally desirable to use data of explanatory variables that the prediction model did not use during training. Therefore, the data selection means 121 may select all or part of the first explanatory variable data by selecting data of explanatory variables that were not used during training of the prediction model. As another example, when estimating the predictive performance of a prediction model for data on a second explanatory variable, it is preferable to use data on an explanatory variable corresponding to data on an explanatory variable that is close to the data on the second explanatory variable in the explanatory variable space. Therefore, the data selection means 121 may select all or part of the data on the first explanatory variable so that the distribution of the data on the selected explanatory variable is close to the spatial distribution of the explanatory variables (hereinafter simply referred to as "distribution") of the data on the second explanatory variable. The method for selecting the data on the reference explanatory variable may be more appropriately determined depending on the performance estimation method used by the performance estimation means 122, which will be described later. The data on the explanatory variables selected by the data selection means 121 is referred to as data on the reference explanatory variable. The data selection means 121 also selects data on the objective variable corresponding to the selected reference explanatory variable data, which is referred to as data on the reference objective variable. The reference explanatory variable data and the data on the objective variable corresponding to the reference explanatory variable data are collectively referred to as reference data.

[0019] The performance estimation means 122 estimates the prediction performance of the prediction model for the second explanatory variable data based on a comparison between the prediction data obtained by inputting the reference explanatory variable data into the prediction model and the reference objective variable data corresponding to the reference explanatory variable data. More specifically, for example, the performance estimation means 122 may use the value of a performance index calculated using the prediction data obtained by inputting the reference explanatory variable data into the prediction model and the reference objective variable data as the estimated value of the performance index for the second explanatory variable data of the prediction model. In this case, the performance estimation means 122 may further consider the difference in trend between the second explanatory variable data and the reference explanatory variable data, and adjust the estimated value of the performance index so that the larger the difference in trend, the worse the performance index becomes. Here, the difference in trend is information based on the results of comparing the contents of two data, and is, for example, a numerical non-negative value that represents the difference between the contents of the two data, i.e., the degree to which the contents of the two data differ. As an example of the difference in trend, a distance or index defined between the distributions of two data may be used.

[0020] The performance index used by the performance estimation means 122 quantitatively evaluates the quality or poorness of the prediction performance of the prediction model. The actual value of the performance index is calculated based on a comparison of the predicted data obtained by inputting data of the second explanatory variable into the prediction model with the data of the dependent variable corresponding to the data of the second explanatory variable. Therefore, if data of the dependent variable is not available, the actual value of the performance index cannot be calculated. Specific examples of performance indexes include mean absolute error, mean squared error, and coefficient of determination in the case of regression, and accuracy, F1 score, and cross entropy in the case of discrimination. Of course, the performance indexes that can be used by the performance estimation means 122 are not limited to these, and any evaluation index can be estimated.

[0021] The estimation device 100 having the above-described configuration executes the estimation method shown in the flowchart of FIG. 3 by using the functions of the data selection means 121 and performance estimation means 122 described above.

[0022] As shown in FIG. 3 , the estimation device 100 selects, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated (step S101), first explanatory variable data and first objective variable data corresponding to the first explanatory variable data, and estimates the prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the selected first explanatory variable data into the prediction model and the first objective variable data corresponding to the first explanatory variable data (step S102).

[0023] As described above, in the present disclosure, for example, by using a prediction model currently in operation, explanatory variable data used in creating the prediction model as data for the first explanatory variable (i.e., explanatory variable data used in training the prediction model and explanatory variable data used in validation), objective variable data used in creating the prediction model as data for the objective variable corresponding to the data for the first explanatory variable (i.e., objective variable data used in training the prediction model and objective variable data used in validation), and explanatory variable data used in running the prediction model as data for the second explanatory variable, it is possible to estimate the predictive performance for the explanatory variable data used in running the prediction model. In other words, when the prediction model is in operation, the estimation device 100 can estimate the predictive performance of the prediction model during operation using the explanatory variable data during operation, without waiting for the objective variable data during operation to be obtained.

[0024] The configuration shown in FIG. 2 is merely an example and does not necessarily limit the configuration of the estimation device according to this embodiment. For example, some functions of the estimation device 100 may be realized by multiple devices working together. As a specific example, the estimation device 100 may be configured with a first estimation device that acquires data on a first explanatory variable, data on a target variable corresponding to the data on the first explanatory variable, and data on a second explanatory variable, and a second estimation device that acquires a prediction model. Furthermore, in order to estimate the values ​​of performance indicators from multiple angles, the estimation device 100 may include multiple data selection means and multiple performance estimation means. Furthermore, the estimation device 100 may include a device that calculates a final estimate by the estimation device 100 based on estimates by the multiple performance estimation means. If the estimation device 100 also has functions other than estimating the values ​​of performance indicators of a prediction model, it may include devices not shown in FIG. 2 .

[0025] Second Embodiment Next, an example of a second embodiment of the present disclosure will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a block diagram for describing an example of the configuration of an estimation device according to this embodiment, and Fig. 5 is a flowchart for describing an example of the operation of the estimation device. Fig. 6 is a diagram for describing the state when the estimation device is actually operated.

[0026] In this embodiment, the data of the explanatory variables used in creating the prediction model is used as the data of the first explanatory variables. Furthermore, the data of the first dependent variable corresponding to the data of the first explanatory variables is used as the data of the dependent variable corresponding to the data of the explanatory variables used in creating the prediction model. Furthermore, the data of the explanatory variables at the time of operation of the prediction model is used as the data of the second explanatory variables. However, these are merely examples of the present embodiment as an example of an embodiment of the present disclosure and do not limit the embodiment of the present disclosure itself. As an example that is not limited to each data of the present disclosure, for example, a prediction model created using data of explanatory variables and data of dependent variables obtained in a certain task may be applied to another task to estimate the predictive performance of the prediction model for the data of the explanatory variables of the other task. In this case, the data of the explanatory variables of the other task can be used as the data of the second explanatory variables, rather than the data of the explanatory variables at the time of operation of the prediction model, and the estimation device according to the present disclosure can be applied. Alternatively, the data of the first explanatory variables and the data of the first dependent variable corresponding to the data of the first explanatory variables may be data other than that used in creating the prediction model. For example, the data of the explanatory variables at the time of operation of a prediction model for which data of the corresponding dependent variable has already been obtained and the data of the corresponding dependent variable may be used.

[0027] The estimation device 1 in this embodiment is configured with one or more information processing devices each including a calculation device and a storage device. As shown in FIG. 4 , the estimation device 1 includes a second data acquisition unit 10, an output unit 20, a control unit 30, and a storage unit 40. The control unit 30 further includes a reference data selection unit 3 and a performance estimation unit 4. The functions of the reference data selection unit 3 and the performance estimation unit 4 included in the second data acquisition unit 10, the output unit 20, and the control unit 30 can be realized by the calculation device executing a program for realizing each function stored in the storage device. The estimation device 1 also includes a storage unit 40 in the storage device that stores first data 41, a prediction model 42, and parameter information 43. The control unit 30 communicates data with the second data acquisition unit 10, the output unit 20, and the storage unit 40 via a communication network or the like.

[0028] The estimation device 1 has the function of estimating the prediction performance of the prediction model 42 stored in the storage unit 40 with respect to the data of the second explanatory variables acquired by the second data acquisition unit 10, i.e., the data of the explanatory variables during operation of the prediction model 42, by using the above-described components. The components will be described in detail below.

[0029] The second data acquisition unit 10 acquires explanatory variable data during operation of the prediction model as second explanatory variable data. The explanatory variable data during operation (i.e., second explanatory variable data) is data obtained during operation, and the prediction model predicts corresponding target variable data. For example, if a user of the estimation device 1 inputs explanatory variable data during operation, the second data acquisition unit 10 may include an interface for accepting user input, specifically, a touch panel, buttons, a voice input device, etc. Furthermore, if the estimation device 1 itself has the function of collecting second explanatory variable data, examples of the second data acquisition unit 10 include a sensor device such as a camera, a device for processing information acquired from the sensor device, and a device for storing the processed information as second explanatory variable data. If the second explanatory variable data is acquired from a device other than the estimation device 1, a communication device with the other device, etc.

[0030] The output unit 20 outputs the estimated value of the performance index calculated by the control unit 30. When the estimated value is output to the user of the estimation device 1, the output unit 20 corresponds to a display for displaying the estimated value, a speaker for outputting sound, or the like. When the estimated value is used in a device other than the estimation device 1, the output unit 20 corresponds to a device for communicating with the other device, or the like. The estimation device 1 may also have an alert function for notifying the user of a deterioration in prediction performance when the estimated value falls below a certain value, for example. In this case, the output unit 20 corresponds to a device for notifying the user of the deterioration in prediction performance.

[0031] The configuration of the storage unit 40 will be described. As shown in Fig. 4, the storage unit 40 pre-stores first data 41, a prediction model 42, and parameter information 43. The storage unit 40 transmits the first data 41, the prediction model 42, and the parameter information 43 to the control unit 30 as necessary.

[0032] The first data 41 includes data on explanatory variables used in creating the prediction model 42 (first explanatory variable data) and data on objective variables corresponding to the explanatory variable data used in creating the prediction model 42 (first objective variable data). The first data 41 includes training data used in training the prediction model 42 (i.e., data on explanatory variables used in training the prediction model 42 and data on objective variables corresponding to the explanatory variable data), and verification data used in verifying the generalization performance of the prediction model 42 (i.e., data on explanatory variables used in verifying the prediction model 42 and data on objective variables corresponding to the explanatory variable data).

[0033] The prediction model 42 is a prediction model created using the first data 41. Specifically, the prediction model 42 may be created by so-called supervised machine learning so that, when data of explanatory variables in training data among the first data 41 is input, the prediction model 42 predicts data of the objective variable in the training data. Furthermore, the generalization performance of the prediction model 42 may be verified using validation data among the first data 41. Examples of algorithms that realize the prediction model 42 include linear regression, decision trees, random forests, and neural networks. These are merely examples, and the algorithm that realizes the prediction model 42 is not limited as long as it is possible to predict data of the objective variable based on data of the explanatory variables.

[0034] The parameter information 43 is information on parameters necessary for estimating the value of the performance index, and may include, for example, information on the performance index to be estimated, a method for acquiring reference data (described later), and a method for estimating the performance index.

[0035] In this embodiment, the prediction model, the data on the first explanatory variables, and the data on the first objective variable corresponding to the data on the first explanatory variables do not change for each estimation, and therefore the processing required for acquisition can be reduced by storing them in the storage unit 40. On the other hand, the data on the second explanatory variables, i.e., the data on the explanatory variables used when the prediction model is operated, changes for each estimation, and therefore by acquiring the data using the second data acquisition unit 10, it becomes possible to estimate the value of the performance index for the data on the explanatory variables during the most recent operation.

[0036] Next, the configuration of the control unit 30 will be described. As shown in Fig. 4, the control unit 30 has a reference data selection unit 3 and a performance estimation unit 4. The control unit 30 also has a CPU, ROM, RAM, etc. (not shown), and performs various controls and calculations on the reference data selection unit 3 and the performance estimation unit 4.

[0037] The reference data selection unit 3 (data selection means) selects all or part of the explanatory variable data of the first data 41 based on the second explanatory variable data acquired by the second data acquisition unit 10, the prediction model 42 acquired from the storage unit 40, the first data 41, and the parameter information 43. At this time, the reference data selection unit 3 also selects objective variable data corresponding to the explanatory variable data of the selected first data 41. Here, the selected explanatory variable data is referred to as reference explanatory variable data. Furthermore, the objective variable data of the first data 41 corresponding to the reference explanatory variable data is referred to as reference objective variable data. The reference explanatory variable data and the reference objective variable data are collectively referred to as reference data.

[0038] As a specific example of a method for selecting reference data by the reference data selector 3, a method may be used in which only the validation data from the first data 41 is used as the reference data. The performance index estimation formula (1) described below uses the value of the performance index of the prediction model 42 for the reference data as a basis for the estimated value of the performance index of the prediction model 42 for the data of the second explanatory variable in the calculation. By not including the training data from the first data 41 and using only the validation data in the reference data, it is possible to avoid overestimation of the performance index value due to overfitting of the training data of the prediction model 42, and more appropriately calculate the estimated value of the performance index of the prediction model 42. Of course, all or part of the training data may be included in the reference data, and increasing the number of samples included in the reference data, including the training data, may be useful for appropriately estimating the value of the performance index. For example, in cases where it is not possible to select data of the first explanatory variable that is related to the data as described below, such as when the attributes or characteristics of the data cannot be classified based on the content of the second explanatory variable, the reference data selector 3 may select only the validation data or all data including the training data.

[0039] As another example of a method for selecting reference data by the reference data selector 3, all or part of the first data 41 that is highly relevant to the prediction performance of the prediction model 42 for the data of the second explanatory variable may be selected as reference data. Here, "highly relevant" refers to a positive correlation between the prediction performance of the prediction model 42 for the data of the second explanatory variable and the prediction performance of the prediction model 42 for the data of the reference explanatory variable, such that the two prediction performances are considered to be comparable. In other words, in this case, a comparison of the data of the second explanatory variable with the data of the first explanatory variable in the first data 41 indicates that the data content is highly relevant according to a predetermined standard for the second explanatory variable data. The performance index estimation formula (1) described below uses the value of the performance index of the prediction model 42 for the reference data as a standard for calculating the estimated value of the performance index of the prediction model 42 for the data of the second explanatory variable. Therefore, by using a portion of the data that is highly relevant to the data of the second explanatory variable as reference data, the estimated value of the performance index of the prediction model 42 can be more appropriately calculated. When selecting data highly relevant to the data of the second explanatory variable from the first data 41, empirical knowledge about the data (generally referred to as domain knowledge) or characteristics of the prediction model 42 may be used. A specific method for acquiring highly relevant data is, for example, when the data of the explanatory variable includes date and time information, to acquire data of the same month and day, the same day of the week, or the same season as any sample included in the data of the second explanatory variable from the first data 41. As another example, for an explanatory variable that the prediction model 42 particularly emphasizes during prediction, data whose values ​​are close to those of the data of the second explanatory variable may be acquired from the first data 41.

[0040] As another example of a method for selecting reference data by the reference data selector 3, a portion of the first data 41 may be selected as reference data, the portion of which exhibits a smaller difference in trend from the data for the second explanatory variable, i.e., a smaller difference in data content based on a preset standard. The performance index estimation formula (1) described below calculates an estimated value such that the greater the difference in trend between the data for the second explanatory variable and the data for the reference explanatory variable, the worse the value of the performance index of the prediction model 42 for the data for the second explanatory variable. The data for the second explanatory variable often contains only a limited amount of data and a limited range in the explanatory variable space compared to the first data 41. Under such circumstances, if all the explanatory variable data in the first data 41 were used as the data for the reference explanatory variable, the difference in trend between the data for the second explanatory variable and the data for the reference explanatory variable would be greater, resulting in a poorly estimated value of the performance index of the prediction model 42 for the data for the second explanatory variable. However, there are cases in which the explanatory variable data of the first data 41 contains samples that are close to each sample of the second explanatory variable in the explanatory variable space. In other words, the prediction model 42 may have already been trained or validated using data (first data 41) used to create the prediction model 42 that is close to the explanatory variable data (second explanatory variable data) during operation. In such cases, the actual value of the prediction performance index for the second explanatory variable data of the prediction model 42 will be better. Therefore, by selecting some of the explanatory variable data in the first data 41 that has a small difference in trend from the second explanatory variable data and using this as the reference explanatory variable data, it is possible to estimate a value close to the actual value of the prediction performance index for the explanatory variable data of the prediction model 42. Furthermore, the trend difference index may be calculated using the same index as the trend difference index used in the performance index estimation formula (1) described below, or may be calculated using a different index. Furthermore, the trend difference index may be calculated using a combination of multiple different indexes.

[0041] Here, as an example of a specific method for the reference data selection unit 3 to select data with a small difference in trend from the explanatory variable data of the first data 41, one or more samples closest to the second explanatory variable data are obtained from the explanatory variable data of the first data 41, and these samples are used as temporary reference explanatory variable data. Next, from each sample of the explanatory variable data of the first data 41 that is not included in the temporary reference explanatory variable data, one or more samples that, when added to the temporary reference explanatory variable data, will better reduce the difference in trend between the temporary reference explanatory variable data and the second explanatory variable data are obtained and added to the temporary reference explanatory variable data. The above addition process is repeated, for example, until the number of samples included in the temporary reference explanatory variable data exceeds a certain number, and finally, the temporary reference explanatory variable data is used as the reference explanatory variable data, thereby making it possible to select data with a small difference in trend from the explanatory variable data of the first data 41 with the second explanatory variable data.

[0042] The reference data selection unit 3 may use a selection method that combines two or more examples of the above-described reference data selection methods. For example, the reference data selection unit 3 may first select validation data from the first data 41, then select a portion of data that is highly relevant to the data of the second explanatory variable, and further select a portion of data that has a small difference in trend from the data of the second explanatory variable. Furthermore, the reference data selection method may be a more suitable selection method that takes into account the specific estimation method used by the performance estimation unit 4, which will be described later. In other words, if it is empirically or theoretically considered that the estimation method used by the performance estimation unit 4 will enable more accurate calculation of performance estimates using a specific reference data selection method, the above-described selection method may be used.

[0043] Note that if the difference in trend between the explanatory variable data of the first data 41 and the second explanatory variable data is large and any sample of the second explanatory variable data has low relevance to samples included in the explanatory variable data of the first data 41, the difference in trend with the second explanatory variable data will never be 0, no matter how all or part of the explanatory variable data of the first data 41 is selected. Therefore, if the trend between the explanatory variable data of the first data 41 and the second explanatory variable data is large, even if all or part of the explanatory variable data of the first data 41 is selected as described above, it is possible to estimate a deterioration in the prediction performance of the prediction model 42 for the second explanatory variable data.

[0044] The performance estimation unit 4 (performance estimation means) calculates an estimated value P of the performance index of the prediction model 42 for the second explanatory variable data, for example, using equation (1) described below, based on the reference data, the data of the second explanatory variable, and the prediction model 42 and parameter information 43 acquired from the storage unit 40. The performance index value estimated by the calculation is output to the output unit 20. <Calculation formula for estimated value of performance index> P = B + D × A ... (1) where, P: estimated value of the performance index of the prediction model 42 for the second explanatory variable data, B: performance index value of the prediction model for the reference explanatory variable data and the reference objective variable data, D: non-negative value indicating the difference in trend between the second explanatory variable data and the reference explanatory variable data, and A: rate of change of the value of the performance index according to D, calculated based on the reference explanatory variable data and a comparison between the prediction data obtained by inputting the reference explanatory variable data into the prediction model 42 and the reference objective variable data.

[0045] In Equation (1), B is the value of the performance index of the predictive model for the reference explanatory variable data and the reference dependent variable data. In particular, when there is absolutely no difference in trend between the reference explanatory variable data and the second explanatory variable data (i.e., when the value of D in Equation (1) is 0), the estimated value P of the performance index is the value of B. In fact, when there is absolutely no difference in trend between the reference explanatory variable data and the second explanatory variable data, the second explanatory variable data and the reference explanatory variable data can be considered identical. Therefore, unless a change in the relationship between the explanatory variable data and the dependent variable data (i.e., concept drift) occurs, the predictive performance of the predictive model 42 for the second explanatory variable data will be almost identical to the predictive performance of the predictive model 42 for the reference explanatory variable data. Equation (1) enables the estimation of the value of the performance index based on this fact. In other words, when concept drift does not occur and the reference explanatory variable data can be selected so that it has the same trend as the second explanatory variable data, Equation (1) can accurately estimate the value of the performance index from the reference explanatory variable data and the reference dependent variable data.

[0046] Furthermore, when there is a difference in trend between the data for the reference explanatory variable and the data for the second explanatory variable (i.e., when the value of D in formula (1) is greater than 0), formula (1) assumes the value of B in formula (1) to be the best value and estimates the deterioration in performance based on the values ​​of D and A in formula (1). Generally, the predictive performance of a predictive model will deteriorate for explanatory variable data that has a difference in trend from the explanatory variable data of the training data for the predictive model, and formula (1) can estimate the deterioration in the predictive performance of the predictive model 42 for the data for the second explanatory variable based on this.

[0047] D in Equation (1) is a non-negative value that quantitatively indicates the difference in trend between the data of the second explanatory variable and the data of the reference explanatory variable. D is 0 when there is no difference in trend between the data of the second explanatory variable and the data of the reference explanatory variable, and a large value when the difference is large. As a specific example of D, the difference between the distribution of the data of the second explanatory variable and the distribution of the data of the reference explanatory variable can be calculated, and this value can be used as D. Examples of indices for measuring the difference in data distribution include Kullback-Leibler divergence, Jensen-Shannon divergence, Wasserstein distance, and Maximum Mean Discrepancy (hereinafter referred to as MMD). These indices are 0 when there is no difference in data distribution, and take larger values ​​as the difference in data distribution increases. As another example, statistics such as the mean and variance of the explanatory variables can be calculated for each of the data of the second explanatory variable and the data of the reference explanatory variable, and D can be calculated based on the difference between them. As another example, the proportion of samples of the data of the second explanatory variable for which no sample of the data of the reference explanatory variable exists within a certain distance from each sample may be calculated, and this value may be set as D.

[0048] In the above example, the value of D may be calculated using all of the explanatory variables in the data for the second explanatory variable and the data for the reference explanatory variable, or only some of the explanatory variables. For example, only some of the explanatory variables that have a particularly strong influence on the prediction by the prediction model 42 may be used. Furthermore, in the above example, the value of D may be calculated using all of the data for the second explanatory variable and the data for the reference explanatory variable, or only some of the data. For example, when the data for the second explanatory variable consists of a very large number of samples, the value of D may be calculated by sampling the data for the second explanatory variable. This reduces the time required to calculate D in equation (1), enabling high-speed estimation.

[0049] Additionally, in calculating the value of D in equation (1), in order to adjust the influence of each explanatory variable on D, the values ​​of each explanatory variable in the data for the second explanatory variable and the data for the reference explanatory variable may be transformed before calculating D. For example, in order to make the influence of each explanatory variable on D the same, each explanatory variable may be transformed using Min-Max normalization or Z-Score normalization. As another example, in order to increase the influence of some explanatory variables that have a particularly strong influence on the prediction results of the prediction model 42 on D, the values ​​of the explanatory variables in the data for the second explanatory variable and the data for the reference explanatory variable may be doubled, for example.

[0050] In formula (1), A is the rate of change in the value of the performance index according to D, calculated based on the data of the standard explanatory variables and a comparison between the predicted data obtained by inputting the data of the standard explanatory variables into the prediction model 42 and the data of the standard objective variable. If the higher the value of the performance index, the better the performance of the prediction model (for example, when the coefficient of determination, accuracy, F1 score, etc. are used as performance indexes), A is a negative value. If the lower the value of the performance index, the better the performance of the prediction model (for example, when the mean absolute error, mean squared error, cross entropy, etc. are used as performance indexes), A is a positive value. This makes it possible to formulate the deterioration in the predictive performance of the prediction model 42 for the data of the second explanatory variable as the value of D in formula (1) increases.

[0051] The value of A in equation (1) may be, for example, a constant. More specifically, for example, when data on explanatory variables and data on dependent variables corresponding to the data on explanatory variables are available in addition to the first data, the value of A may be set so that the estimated value of P according to equation (1) when the data on the explanatory variables is used as data on the second explanatory variable becomes the actual value of the performance index of the prediction model 42 for the data on the explanatory variables and the data on the dependent variable. Furthermore, the value of A may be suitably set based on, for example, theoretical analysis or empirical experimental results.

[0052] Furthermore, the value of A in formula (1) may be calculated based on data for the reference explanatory variables and a comparison between the predicted data obtained by inputting the data for the reference explanatory variables into the prediction model 42 and the data for the reference objective variable. As an example of a specific calculation method, the calculation formula for the rate of deterioration of prediction performance based on distribution robust optimization shown in the following formula (2) may be used. This calculation formula for the rate of deterioration of prediction performance based on distribution robust optimization is a method of mathematically calculating the value of A from the value of a prediction performance index for the data for the reference explanatory variables of the prediction model 42 and the distribution of the data for the reference explanatory variables. Note that when the calculation formula based on formula (2) is used, the value of A is positive, and therefore the prediction performance index used is one in which the lower the value of the index, the better the performance of the prediction model.

[0053] <Formula for calculating the rate of deterioration of prediction performance based on distribution robust optimization> A = (Σij Kij × Li × Lj) ^ (1 / 2) ... (2) where, A: the value of A in formula (1), Σij: the summation symbol when i and j are each changed from 1 to the number of data points of the reference explanatory variable, Kij: the value resulting from the calculation of the Laplace kernel of the i-th and j-th explanatory variables of the data point of the reference explanatory variable, Li: the value of the index of prediction performance calculated based on the i-th value of the data point of the reference objective variable and the output value when the i-th sample of the data point of the reference explanatory variable is input into the prediction model, Lj: the value of the index of prediction performance calculated based on the j-th value of the data point of the reference objective variable and the output value when the j-th sample of the data point of the reference explanatory variable is input into the prediction model.

[0054] By using the value of MMD based on the Laplace kernel for D in equation (1) and then setting the value calculated by equation (2) as A in equation (1), it is possible to estimate the value of the performance index when the prediction performance is the worst, based on the MMD distance between the data of the reference explanatory variable and the data of the second explanatory variable.

[0055] <Explanation of Operation> Next, an example of the overall operation according to this embodiment will be described in detail with reference to the block diagram of Fig. 4 and the flowchart of Fig. 5. Fig. 5 is an example of a flowchart showing the processing procedure of the estimation device 1 in the first embodiment.

[0056] First, when the processing of the estimation device 1 is started, the second data acquisition unit 10 acquires data on the second explanatory variable (step S1). Next, the control unit 30 acquires the first data 41, the prediction model 42, and the parameter information 43 from the storage unit 40 (step S2). The reference data selection unit 3 selects reference data based on the data on the second explanatory variable, the first data 41, and the parameter information 43 (step S3). The performance estimator 4 calculates an estimate of the performance index using equation (1) based on the reference data, the data on the second explanatory variable, the prediction model 42, and the parameter information 43 (step S4). Finally, the estimation device 1 outputs the estimate of the performance index obtained by the performance estimator 4 via the output unit 20 (step S5).

[0057] 5 is merely an example of the present embodiment and does not necessarily limit the flow of operations according to the present embodiment. As a specific example, the process S1 for acquiring the second explanatory variables may be executed after the process (step S2) for acquiring the first data 41, the prediction model 42, and the parameter information 43. Furthermore, step S2 may be divided into multiple steps, and for example, the process for acquiring the prediction model 42 in step S2 may be executed after the process (step S3) for selecting the reference data.

[0058] <Example> Next, as an example of an estimation device according to an embodiment of the present disclosure, an example of operation of the system assuming a specific use case will be described.

[0059] (Example 1: Demand forecast for ice cream at a supermarket one month from now) First, in Example 1, an example will be described in which the demand for ice cream one month from now is predicted as the objective variable at a physical supermarket, using the current month, date, day of the week, season, weather, temperature, humidity, number of customers, sales of related products, etc. as explanatory variables. In this case, the actual sales volume cannot be known until one month has passed and the ice cream is actually sold. In other words, the demand for ice cream one month from now, which is the value of the objective variable, cannot be known until one month has passed since the prediction was made, and the predictive performance of the current prediction model, more specifically, for example, the mean squared error of the predicted value of the prediction model for the most recent week, cannot be known. Therefore, the present disclosure estimates the mean squared error of the prediction model's prediction for the most recent week.

[0060] The prediction model is created using data acquired over a certain period of time in the past, for example, three years' worth of data (i.e., including daily explanatory variable data for the three years and daily objective variable data, i.e., the number of ice creams sold one month after each day) as data for creating the prediction model. Here, the three years' worth of data is referred to as data for the first explanatory variable and data for the objective variable corresponding to the data for the first explanatory variable, and is collectively referred to as the first data. The created prediction model and the first data are stored in a predetermined storage area.

[0061] Next, during actual operation, the mean square error of the prediction model for the most recent week is estimated. That is, the data of the explanatory variables for the most recent week is used as the data of the second explanatory variables. Estimating the prediction performance requires a method for selecting reference data, a method for calculating the value of D in equation (1), and a method for calculating the value of A in equation (1).

[0062] First, as the reference data, data from the first data whose dates match those included in the data for the second explanatory variable are selected. This is based on the idea that the predictive performance of the prediction model for the same month in the same year will generally be comparable, i.e., the predictive performance of the prediction model for the data for the second explanatory variable is highly relevant. This idea can be obtained, for example, from experience with supermarket sales or a detailed analysis of the first data (i.e., domain knowledge). This makes it possible to more appropriately estimate the mean squared error value for the most recent week, which is the target of estimation, based on data that is highly relevant to the data for the second explanatory variable.

[0063] Next, the MMD of the data of the reference explanatory variable and the data of the second explanatory variable is used as the value of D in equation (1). More specifically, MMD is calculated using a Laplace kernel with sigma (parameter) set to 1. Furthermore, Z-score normalization is performed before calculating D. The method of calculating the value of D using MMD, together with the method of calculating the value of A in equation (1) using equation (3) described below, is a method of estimating mean square error based on distribution robust optimization, and is an estimation method based on theoretical analysis.

[0064] The value of A in formula (1) is calculated by the following formula (3) by specifying the above formula (2) based on the reference data and the prediction model. The calculation of the value of A using formula (3) is a calculation method derived from analysis based on distribution robust optimization.

[0065] <Formula for calculating rate of change A> A = (Σij Kij × (Yi - Pi)^2 × (Yj - Pj)^2)^(1 / 2) ... (3) where, A: the value of A in formula (1), Σij: the summation symbol when i and j are each changed from 1 to the number of data points of the reference explanatory variable, Kij: the value resulting from the Laplace kernel calculation with sigma (parameter) of the ith and jth explanatory variables of the Z-Score-normalized reference explanatory variable data set set to 1, Yi: the ith value of the reference objective variable data, Yj: the jth value of the reference objective variable data, Pi: the output value when the ith sample of the reference explanatory variable data is input to the prediction model, Pj: the output value when the jth sample of the reference explanatory variable data is input to the prediction model.

[0066] By using the method for selecting reference data specified above, the method for calculating the value of D in equation (1), and the method for calculating the value of A in equation (1) using equation (3), it is possible to estimate the mean square error of the prediction model for the most recent week during actual operation.

[0067] FIG. 6 is a schematic diagram of a user interface that displays the mean square error estimation results according to this embodiment to the user in the form of a graph. The graph in FIG. 6 shows the actual measured mean square error values ​​and the predicted values ​​according to this embodiment in a line graph, with the horizontal axis representing the date and the vertical axis representing the mean square error. FIG. 6 also shows a bar graph showing the breakdown of P in equation (1) for the predicted mean square error values ​​according to this embodiment, broken down into the value of B in the first term on the right-hand side of equation (1) and the value of D×A in the second term. Because it takes one month to determine the value of the dependent variable, i.e., the number of ice creams sold, the actual measured mean square error values ​​can only be calculated and plotted for predictions up to one month in advance. On the other hand, the prediction performance estimate according to this disclosure can be calculated to include predictions up to today. Simultaneous display of the breakdown also makes it easier for users to understand the estimated prediction performance values ​​and changes in the estimated values.

[0068] Example 2: Disease Diagnosis Prediction Based on Early Symptoms Next, as Example 2, an example of disease diagnosis prediction based on early symptoms will be described. More specifically, a patient's symptoms, such as today's body temperature, yesterday's body temperature, the day before yesterday's body temperature, whether or not they have a sore throat, whether or not they have a stuffy nose, whether or not they have a cough, and whether or not they feel fatigued, are used as explanatory variables, and a prediction model is used to predict the patient's illness, such as a cold, influenza, allergic rhinitis, streptococcal infection, acute bronchitis, or measles, is used as a response variable. Actual illness diagnosis requires a doctor's diagnosis, and it may take time for the diagnosis to be made, for the symptoms required for diagnosis to develop, and for the test results required for diagnosis to be obtained. Therefore, the prediction accuracy of the prediction model for, for example, the most recent 30 patients cannot be known until the diagnosis results for all patients are available. Therefore, an estimation device according to the present disclosure estimates the prediction accuracy of predictions for the most recent 30 patients.

[0069] First, a prediction model is created by using, as data for creating the prediction model, the symptoms (which are explanatory variable data) and the disease assessment results (which are objective variable data) for a certain number of people in the past, for example, 1,000 people. Here, the data for the past 1,000 people is referred to as data for the first explanatory variable and data for the objective variable corresponding to the data for the first explanatory variable, and is collectively referred to as the first data. The created prediction model and the first data are stored in a predetermined storage area.

[0070] Next, during actual operation, the prediction accuracy of the prediction model for the most recent 30 people is estimated. That is, the data of the explanatory variables for the most recent 30 people is used as the data of the second explanatory variable. Estimating the prediction accuracy requires a method for selecting reference data, a method for calculating the value of D in equation (1), and a method for calculating the value of A in equation (1).

[0071] In this example, all of the first data is used as the reference data. In other words, all of the first data is selected as the reference data. In this embodiment, the value of D in formula (1) is calculated based on whether the symptom is unknown compared to each symptom in the reference data, as described below. In reality, all of the first data are known symptoms. Therefore, by selecting all of the first data as the reference data, all of the first data can be considered known symptoms, enabling more suitable estimation of prediction accuracy. Note that, depending on the calculation method for the value of D, the number of first data, and the presence or absence of domain knowledge related to the symptoms, the reference data may be selected using a more detailed method; however, in this embodiment, all of the data is selected for the reasons described above.

[0072] The value of D in formula (1) is the proportion of samples in the Min-Max normalized second explanatory variable data that are not within a certain distance, for example, a Euclidean distance of 1.0, from each sample. This calculates the proportion of samples in the second explanatory variable data that are not included in the first explanatory variable data, i.e., have explanatory variables unknown to the predictive model that the predictive model has not trained or validated. If all samples in the second explanatory variable data are included in the first explanatory variable data, the value of D is 0.

[0073] Finally, the value of A in Equation (1) is set to a constant of −1. This estimation method, along with the calculation method for D, is based on the assumption that predictions made by a prediction model for samples that the model has not been trained or validated on will generally be incorrect.

[0074] By using the above method for selecting reference data and setting the values ​​of D and A in equation (1), the accuracy of the prediction model for the data used to create the model is set as an upper limit, and predictions by the prediction model for samples that the prediction model has not trained or verified will be incorrect. As such, the prediction accuracy of the prediction model for the most recent 30 people can be estimated.

[0075] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each of the above-described means and functions may be executed by an information processing device installed and connected anywhere on a network, i.e., may be executed by so-called cloud computing.

[0076] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0077] <Supplementary Notes> Some or all of the above embodiments can be described as in the following supplementary notes. Below, an outline of the configurations of the estimation device, estimation method, and program of the present invention will be described. However, the present invention is not limited to the following configurations. (Supplementary Note 1) An estimation device comprising: a data selection means that selects, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated; and a performance estimation means that estimates the prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first objective variable data corresponding to the first explanatory variable data. (Supplementary Note 2) The estimation device according to Supplementary Note 1, wherein the performance estimation means estimates the prediction performance of the prediction model for the second explanatory variable data based on a comparison between the prediction data and the first dependent variable data and a comparison between the second explanatory variable data and the selected first explanatory variable data. (Supplementary Note 3) The estimation device according to Supplementary Note 2, wherein the performance estimation means estimates the prediction performance of the prediction model for the second explanatory variable data based on the prediction performance of the prediction model for the selected first explanatory variable data based on the comparison between the prediction data and the first dependent variable data and a difference in data content between the second explanatory variable data and the selected first explanatory variable data based on a preset criterion. (Supplementary Note 4) The estimation device according to Supplementary Note 2, wherein the performance estimation means estimates that the performance of the prediction model for the second explanatory variable data deteriorates as the difference in data content between the second explanatory variable data and the selected first explanatory variable data based on a preset criterion increases.(Supplementary Note 5) The estimation device according to Supplementary Note 4, wherein the performance estimation means estimates that the value of the prediction performance of the prediction model for the selected first explanatory variable data, based on a comparison between the prediction data and the first dependent variable data, deteriorates as a difference between the data contents of the second explanatory variable data and the selected first explanatory variable data, based on a preset criterion, increases. (Supplementary Note 6) The estimation device according to Supplementary Note 4, wherein the performance estimation means calculates a deterioration rate of the performance of the prediction model for the second explanatory variable data, corresponding to a difference between the distribution of the second explanatory variable data and the distribution of the selected first explanatory variable data, using an equation for a prediction performance deterioration rate based on distribution robust optimization calculated from the selected first explanatory variable data and a comparison between the prediction data and the selected first dependent variable data, and estimates the performance of the prediction model for the second explanatory variable data based on the deterioration rate. (Supplementary Note 7) The estimation device according to Supplementary Note 1, wherein the data selection means selects data from the first explanatory variable data that is different from data used when training the prediction model. (Supplementary Note 8) The estimation device according to Supplementary Note 1, wherein the data selection means selects, from the first explanatory variable data, data that has a predetermined correlation with the second explanatory variable data with regard to the prediction performance of the prediction model. (Supplementary Note 9) The estimation device according to Supplementary Note 1, wherein, from the first explanatory variable data, the data selection means selects the first explanatory variable data that has a smaller difference from the second explanatory variable data based on a predetermined criterion in data content than other data.(Supplementary Note 10) An estimation method comprising: selecting, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated; and estimating prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the selected first explanatory variable data into the prediction model and the first objective variable data corresponding to the first explanatory variable data. (Supplementary Note 11) A computer-readable storage medium storing a program for causing a computer to execute a process of selecting, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated, first explanatory variable data and first objective variable data corresponding to the first explanatory variable data, and estimating prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the selected first explanatory variable data into the prediction model and the first objective variable data corresponding to the first explanatory variable data.

[0078] REFERENCE SIGNS LIST 1 Estimation device 3 Reference data selection unit 4 Performance estimation unit 10 Second data acquisition unit 20 Output unit 30 Control unit 40 Storage unit 41 First data 42 Prediction model 43 Parameter information 100 Estimation device 101 CPU 102 ROM 103 RAM 104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input / output interface 109 Bus 110 Storage medium 111 Communication network 121 Data selection means 122 Performance estimation means

Claims

1. a data selection means for selecting, from first explanatory variable data prepared according to a preset prediction model and first objective variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no objective variable is associated; a performance estimation means for estimating a prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first dependent variable data corresponding to the first explanatory variable data; An estimation device having the following:

2. The estimation device according to claim 1 , the performance estimation means estimates a prediction performance of the prediction model for the second explanatory variable data based on a comparison between the prediction data and the first dependent variable data and a comparison between the second explanatory variable data and the selected first explanatory variable data; Estimation device.

3. The estimation device according to claim 2, the performance estimation means estimates the prediction performance of the prediction model for the selected first explanatory variable data based on a comparison between the prediction data and the first dependent variable data, and based on a difference in data content between the second explanatory variable data and the selected first explanatory variable data based on a preset criterion. Estimation device.

4. The estimation device according to claim 2, the performance estimation means estimates that the performance of the prediction model for the second explanatory variable data becomes worse as a difference between data contents of the second explanatory variable data and the selected first explanatory variable data based on a preset criterion becomes larger. Estimation device.

5. The estimation device according to claim 4, the performance estimation means estimates that a value of prediction performance of the prediction model for the selected first explanatory variable data based on a comparison between the prediction data and the first dependent variable data becomes worse as a difference between data contents of the second explanatory variable data and the selected first explanatory variable data based on a preset criterion becomes larger. Estimation device.

6. The estimation device according to claim 4, the performance estimation means calculates a deterioration rate of performance of the prediction model for the second explanatory variable data according to a difference between a distribution of the second explanatory variable data and a distribution of the selected first explanatory variable data, using an arithmetic expression for a prediction performance deterioration rate based on distribution robust optimization calculated from the selected first explanatory variable data and a comparison between the prediction data and the selected first objective variable data, and estimates the performance of the prediction model for the second explanatory variable data based on the deterioration rate. Estimation device.

7. The estimation device according to claim 1 , The data selection means selects, from the first explanatory variable data, data different from data used at the time of training the prediction model. Estimation device.

8. The estimation device according to claim 1, The predictive model is created by machine learning. Estimation device.

9. selecting, from first explanatory variable data prepared according to a preset prediction model and first dependent variable data corresponding to the first explanatory variable data, the first explanatory variable data and the first dependent variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no dependent variable is associated; estimating a prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first dependent variable data corresponding to the first explanatory variable data; Estimation method.

10. selecting, from first explanatory variable data prepared according to a preset prediction model and first dependent variable data corresponding to the first explanatory variable data, the first explanatory variable data and the first dependent variable data corresponding to the first explanatory variable data, based on second explanatory variable data to which no dependent variable is associated; estimating a prediction performance of the prediction model for the second explanatory variable data based on a comparison between prediction data obtained by inputting the first explanatory variable data selected for the prediction model and the first dependent variable data corresponding to the first explanatory variable data; A program that causes a computer to execute a process.