Prediction device, prediction method, and storage medium

Abstract

A prediction device 1X mainly includes: an acquisition means 24X; a prediction model execution means 25X; and a correction model execution means 27X. The acquisition means 24X acquires prediction data to be used for prediction by a prediction model. The prediction model execution means 25X acquires prediction-target prediction results output by the prediction model on the basis of the prediction data. The correction model execution means 27X acquires correction results obtained by correcting the prediction results on the basis of the prediction results, the prediction data, and a correction model that is a machine learning model.

Description

Prediction device, prediction method, and storage medium

[0001] The present disclosure relates to the technical fields of a prediction device, a prediction method, and a storage medium that perform processing related to prediction.

[0002] In a prediction model using supervised learning, prediction errors occur due to concept drift, which is a change in the characteristics of data during learning and data after operation begins. Patent Literature 1 discloses a technology for correcting the output of a prediction model using correction rules created based on the experience of a forecaster.

[0003] Japanese Patent Application Laid-Open No. 2003-346070

[0004] When factors that affect predictions and correction methods based on those factors are created in advance on a rule-based basis, as in Patent Document 1, the correction rules are created based on the experience and past performance of the person in charge. Therefore, in this case, there are problems such as high work costs, difficulty in incorporating experience and know-how that is not clearly written down or cannot be incorporated into the correction rules, and failure to consider the interactions (synergistic effects) resulting from the combination of multiple influencing factors.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide a prediction device, a prediction method, and a storage medium that are capable of suitably correcting prediction results.

[0006] One aspect of the prediction device is a prediction device comprising: an acquisition means for acquiring prediction data to be used for prediction using a prediction model; a prediction model execution means for acquiring a prediction result of a prediction target output by the prediction model based on the prediction data; and a correction model execution means for acquiring a correction result obtained by correcting the prediction result based on the prediction result, the prediction data, and a correction model that is a machine learning model, wherein the correction model is a model that has been obtained by machine learning the relationship between the prediction result, the prediction data, and the correction result.

[0007] One aspect of the prediction method is a prediction method in which a computer acquires prediction data to be used for prediction using a prediction model, acquires a prediction result of a prediction target output by the prediction model based on the prediction data, and acquires a correction result in which the prediction result is corrected based on the prediction result, the prediction data, and a correction model that is a machine learning model, and the correction model is a model that has learned by machine learning the relationship between the prediction result, the prediction data, and the correction result.

[0008] One aspect of the storage medium is a storage medium storing a program that causes a computer to execute a process of acquiring prediction data to be used for prediction using a prediction model, acquiring a prediction result of a prediction target output by the prediction model based on the prediction data, and acquiring a correction result in which the prediction result is corrected based on the prediction result, the prediction data, and a correction model that is a machine learning model, wherein the correction model is a model that has learned by machine learning the relationship between the prediction result, the prediction data, and the correction result.

[0009] One example of the effect of the present disclosure is that prediction results can be suitably corrected.

[0010] 1 shows the configuration of a prediction system. FIG. 2 shows the correspondence between record-by-record data in a database and a time axis indicating the creation date and time of the data. FIG. 3 shows the hardware configuration of a prediction device. FIG. 4 shows an example of a functional block diagram of a processor related to correction model learning processing. FIG. 5 shows a specific example of extraction processing of correction input data when the database is table data. FIG. 6 shows a coordinate space representing the distribution of each data when the database is continuous data. FIG. 7 shows a first specific example representing a set of correction input data, a predicted value output by a prediction model based on the correction input data, and a user-specified correction result. FIG. 8 shows a second specific example representing a set of correction input data, a predicted value output by a prediction model based on the correction input data, and a user-specified correction result. FIG. 9 shows an example of a flowchart of correction model learning processing executed by a prediction device. FIG. 10 shows an example of a functional block diagram of a processor related to post-learning prediction processing. FIG. 11 shows an example of a flowchart of post-learning prediction processing executed by a prediction device. FIG. 12 shows the configuration of a prediction system. FIG. 13 is a functional block diagram of a prediction device. FIG. 14 shows an example of a flowchart executed by a prediction device.

[0011] Hereinafter, embodiments of a prediction device, a prediction method, and a storage medium will be described with reference to the drawings.

[0012] <First embodiment> (1) System configuration Fig. 1 shows the configuration of a prediction system 100. The prediction system 100 mainly includes a prediction device 1, an input device 2, an output device 3, and a storage device 4. The prediction device 1 performs data communication with the input device 2, the output device 3, and the storage device 4 via a communication network or by direct wireless or wired communication.

[0013] The prediction device 1 predicts an index to be predicted based on time-series data and outputs the prediction result obtained by the prediction. In this embodiment, the prediction device 1 obtains the final prediction result to be output using a prediction model (prediction model), which is a machine learning model that outputs the prediction result, and a correction model (second model), which is a machine learning model that corrects the prediction result of the prediction model. As a result, even if the accuracy of the prediction model deteriorates due to concept drift, in which the characteristics of data at the time of learning and data after operation start change, the prediction device 1 obtains a prediction result in which the prediction result of the prediction model is suitably corrected using the correction model. Note that in this embodiment, the prediction model is a model for which machine learning has already been performed, and the correction model is a model for which machine learning has not yet been performed. As described below, the prediction device 1 generates corrective data for machine learning based on external input from a user and performs machine learning of the corrected model based on the generated corrective data.

[0014] Examples of predictions using a prediction model include demand forecasts, market forecasts, and predictions regarding diseases such as infectious diseases. Examples of indicators that can be predicted using a prediction model include indicators related to product sales, the number of patients with diseases such as infectious diseases, the number of hospitalizations, and other indicators that increase or decrease depending on the spread of the disease. For example, when predicting an indicator related to product sales, concept drift occurs in the prediction model due to the influence of factors not anticipated when the prediction model was trained (such as the release of a new product). The prediction device 1 in this embodiment trains a correction model to take into account factors not anticipated when training the prediction model, making it possible to obtain prediction results corrected by the correction model without modifying the prediction model. For example, when predicting the number of patients due to an epidemic of a new infectious disease, training a prediction model in the early stages of the epidemic can easily miss signs at the start of the epidemic due to a lack of diagnoses and information, resulting in low accuracy. The prediction device 1 in this embodiment trains a correction model to take into account newly acquired information after training (i.e., after the epidemic has begun), making it possible to obtain prediction results corrected by the correction model without modifying the prediction model.

[0015] The input device 2 is an interface that accepts user input, which is external input, and corresponds to, for example, a touch panel, buttons, a keyboard, a voice input device, etc. The input device 2 supplies input information generated based on the user input to the prediction device 1.

[0016] The output device 3 outputs information based on the control of the prediction device 1. Examples of the output device 3 include display devices such as a display and a projector, and sound output devices such as a speaker. The output device 3 displays information and / or outputs sound based on the output information supplied from the prediction device 1.

[0017] The storage device 4 is a memory that stores various information necessary for prediction calculations. The storage device 4 may be a storage device such as a hard disk connected to or built into the prediction device 1, or may be a storage medium such as a flash memory. The storage device 4 may also be a server device that performs data communication with the prediction device 1. In this case, the storage device 4 may be composed of multiple server devices.

[0018] The storage device 4 stores at least a database 40 , prediction model information 41 , and correction model information 42 .

[0019] Database 40 is a database that stores records that associate input data, which is time-series data used for predictions by the prediction model, with correct answer data (including actual values) that indicate the correct answer that the prediction model should output based on the input data.

[0020] FIG. 2 is a diagram clearly showing the correspondence between record-by-record data contained in database 40 and a time axis indicating the creation date and time of the data. Database 40 includes learning input data, learning correct answer data, post-learning acquired input data, and prediction target actual value. The learning input data and learning correct answer data are learning data (training data) used in machine learning of the prediction model. The post-learning acquired input data is input data obtained after learning of the prediction model, from operation start time "t1" when operation using the prediction model began until the current time "t2", and the prediction target actual value indicates the actual value of the indicator to be predicted obtained from operation start time t1 to the current time t2. The prediction target actual value is used to determine whether additional learning of the correction model is necessary.

[0021] 2, input data obtained after the current time t2 and predicted values ​​of the indicators to be predicted and corrected by the correction model are each indicated by a dashed frame. In this way, the correction model after machine learning corrects the predicted values ​​calculated by the prediction model based on the input data acquired after learning, etc., so as to reduce the prediction error of the predicted values ​​indicated by the dashed frame. To obtain such a correction model, machine learning is performed on the correction model based on data in database 40 obtained up to the current time t2 and correct data obtained by external input by the user.

[0022] The prediction model information 41 includes trained parameters of the prediction model. Here, the prediction model is a model (engine) that, when input data is input to the model, outputs a prediction result of a prediction target based on the input data. The prediction model is, for example, a machine learning model (including a statistical model; the same applies hereinafter) having any architecture, such as a neural network or a support vector machine. In this case, the prediction model is a model that has learned the relationship between the input data and the predicted value of the prediction target based on the input data. The prediction model is trained using training input data and training correct answer data contained in the database 40, and the parameters of the prediction model obtained by training are pre-stored in the prediction model information 41. When the prediction model is configured using a neural network, the prediction model information 41 includes various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weights of each element of each filter.

[0023] The correction model information 42 includes trained parameters of the correction model. Here, the correction model is a model (engine) that outputs a correction result of the prediction result when input data and a prediction result of a prediction model based on the input data are input to the model. Note that supplementary data, described below, may also be input to the correction model. The correction result may be a correction value for correcting the prediction result (predicted value), or may be a corrected prediction result (i.e., the corrected predicted value). The correction model is, for example, a machine learning model having any architecture, such as a neural network or a support vector machine. In this case, the correction model is a model that has learned the relationship between data including a predicted value of a prediction target input to the correction model and the correction result of the predicted value. Machine learning of the correction model is performed by the prediction device 1 using the database 40, and parameters of the correction model obtained by the machine learning are stored in the correction model information 42. Note that the correction model information 42 before the correction model is trained may also store initial values ​​of the correction model (including parameters related to the architecture of the correction model).

[0024] The storage device 4 may store various information necessary for the processing executed by the prediction device 1, in addition to the database 40, the prediction model information 41, and the correction model information 42. For example, the storage device 4 may store supplementary data, which is data to be input to the correction model. The supplementary data is additional parameters (variables, etc.) that are not used in training the prediction model or in predictions using the prediction model, but are found to affect the prediction after training the prediction model. For example, in product demand forecasting, the supplementary data may be information about a new product other than the product whose demand is being forecast, or information about a new store that has been built after training.

[0025] The configuration of the prediction system 100 shown in FIG. 1 is an example, and various modifications may be made to the configuration. For example, the input device 2 and the output device 3 may be configured as an integrated device. In this case, the input device 2 and the output device 3 may be configured as a tablet terminal integrated with the prediction device 1. The prediction device 1 may also be configured as a plurality of devices. In this case, the plurality of devices that make up the prediction device 1 exchange information necessary to execute pre-assigned processing between these plurality of devices.

[0026] (2) Hardware Configuration of the Prediction Device Fig. 3 shows the hardware configuration of the prediction device 1. The prediction device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 99.

[0027] The processor 11 executes predetermined processes by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0028] The memory 12 is composed of various types of volatile and non-volatile memories, such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory 12 also stores programs for the prediction device 1 to execute various processes. The memory 12 is also used as a working memory, and temporarily stores information obtained from the storage device 4. The memory 12 may also function as the storage device 4. Similarly, the storage device 4 may also function as the memory 12 of the prediction device 1. The programs executed by the prediction device 1 may be stored in a storage medium other than the memory 12.

[0029] The interface 13 is an interface for electrically connecting the prediction device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0030] The hardware configuration of the prediction device 1 is not limited to the configuration shown in Fig. 3. For example, the prediction device 1 may include at least one of an input device 2 or an output device 3. Furthermore, the prediction device 1 may be connected to or built into a sound output device such as a speaker.

[0031] (3) Correction Model Learning Process First, the process related to learning the correction model (correction model learning process) will be described.

[0032] (3-1) Functional Blocks Figure 4 is an example of a functional block diagram of the processor 11 related to the correction model learning process. Functionally, the processor 11 has a data extraction unit 14, a prediction model execution unit 15, a correction input unit 16, a correction model learning unit 17, an output integration unit 18, and a correction evaluation unit 19. Note that in Figure 4, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to that shown in Figure 4. The same applies to other functional block diagrams described later.

[0033] The data extraction unit 14 extracts input data to be corrected (also referred to as "correction input data") from the database 40 on a record-by-record basis. Hereinafter, all data registered in the database 40 will be referred to as "X," and the correction input data will be referred to as "X'." The correction input data X' is, for example, data specified by the user using the input device 2 or the like from among the post-learning acquired input data shown in FIG. 2 . In this case, the data extraction unit 14 displays an input screen for specifying the correction input data X' on the output device 3, receives input information indicating the data specified on the input screen from the input device 2, and extracts the correction input data X' from the database 40 based on the received input information. In this case, the input information supplied from the input device 2 is an example of "external input by the user" and "input related to extraction of correction data." Details of the processing performed by the data extraction unit 14 will be described later. The data extraction unit 14 supplies the correction input data X' to the prediction model execution unit 15 and the correction model learning unit 17, respectively.

[0034] The prediction model executing unit 15 calculates a predicted value of the prediction target based on the correction input data X' supplied from the data extracting unit 14 and the prediction model constructed with reference to the prediction model information 41. The prediction model executing unit 15 inputs the correction input data X' to the prediction model, and obtains the prediction result output by the prediction model as the above-mentioned predicted value. Hereinafter, the prediction model will be referred to as a function "f １ ", and the predicted value output by the prediction model using the correction input data X' as input is expressed as "Y = f １ The prediction model executing unit 15 supplies the calculated prediction value Y to the correction model learning unit 17 and the output integrating unit 18.

[0035] The correction input unit 16 generates a corrected result (corrected predicted value) that should be output by the prediction model when the correction input data X' is input, based on input information supplied from the input device 2 in response to a user operation. Then, the correction input unit 16 supplies the correction result specified by the user to the correction model learning unit 17 as corrected data that should be output by the prediction model. In this case, the input information supplied from the input device 2 is an example of an "external input by a user" and an "input related to corrected data."

[0036] Hereinafter, the correction result will be denoted as "Y'", and the correction term corresponding to the difference between the user-specified correction result Y' and the predicted value Y will be denoted as "λ". In this case, the user-specified correction result Y' is expressed by the following formula: Y'=Y+λ

[0037] In addition to the input related to the correction result, the correction input unit 16 receives input information related to the reliability of the correction result from the input device 2 based on user operation. In this case, the input information supplied from the input device 2 is an example of an "external input by the user" and an "input related to the confidence level of the correct data." The confidence level corresponds to the user's subjective confidence in the correction result specified by the user, and is used to integrate the prediction result and the correction result by the output integration unit 18, which will be described later. With this configuration, the prediction device 1 can accept a direct specification of the correction result Y' based on the experience of the person in charge, and can also adjust the weight of the correction based on the confidence level.

[0038] The correction model learning unit 17 uses the correction input data X' supplied from the data extraction unit 14, the correction result Y' supplied from the correction input unit 16, the predicted value Y supplied from the prediction model execution unit 15, and the supplementary data "X' obtained from the storage device 4, etc. Ｓ " and a correction model is learned based on the function "f ２ " should be written as ".

[0039] Supplementary Data X Ｓare additional variables that are not used in training the prediction model or in predictions made by the prediction model, but are found to affect the prediction after training the prediction model. For example, when forecasting product demand, supplementary data X that indicates the features of the new product and its similarity to existing products is used. Ｓ By using this, it is possible to train a correction model to output correction results that take into account the features of the new product and the similarity with the conventional product. Ｓ The use of is not mandatory.

[0040] For example, the correction model learning unit 17 calculates the correction input data X′, the predicted value Y, and the supplementary data X as shown in the following equation: Ｓ The correction model is trained so that the output when the correction model is executed with arguments f and f becomes the correction result Y' specified by the user. ２ (X', Y, X Ｓ ) → Y'

[0041] For example, the correction model learning unit 17 sequentially extracts records of the correction input data X′ and updates the parameters of the correction model for each extracted record. Ｓ , and data based on the predicted value Y to the correction model, and the parameters of the correction model are determined so that the error (loss) between the correction result output by the correction model and the user-specified correction result is minimized. The algorithm for determining the above parameters so as to minimize the loss may be any learning algorithm used in machine learning, such as gradient descent or backpropagation. Furthermore, the data input to the correction model may be formatted to match the input format of the correction model (e.g., a predetermined tensor format). The correction model training unit 17 then repeats the above-described parameter update for each record of the correction input data X'. The correction model training unit 17 then stores the correction model parameters obtained by training in the correction model information 42.

[0042] Furthermore, the correction model learning unit 17 supplies the correction result Y' obtained by executing the correction model to the output integration unit 18 for correction evaluation by the correction evaluation unit 19. In this case, the correction model learning unit 17 may extract, from the database 40 or the like, input data for verification that is different from the correction input data X' used to learn the correction model, and calculate the correction result Y' based on the extracted input data. In this case, the correction model learning unit 17 obtains the correction result Y' obtained from the correction model using the input data for verification and a predicted value Y predicted using the input data for verification. In this case, the predicted value Y supplied from the prediction model execution unit 15 to the output integration unit 18 becomes the predicted result output from the prediction model by inputting the input data for verification into the prediction model.

[0043] The output integrating unit 18 calculates a predicted value (also referred to as an "integrated predicted value") by integrating the predicted value Y supplied from the prediction model executing unit 15 and the correction result Y' supplied from the correction model learning unit 17, and updates the integrated predicted value as the final predicted value Y. Here, the algorithm used by the output integrating unit 18 for the above-mentioned integration is defined as a function "f ｏｕｔ ", the output integration unit 18 executes the process corresponding to the following equation: Y ← f ｏｕｔ (Y, Y')

[0044] Preferably, the output integrating unit 18 performs the above-described integration process based on the user-specified correction result used in learning the correction model and the user-specified confidence acquired by the correction input unit 16. When performing integration using a weighted linear sum, the output integrating unit 18 determines a weight based on the user-specified confidence. For example, the output integrating unit 18 sets a value obtained by normalizing the confidence so that it falls within the range of 0 to 1 as the weight "w", and sets the final predicted value Y as follows: Y ← f ｏｕｔ (Y, Y') = (1 - w) Y + wY', where w is set to a higher value as the confidence level of the user-specified correction result increases. The output integration unit 18 then supplies the final predicted value Y to the correction evaluation unit 19.

[0045] The correction evaluation unit 19 compares the final predicted value Y supplied from the output integration unit 18 with the predicted target actual value "Y" stored in the database 40 in association with the correction input data X' used for learning the correction model. ｔｒｕｅ " is compared with the result of the comparison, the correction evaluation unit 19 determines whether or not additional learning of the correction model is necessary (i.e., whether or not additional acquisition of the user-specified correction result is necessary). Then, the correction evaluation unit 19 displays and / or outputs by voice information on the determination result of the necessity on the output device 3.

[0046] For example, the correction evaluation unit 19 calculates the final predicted value Y and the corresponding predicted actual value Y ｔｒｕｅ If the error between and is less than a predetermined threshold, the correction evaluation unit 19 determines that additional learning of the correction model is not necessary, and outputs information that the learning of the correction model has been completed via the output device 3. On the other hand, if the error is equal to or greater than the threshold, the correction evaluation unit 19 determines that additional learning of the correction model is necessary, and outputs information that prompts the user to input (designation of data to be extracted by the data extraction unit 14, external input of the correction result and confidence factor acquired by the correction input unit 16) necessary for the additional learning of the correction model via the output device 3. In this way, the correction evaluation unit 19 evaluates the difference between the predicted value based on the correction result and the actual value, thereby enabling sequential modification of the correction model and an instruction to end the correction.

[0047] Each of the components described in FIG. 4 (the data extraction unit 14, the prediction model execution unit 15, the correction input unit 16, the correction model learning unit 17, the output integration unit 18, and the correction evaluation unit 19) can be realized, for example, by the processor 11 executing a program. Alternatively, each component may be realized by recording the necessary program on an arbitrary non-volatile storage medium and installing it as needed. Note that at least a portion of each of these components is not limited to being realized by software programs, but may also be realized by any combination of hardware, firmware, and software. Furthermore, at least a portion of each of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program consisting of each of the above components may be realized using this integrated circuit. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0048] (3-2) Specific Examples Here, specific examples of the processing executed by the data extraction unit 14 will be described.

[0049] FIG. 5 is a diagram showing a specific example of the process of extracting correction input data X' by the data extraction unit 14 when the database 40 is table data X. In this example, the table data X has columns for "day of the week," "product," "store," and "sales promotion," and has multiple records. The data extraction unit 14 then acquires from the input device 2 user input specifying the search criterion "weekend" for "day of the week," the search criterion "A, B, or C" for "store," and the search criterion "sunscreen" for "product." The data extraction unit 14 then extracts, as correction input data X', records from the database 40 that satisfy all of the above search conditions.

[0050] The data extraction unit 14 may suggest to the user data to be designated as correction input data X' based on the relevance of the data. For example, for a specific column, records having a field similar to a field (also referred to as an element or cell) that matches the search criteria specified by the user may be extracted and displayed as candidates to be additionally extracted as correction input data X'. For example, for the "product" column, the data extraction unit 14 may also extract records of products (e.g., "skin care products" not shown) similar to the product "sunscreen" that matches the search criteria specified by the user as candidates for correction input data X' and present these records as candidates to be additionally designated as correction input data X'. Then, when the data extraction unit 14 receives user input specifying that the presented candidate should be included in the correction input data X', the data extraction unit 14 includes the specified candidate in the correction input data X'.

[0051] 6 shows a coordinate space (feature space) that represents the distribution of each piece of data in each record when each field (element) in the database 40 is continuous data (numeric data). This coordinate space has a number of dimensions corresponding to the number of elements (number of fields) that each piece of data in each record has, and each piece of data in each record is plotted as a data point in the coordinate space.

[0052] In this case, the data extraction unit 14, for example, accepts user input specifying a data point corresponding to one or more records and determines the data point specified by the user input as a representative point. The data extraction unit 14 then recognizes the representative point and other data points similar to the representative point as data points representing the correction input data X'. In the example of FIG. 6 , the data extraction unit 14 recognizes data points present within the dashed-line frame (here, within a predetermined distance from the representative point) as the correction input data X'. In this case, the data extraction unit 14 recognizes other similar data points using any index (e.g., distance) representing similarity, such as Euclidean distance or KL distance. For example, if the index value representing the similarity between any representative point and other data points is a value indicating a similarity greater than a predetermined threshold, the data represented by the other data points is included in the correction input data X'.

[0053] In addition, even if the database 40 contains data other than continuous data such as text, the data extraction unit 14 may digitize the data by encoding (i.e., converting it into a feature vector) using any natural language processing model such as Word2vec, and then perform the above-mentioned processing.

[0054] Next, a specific example of the process executed by the correction input unit 16 will be described.

[0055] FIG. 7 shows a first specific example of a set of correction input data X', a predicted value Y output by a prediction model based on the correction input data X', and a correction result Y' designated by a user.

[0056] In the first specific example, a group of correction input data X', each containing four records, is input to the prediction model for each record, and the predicted sales value Y output by the prediction model is shown for each record. Furthermore, for each record of the correction input data X', the user-specified correction result Y' is a value obtained by increasing the predicted value Y using a correction term λ of 10% of each sales amount, and the user-specified confidence level is set to "0.8." In this case, the correction model learning unit 17 updates the parameters of the correction model using the correction input data X' and the corresponding predicted value Y and correction result Y'. Furthermore, the output integration unit 18 integrates the predicted value Y and correction result Y' using a weighted linear sum using the confidence level "0.8" as a weight.

[0057] FIG. 8 shows a second specific example of a set of correction input data X', a predicted value Y output by a prediction model based on the correction input data X', and a correction result Y' designated by a user.

[0058] In the second specific example, a group of correction input data X' containing two records is input to the prediction model for each record, and the predicted sales value Y output by the prediction model is shown for each record. The user-specified correction result Y' is increased by a normal distribution with parameters of mean μ and variance σ (μ and σ are arbitrary values ​​specified by the user), and the user-specified confidence level is set to "0.5." In this case, the correction model learning unit 17 updates the correction model parameters using the correction input data X' and the corresponding predicted value Y and correction result Y'. The output integration unit 18 integrates the predicted value Y and correction result Y' using a weighted linear sum using the confidence level of 0.5 as a weight.

[0059] In addition, when a user specifies different correction terms λ and confidence factors for each group of records of the correction input data X′, the correction model training unit 17 may train a correction model as a different correction model for each group. For example, the correction model training unit 17 trains one correction model using four records of the correction input data X′ shown as a first specific example, and trains another correction model using two records of the correction input data X′ shown as a second specific example.

[0060] Note that instead of training a correction model for each group of correction input data X' having different specified certainties, one correction model may be trained using all of the correction input data X'. In this case, when there are multiple certainties corresponding to the trained correction model, the prediction device 1 may calculate a representative value, such as an average value of the certainties, and use this representative value of the certainty as a weight for integrating the predicted value Y and the correction result Y'. Similarly, even when a different correction term λ is specified for each group of correction input data X', the prediction device 1 may train one correction model using all of the correction input data X'.

[0061] (3-3) Processing Flow Fig. 9 is an example of a flowchart of the correction model learning process executed by the prediction device 1. The prediction device 1 executes the process of the flowchart, for example, when a predetermined start condition for the correction model learning process is satisfied. For example, when the prediction device 1 receives an input instructing the start of the correction model learning process, the prediction device 1 determines that the start condition is satisfied and executes the process of the flowchart.

[0062] First, the prediction device 1 extracts correction input data X' from the database 40 (step S11). Next, the prediction device 1 obtains a prediction result of a prediction model using the correction input data X' (step S12). In this case, the prediction device 1 inputs the correction input data X' to a prediction model configured based on the prediction model information 41, thereby obtaining a predicted value Y output by the prediction model. The prediction device 1 may sequentially input record-by-record data of the correction input data X' to the prediction model and obtain a predicted value Y for each record from the prediction model, or may input all of the correction input data X' to the prediction model and obtain a single predicted value Y from the prediction model.

[0063] Next, the prediction device 1 acquires the user-specified correction result Y′ and its confidence level (step S13). The prediction device 1 acquires the user-specified correction result Y′ for the predicted value Y and its confidence level based on input information supplied from the input device 2 operated by the user.

[0064] The prediction device 1 then learns a correction model based on the correction input data X', the prediction result of the prediction model, the supplementary data, and the user-specified correction result (step S14). In this case, the prediction device 1 updates the parameters of the correction model so as to minimize the error between the correction result output by the correction model when data based on the correction input data X', the prediction result of the prediction model, and the supplementary data is input to the correction model and the user-specified correction result.

[0065] Next, the prediction device 1 integrates the prediction result of the prediction model and the correction result of the correction model based on the confidence level obtained in step S13, and outputs the integrated prediction value as the final prediction value via the output device 3 (step S15).

[0066] In addition, the prediction device 1 compares the final predicted value, which is the result of integrating the predicted values ​​in step S15, with the corresponding actual value of the prediction target, and outputs information suggesting the continuation or termination of learning of the correction model via the output device 3 (step S16).

[0067] (4) Post-Learning Prediction Process Next, a prediction process using a correction model that has been learned through the correction model learning process (also referred to as a "post-learning prediction process") will be described.

[0068] 10 is an example of a functional block diagram of the processor 11 related to the post-learning prediction process. Functionally, the processor 11 includes an input data acquisition unit 24, a prediction model execution unit 25, a correction model execution unit 27, an output integration unit 28, and a prediction result output unit 29.

[0069] The input data acquiring unit 24 acquires input data (also referred to as "input data for prediction") to be used in the post-learning prediction process from the database 40. In this case, the input data acquiring unit 24 may acquire all data recorded in the database 40 as input data for prediction, or may acquire data that satisfies predetermined conditions (e.g., data generated within a predetermined period) from the database 40 as input data for prediction. The input data acquiring unit 24 supplies the acquired input data for prediction to the prediction model executing unit 25 and the correction model executing unit 27, respectively.

[0070] The prediction model executing unit 25 generates a prediction result indicating a predicted value of a prediction target based on a prediction model constructed with reference to the prediction model information 41 and the input data for prediction supplied from the input data acquiring unit 24. In this case, the prediction model executing unit 25 acquires a prediction result output by the prediction model by inputting the input data for prediction into the prediction model. The prediction model executing unit 25 then supplies the generated prediction result to the correction model executing unit 27 and the output integrating unit 28, respectively.

[0071] The correction model executing unit 27 generates a correction result by correcting the prediction result based on the input data for prediction supplied from the input data acquiring unit 24, the prediction result supplied from the prediction model executing unit 25, the supplementary data, and a trained correction model constructed with reference to the correction model information 42. In this case, the correction model executing unit 27 acquires the correction result output by the correction model when data based on the input data for prediction, the prediction result, and the supplementary data is input to the correction model. The data input to the correction model may be formatted to match the input format of the correction model (e.g., a predetermined tensor format), for example. The supplementary data is the same as the supplementary data used in the correction model training process and is stored, for example, in the storage device 4 or memory 12. Note that if the supplementary data was not used in the correction model training process, the supplementary data is not used in the post-training prediction process either. The correction model executing unit 27 supplies the correction result output by the correction model to the output integrating unit 28.

[0072] The output integrating unit 28 calculates an integrated predicted value by integrating the prediction result supplied from the prediction model executing unit 25 and the correction result supplied from the correction model executing unit 27, determines the integrated predicted value as the final predicted value, and supplies the final predicted value to the prediction result output unit 29. In this case, preferably, the output integrating unit 28 sets a weight based on the confidence level specified by external input from the user in the machine learning of the correction model used, and sets the final predicted value by a weighted linear sum or the like. In this case, for example, like the output integrating unit 18, the output integrating unit 28 sets a value obtained by normalizing the confidence level from 0 to 1 as the weight w, and integrates the predicted value Y and the correction result Y' using the following formula: (1-w)Y+wY'

[0073] The prediction result output unit 29 displays and / or outputs information about the final predicted value supplied from the output integration unit 28 on the output device 3. As a result, the prediction result output unit 29 can present to the user the prediction result of the prediction target corrected using the trained correction model. That is, even if the accuracy of the prediction model deteriorates due to concept drift, which is a change in the characteristics of data at the time of learning and data after operation has begun, the prediction result output unit 29 can present to the user the prediction result in which the prediction result based on the prediction model has been appropriately corrected using the correction model. For example, in the case of demand forecasting, the prediction result output unit 29 can present to the user the demand forecast result in which the prediction error has been corrected using a correction model that corrects the prediction result output by the prediction model. As a result, the prediction result output unit 29 can appropriately support the user's decision-making (e.g., the market supply volume of a product) based on the predicted demand. Furthermore, in the case of disease prediction, the prediction result in which the prediction error has been corrected using a correction model that corrects the prediction result output by the prediction model can be presented to the user, i.e., a medical professional such as a nurse or a doctor. This allows the prediction result output unit 29 to suitably support the user in making decisions (such as the number of hospital beds to be secured) according to the predicted situation.

[0074] 11 is an example of a flowchart of post-learning prediction processing executed by the prediction device 1. The prediction device 1 executes the processing of the flowchart, for example, when a predetermined start condition for the post-learning prediction processing is satisfied. For example, when the prediction device 1 receives an input instructing the start of the post-learning prediction processing, the prediction device 1 determines that the start condition is satisfied and executes the processing of the flowchart.

[0075] First, the prediction device 1 extracts input data for prediction from the database 40 (step S21). Then, the prediction device 1 obtains a prediction result of a prediction model using the input data for prediction (step S22). In this case, the prediction device 1 inputs the input data for prediction into a prediction model configured using prediction model information 41, and obtains a prediction result output by the prediction model.

[0076] Next, the prediction device 1 acquires a correction result by the correction model based on the input data for prediction acquired in step S21, the prediction result by the prediction model acquired in step S22, and the supplementary data (step S23). In this case, the prediction device 1 configures a trained correction model by referring to the correction model information 42, and acquires the correction result output by the correction model by inputting the input data for prediction, the prediction result by the prediction model, and the supplementary data into the correction model.

[0077] Next, the prediction device 1 integrates the prediction result of the prediction model acquired in step S22 and the correction result of the correction model acquired in step S23 based on the confidence level specified by the user in the model learning process, and outputs the integrated result as a final predicted value via the output device 3 (step S24). Outputting such information can favorably support the user's decision-making. The final predicted value output in step S24 is an example of "information based on the correction result."

[0078] 12 shows the configuration of a prediction system 100A. The prediction system 100A mainly includes a prediction device 1A and a terminal device 5. The prediction device 1A and the terminal device 5 perform data communication via a network 6.

[0079] The prediction device 1A is one or more devices that function as a server (including a cloud server), and performs the model learning process and post-learning prediction process executed by the prediction device 1 in the first embodiment. In this case, the prediction device 1A receives input information from the terminal device 5 via the network 6, which the prediction device 1 receives from the input device 2 in the first embodiment. The prediction device 1A also transmits output information that the prediction device 1 transmitted to the output device 3 in the first embodiment to the terminal device 5 via the network 6. The prediction device 1A also stores the information that the storage device 4 stores in the first embodiment.

[0080] The terminal device 5 is a terminal having input, display, and communication functions, and functions as the input device 2 and the output device 3 in the first embodiment. The terminal device 5 may be, for example, a personal computer, a tablet terminal, a PDA (Personal Digital Assistant), or the like. The terminal device 5 transmits input information generated based on the received user input to the prediction device 1A via the network 6. Furthermore, when the terminal device 5 receives output information from the prediction device 1A, it outputs information regarding the final prediction result, etc., based on the output information.

[0081] The prediction device 1A according to the second embodiment can preferably perform the input process and output process that the prediction device 1 according to the first embodiment performs on the user of the terminal device 5 .

[0082] 13 is a functional block diagram of a prediction device 1X. The prediction device 1X mainly includes an acquisition unit 24X, a prediction model execution unit 25X, and a correction model execution unit 27X. The prediction device 1X may be composed of multiple devices.

[0083] The acquisition unit 24X acquires prediction data to be used for prediction using a prediction model. The acquisition unit 24X can be, for example, the input data acquisition unit 24 in the first or second embodiment.

[0084] The prediction model execution means 25X acquires a prediction result of a prediction target output by a prediction model based on prediction data. The prediction model execution means 25X may be, for example, the prediction model execution unit 25 in the first or second embodiment.

[0085] The correction model executing means 27X obtains a correction result obtained by correcting the prediction result based on the prediction result, the prediction data, and a correction model that is a machine learning model. Here, the correction model is a model that has been obtained by machine learning the relationship between the prediction result, the prediction data, and the correction result. The correction model executing means 27X can be, for example, the correction model executing unit 27 in the first or second embodiment.

[0086] 14 is an example of a flowchart executed by the prediction device 1X. The acquisition unit 24X acquires prediction data to be used for prediction using the prediction model (step S31). The prediction model execution unit 25X acquires a prediction result of the prediction target output by the prediction model based on the prediction data (step S32). The correction model execution unit 27X acquires a correction result obtained by correcting the prediction result based on the prediction result, the prediction data, and a correction model, which is a machine learning model (step S33).

[0087] The prediction device 1X according to the third embodiment can obtain a correction result in which a prediction result by a prediction model is corrected by a correction model that is a machine learning model. This makes it possible to obtain a correction result in which the prediction error is corrected even if a prediction error occurs in the prediction model.

[0088] In addition, part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0089] [Supplementary Note 1] A prediction device comprising: an acquisition means for acquiring prediction data to be used for prediction using a prediction model; a prediction model execution means for acquiring a prediction result of a prediction target output by the prediction model based on the prediction data; and a correction model execution means for acquiring a correction result obtained by correcting the prediction result based on the prediction result, the prediction data, and a correction model that is a machine learning model, wherein the correction model is a model obtained by machine learning to determine the relationship between the prediction result, the prediction data, and the correction result. [Supplementary Note 2] The prediction device according to Supplementary Note 1, wherein the correction model execution means acquires the correction result based on the prediction result, the prediction data, the correction model, and supplementary data that affects the prediction and was not considered in generating the prediction model, and the correction model is a model obtained by machine learning to determine the relationship between the prediction result, the prediction data, the supplementary data, and the correction result. [Supplementary Note 3] The prediction device according to Supplementary Note 1, further comprising a prediction result output means for supporting a user's decision-making by displaying or outputting audio information based on the correction result. [Supplementary Note 4] The prediction device according to Supplementary Note 1, comprising: a correction input means for accepting an external input from a user regarding correction of the prediction result using the correction model; and a correction model learning means for generating corrective data for machine learning to be output by the correction model based on the external input, and for performing machine learning of the correction model based on the corrective data. [Supplementary Note 5] The prediction device according to Supplementary Note 4, wherein the external input includes an input regarding the corrective data and an input regarding a confidence level of the corrective data, and further comprising an integration means for generating a final prediction result by integrating the prediction result and the correction result based on the confidence level of the corrective data. [Supplementary Note 6] The prediction device according to Supplementary Note 4, further comprising data extraction means for extracting correction data to be used in machine learning of the correction model from a database used for the prediction. [Supplementary Note 7] The prediction device according to Supplementary Note 6, wherein the external input includes an input regarding the corrective data and an input regarding extraction of the correction data, and the data extraction means extracts the correction data from the database based on the input regarding the extraction.[Supplementary Note 8] The prediction device according to Supplementary Note 7, wherein the data extraction means extracts representative data designated based on input related to the extraction and data similar to the representative data from the database. [Supplementary Note 9] The prediction device according to Supplementary Note 4, further comprising correction evaluation means for determining whether additional machine learning of the correction model is necessary, based on the correction result and an actual value of the prediction object corresponding to the correction result. [Supplementary Note 10] A prediction method, wherein a computer: acquires prediction data to be used for prediction using a prediction model; acquires a prediction result of the prediction object output by the prediction model based on the prediction data; and acquires a correction result in which the prediction result is corrected based on the prediction result, the prediction data, and a correction model that is a machine learning model, wherein the correction model is a model obtained by machine learning of the relationship between the prediction result, the prediction data, and the correction result. [Supplementary Note 11] A storage medium storing a program that causes a computer to execute a process of acquiring prediction data to be used for prediction by a prediction model, acquiring a prediction result of a prediction target output by the prediction model based on the prediction data, and acquiring a correction result in which the prediction result is corrected based on the prediction result, the prediction data, and a correction model that is a machine learning model, wherein the correction model is a model that has learned by machine learning the relationship between the prediction result, the prediction data, and the correction result.

[0090] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0091] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0092] 1, 1A, 1X Prediction device 2 Input device 3 Output device 4 Storage device 5 Terminal device 100, 100A Prediction system

Claims

1. A prediction device comprising: an acquisition means for acquiring prediction data to be used for prediction by a prediction model; a prediction model execution means for acquiring a prediction result of a prediction target output by the prediction model based on the prediction data; and a correction model execution means for acquiring a correction result in which the prediction result is corrected based on the prediction result, the prediction data, and a correction model which is a machine learning model, wherein the correction model is a model which has been machine learned to determine the relationship between the prediction result, the prediction data, and the correction result.

2. The prediction device of claim 1, wherein the correction model execution means obtains the correction result based on the prediction result, the prediction data, the correction model, and supplementary data that affects the prediction and was not taken into account in generating the prediction model, and the correction model is a machine-learned model of the relationship between the prediction result, the prediction data, the supplementary data, and the correction result.

3. The prediction device according to claim 1, further comprising a prediction result output means for supporting a user's decision-making by displaying or outputting by voice information based on the correction result.

4. A prediction device as described in claim 1, comprising: a correction input means for receiving external input from a user regarding correction of the prediction result by the correction model; and a correction model learning means for generating corrective data for machine learning to be output by the correction model based on the external input, and for performing machine learning of the correction model based on the corrective data.

5. The prediction device according to claim 4, wherein the external input includes an input relating to the correct data and an input relating to a degree of certainty of the correct data, and further comprises an integration means for generating a final prediction result by integrating the prediction result and the correction result based on the degree of certainty of the correct data.

6. The prediction device according to claim 4, further comprising a data extraction means for extracting correction data used in machine learning of the correction model from a database used for the prediction.

7. The prediction device described in claim 6, wherein the external input includes an input regarding the correct data and an input regarding the extraction of the correction data, and the data extraction means extracts the correction data from the database based on the input regarding the extraction.

8. A prediction device according to claim 7, wherein said data extraction means extracts representative data designated based on an input relating to said extraction, and data similar to said representative data, from said database.

9. A prediction device as described in claim 4, further comprising a correction evaluation means for determining whether or not additional machine learning of the correction model is necessary based on the correction result and the actual value of the prediction target corresponding to the correction result.

10. A prediction method, comprising: a computer acquiring prediction data to be used for prediction using a prediction model; acquiring a prediction result of a prediction target output by the prediction model based on the prediction data; acquiring a correction result in which the prediction result is corrected based on the prediction result, the prediction data, and a correction model which is a machine learning model; and the correction model being a model which has learned by machine learning the relationship between the prediction result, the prediction data, and the correction result.

11. A storage medium having stored therein a program, which causes a computer to execute a process of acquiring prediction data to be used for prediction using a prediction model, acquiring a prediction result of a prediction target output by the prediction model based on the prediction data, and acquiring a correction result in which the prediction result is corrected based on the prediction result, the prediction data, and a correction model which is a machine learning model, wherein the correction model is a model which has been machine-learned to determine the relationship between the prediction result, the prediction data, and the correction result.

Images