Prediction device, prediction method, and prediction program

Abstract

To provide a technique for performing ensemble prediction at low cost.SOLUTION: A prediction device includes management means for managing a model pool including a first model and one or a plurality of second models that generate one or a plurality of modulated outputs with reference to an output of the first model, and prediction means for performing a prediction process using a plurality of models included in the model pool.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a prediction device, a prediction method, and a prediction program. [Background technology]

[0002] There are known techniques for ensemble prediction using a model that combines multiple models. For example, Patent Literature 1 describes a method for generating multiple ensemble models using supervised machine learning operations. In this method, multiple ensemble models are generated from one or more datasets generated by combining one or more clusters of data points from a minority class with selected data points from a majority class. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-527366 Summary of the Invention [Problem to be solved by the invention]

[0004] The method described in Patent Document 1 has a problem in that it generates multiple ensemble models using supervised machine learning operations, which results in high costs.

[0005] The present disclosure has been made in view of the above-mentioned problems, and an exemplary purpose thereof is to provide a technique for performing ensemble prediction at low cost. [Means for solving the problem]

[0006] A prediction device according to an exemplary aspect of the present disclosure includes a management means for managing a model pool including a first model and one or more second models that generate one or more modulated outputs by referencing an output of the first model, and a prediction means for performing a prediction process using the multiple models included in the model pool.

[0007] A prediction method according to an exemplary aspect of the present disclosure includes a management process in which at least one processor manages a model pool including a first model and one or more second models that generate one or more modulated outputs by referencing an output of the first model, and a prediction process in which the at least one processor performs a prediction process using multiple models included in the model pool.

[0008] A prediction program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as a prediction device, and causes the computer to function as: a management means that manages a model pool that includes a first model and one or more second models that generate one or more modulated outputs by referencing the output of the first model; and a prediction means that executes a prediction process using multiple models included in the model pool. [Effects of the Invention]

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technology for performing ensemble prediction at low cost can be provided. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of a prediction device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of a prediction method according to the present disclosure. [Figure 3] FIG. 1 is a block diagram illustrating a configuration of a prediction device according to the present disclosure. [Figure 4] FIG. 1 is a flow diagram showing the flow of a prediction method according to the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating a display example according to the present disclosure. [Figure 6] FIG. 10 is a diagram illustrating a display example according to the present disclosure. [Figure 7] FIG. 1 is a schematic diagram illustrating a specific example of a prediction device applied in the medical field according to the present disclosure. [Figure 8] FIG. 1 is a schematic diagram illustrating a specific example of a prediction device applied in the retail field according to the present disclosure. [Figure 9] FIG. 1 is a block diagram illustrating a configuration of a prediction device according to the present disclosure. [Figure 10] FIG. 10 is a flow diagram illustrating an example of a detailed flow of a weight update process according to the present disclosure. [Figure 11] FIG. 10 is a flow diagram illustrating an example of a detailed flow of a prediction process according to the present disclosure. [Figure 12] FIG. 10 is a diagram illustrating a display example according to the present disclosure. [Figure 13] FIG. 10 is a diagram illustrating a display example according to the present disclosure. [Figure 14] FIG. 1 is a block diagram illustrating a configuration of a prediction device according to the present disclosure. [Figure 15] FIG. 10 is a flow diagram illustrating an example of a detailed flow of a weight update process according to the present disclosure. [Figure 16] FIG. 10 is a flow diagram illustrating an example of a detailed flow of a prediction process according to the present disclosure. [Figure 17] FIG. 10 is a diagram illustrating a display example according to the present disclosure. [Figure 18] FIG. 1 is a block diagram illustrating a configuration of a computer that functions as a prediction device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] First Exemplary Embodiment A first exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referred to in describing this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise.

[0013] (Configuration of prediction device) The configuration of the prediction device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the prediction device 1. As shown in Fig. 1, the prediction device 1 includes a management unit 13 and a prediction unit 12. In this exemplary embodiment, the management unit 13 and the prediction unit 12 respectively realize a management means and a prediction means.

[0014] The management unit 13 manages a model pool including a first model and one or more second models that refer to the output of the first model to generate one or more modulated outputs.

[0015] The process by which the management unit 13 manages the model pool includes, for example, A model generation process for generating a second model that generates a modulated output by referring to the output of the first model. Model addition process that adds the generated second model to the model pool Here, the model generation process may generate a plurality of second models that generate a plurality of modulated outputs different from each other for a certain output of the first model.

[0016] For example, when the function representing the first model or the output of the function is represented as f1, the management unit 13 performs the following in the model generation process: A second model, f2, takes f1 as input and outputs f1+10 A second model, f3, takes f1 as input and outputs f1-10. In this way, a plurality of second models different from each other may be generated.

[0017] The first model is a model that performs prediction, and is, for example, a machine learning model generated by machine learning. Examples of machine learning models include, but are not limited to, deep neural networks (DNNs), gradient boosting decision trees (GBDTs), and linear regression models.

[0018] The prediction unit 12 executes a prediction process using a plurality of models included in the model pool. As an example, the prediction unit 12 generates a prediction result by referring to the output of each of the plurality of models included in the model pool and the weight of each of the plurality of models.

[0019] (Effect of prediction device) As described above, the prediction device 1 employs a configuration including a management unit 13 that manages a model pool including a first model and one or more second models that generate one or more modulated outputs by referring to the output of the first model, and a prediction unit 12 that executes prediction processing using the multiple models included in the model pool.

[0020] Therefore, the prediction device 1 generates a variety of second models without training and executes prediction processing using a model pool including the second models. Therefore, the prediction device 1 has the effect of being able to perform ensemble prediction at low cost.

[0021] (Example of program implementation) When the prediction device 1 is configured by a computer including at least one processor and a memory, the memory stores the following prediction program. The prediction program causes a computer to function as the prediction device 1, and causes the computer to function as a management unit 13 that manages a model pool including a first model and one or more second models that generate one or more modulated outputs by referring to the output of the first model, and a prediction unit 12 that executes prediction processing using the multiple models included in the model pool.

[0022] (Prediction method flow) The flow of the prediction method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the prediction method S1. As shown in Fig. 2, the prediction method S1 includes a management process S13 and a prediction process S12.

[0023] In management processing S13, at least one processor manages a model pool including a first model and one or more second models that reference the output of the first model to generate one or more modulated outputs.

[0024] In the prediction process S12, at least one processor executes the prediction process using a plurality of models included in the model pool.

[0025] (Effect of prediction method S1) As described above, the prediction method S1 employs a configuration in which at least one processor includes a management process S13 for managing a model pool including a first model and one or more second models that generate one or more modulated outputs by referencing the output of the first model, and a prediction process S12 for executing a prediction process using the multiple models included in the model pool.

[0026] Therefore, according to the prediction method S1, the same effects as those of the prediction device 1 described above can be obtained.

[0027] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0028] (Configuration of prediction device 1A) The configuration of the prediction device 1A will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the prediction device 1A. As shown in Fig. 3, the prediction device 1A includes a weight update unit 11, a prediction unit 12, a model management unit 13 (corresponding to the management unit 13 in the exemplary embodiment described above), a weight storage unit 14, a display information generation unit 15, and an input / output unit 16. In this exemplary embodiment, the weight update unit 11, the prediction unit 12, the model management unit 13, and the display information generation unit 15 respectively realize an update means, a prediction means, a management means, and a display information generation means.

[0029] (Model Management Section 13) The model management unit 13 manages a model pool MP. As shown in Fig. 3, the model pool MP includes Nmodel models f_1, f_2, ..., f_Nmodel. Nmodel is a natural number equal to or greater than 2. In other words, the model pool MP includes a model set F expressed by the following formula (1).

[0030]

number

[0031] In the following, the model f_1 will be referred to as the first model, and the models other than the model f_1 (model f_2, ..., model f_Nmodel) will be referred to as the second model. Furthermore, the model management unit 13 includes a model generation unit 131 and a model addition unit 132, as shown in FIG.

[0032] (Model generation unit 131) The model generation unit 131 executes a model generation process to generate models f_2, ..., model f_Nmodel that generate modulated outputs by referring to the output of model f_1. Note that the model generation unit 131 can also be expressed as generating multiple modulated outputs by referring to the output of model f_1.

[0033] As an example of a model generated by the model generating unit 131, when the model f_1 is a regression model, a model that outputs a value obtained by adding an offset to the output value f1 of the model f_1 is given below. An example of the model is given below. · Model f_2 output value f2=f1+10 · Model f_3 output value f3=f1-10 Model f_4 output value f4=f1+20 Model f_5 output value f5=f1-20 Another example of a model generated by the model generation unit 131 is a model that outputs a result obtained by exchanging any two or more certainties in the output result of a model f_1 that outputs an output value and certainty (prediction probability) of the output value. An example of such a model is shown below. In the following, when the model f_1 outputs an output value f with a prediction probability pp1, o1 , the output value f with a prediction probability pp2 lower than the prediction probability pp1 o2 , and the output value f with a predicted probability pp3 lower than the predicted probability pp2 o3 Hereinafter, the prediction probabilities will be referred to as "first highest prediction probability," "second highest prediction probability," etc., in descending order of prediction probability. Model f_2: Output value f of predicted probability pp2 o1 , output value f with predicted probability pp1 o2 , output value f with predicted probability pp3 o3 Output Model f_3: Output value f of predicted probability pp3 o1 , output value f with predicted probability pp2 o2 , output value f with predicted probability pp1 o3 Output Model f_4: Output value f of predicted probability pp1 o1 , output value f with predicted probability pp3 o2 , output value f with predicted probability pp2 o3 Output For example, if model f_1 is a classification model and has a classification probability pp1 for class 1 and a classification probability pp2 for class 2, the model generation unit 131 generates a model f_2 that outputs a classification probability pp2 for class 1 and a classification probability pp1 for class 2.

[0034] The modulated output output by the second model is not limited to the above example, and may be any modulated output that allows interpretation of the value of the modulated output. For example, the second model may be a model that generates a modulated output using a polynomial function that takes the output of the first model as an argument. For example, the second model may be a model that generates a modulated output by squaring the output of the first model, or a model that generates a modulated output by cubed the output of the first model.

[0035] (Model Addition Section 132) The model adding unit 132 executes a model adding process for adding the second model generated by the model generating unit 131 to the model pool MP.

[0036] (Weight storage unit 14) A weight vector w is stored in the weight storage unit 14. The weight vector w is expressed by the following equation (2).

[0037]

number

[0038] Weight w i is a weight assigned to the model f_i. The weight vector w is updated by the weight update unit 11, and the initial value of each element is determined arbitrarily. For example, the initial values ​​of each element may all be the same or may be determined randomly.

[0039] (Prediction Section 12) The prediction unit 12 executes a prediction process by inputting prediction target information related to a prediction target into a model f_1. The prediction target information is also called an explanatory variable. The prediction unit 12 supplies a prediction result to the display information generation unit 15.

[0040] The prediction target is a target for which prediction is performed using each model f_i, and is, for example, sales amount, hospital bed occupancy rate, classification of human behavior, etc., but is not limited to these. The prediction target is also called, for example, a dependent variable. Furthermore, prediction target information is information input to model f_1, and is also called an explanatory variable. When the prediction target is sales amount on a target day, the prediction target information may include, for example, the weather on the target day. Furthermore, when the prediction target is a hospital bed occupancy rate one week from now, the prediction target information may include the most recent hospital bed occupancy rate. Furthermore, when the prediction target is a classification of human behavior, the prediction target information may include an image of the person.

[0041] The prediction unit 12 also calculates the output of each of the multiple models f_i included in the model pool MP and the weights w i For regression tasks, the output when the prediction target information is input to the model f_i is called f i Then, the prediction result is expressed by the following equation (3).

[0042]

number

[0043] In other words, the prediction results are calculated by the output f i Weights about w i The prediction result is also called the predicted value of the ensemble prediction. In addition, the weight w in Equation (3) i is the normalized value. The unnormalized weight w i In this case, we use the weight w i Just divide by the sum of the above.

[0044] For classification tasks, for example, i represents a vector of dimension equal to the number of classes, and f i The predicted probability for each class label i is expressed by: Then, the prediction unit 12 can calculate the predicted probability after integration using an equation similar to the above equation (3). When finally determining a label as a predicted value, the prediction unit 12 determines the class label with the highest probability as the label as the predicted value.

[0045] For example, a case will be described in which the prediction target is the sales amount on a target day, and the prediction target information includes store surrounding images taken around the store on the target day and weekday / holiday labels indicating weekdays or holidays. In this case, the prediction unit 12 inputs the prediction target information to a model f_1, and thereby predicts the output value f1 of the model f_1, the output value f2 of a model f_2 that referenced the output value f1, ..., the output value f of a model f_Nmodel that referenced the output value f1. NmodelThen, the prediction unit 12 calculates the sales amount, which is the prediction result, using the above-mentioned formula (3).

[0046] (Weight update unit 11) The weight update unit 11 updates the weights. For example, the weight update unit 11 updates the weights w i Update.

[0047] As an example, the weight update unit 11 refers to the evaluation information to determine the weight w i The evaluation information is information obtained over time after the start of operation, and is different from the prediction target information. The evaluation information may be prediction target information that the prediction unit 12 has referred to in past prediction processes, but is not limited to this. The evaluation information may also include a true value corresponding to the prediction result. In this case, the weight update unit 11 updates the weight w by comparing the prediction result predicted by each model with reference to the prediction target information with the true value included in the evaluation information. i Update.

[0048] For example, the weight update unit 11 updates the weight w based on a plurality of pieces of evaluation information and the evaluation results of the performance of each model f_i with reference to the plurality of pieces of evaluation information. i The evaluation result of the performance of each model f_i with reference to a plurality of pieces of evaluation information may be, for example, a statistical value (e.g., average value, maximum value, minimum value, etc.) of the evaluation result of the performance of each model f_i with reference to each piece of evaluation information.

[0049] (Display information generation unit 15) The display information generating unit 15 generates display information. For example, the display information generating unit 15 generates display information by generating weights w i , and the weight w i The display information generating unit 15 generates display information including at least one of the pieces of information obtained from the above. The display information generating unit 15 supplies the generated display information to the input / output unit 16. Examples of the display information generated by the display information generating unit 15 will be described later.

[0050] (Input / output section 16) The input / output unit 16 is configured to include at least one of an input / output device such as a keyboard, a mouse, a display, a printer, or a touch panel. Alternatively, the input / output unit 16 may be configured to have an input / output device such as a keyboard, a mouse, a display, a printer, or a touch panel connected to it. In this configuration, the input / output unit 16 accepts various types of information input to the prediction device 1A from the connected input device. The input / output unit 16 also outputs various types of information to the connected output device. An interface such as a USB (Universal Serial Bus) may be adopted as the input / output unit 16. As an example, the input / output unit 16 displays the display information generated by the display information generation unit 15. In other words, the input / output unit 16 also has a configuration as a display unit that displays the display information.

[0051] (Flow of prediction method S1A) The flow of the prediction method S1A executed by the prediction device 1A will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the prediction method S1A.

[0052] (Management process S13) In the management process S13, the model management unit 13 manages a model pool MP including a model f_1 and f_2, ..., f_Nmodel that generates one or more modulated outputs by referring to the output of the model f_1. More specifically, in the management process S13, the model management unit 13 executes the following model generation process S131 and model addition process S132.

[0053] (Model generation process S131) In the model generation process S131, the model generation unit 131 executes a model generation process to generate a model f_2, ..., model f_Nmodel, which is a second model that generates a modulated output by referring to the output of the model f_1, which is a first model. The method by which the model generation unit 131 generates the model f_2, ..., model f_Nmodel that generates a modulated output is as described above.

[0054] (Model addition process S132) In the model addition process S132, the model adding unit 132 adds the second models, ie, models f_2, . . . , model f_Nmodel, to the model pool MP.

[0055] (Weight update process S11) In the weight update process S11, the weight update unit 11 updates the weights w i More specifically, in the weight update process S11, the weight update unit 11 executes an evaluation information acquisition process S111 and an update process S112.

[0056] (Evaluation information acquisition process S111) In the evaluation information acquisition process S111, the weight update unit 11 acquires evaluation information.

[0057] (Update process S112) In the update process S112, the weight update unit 11 refers to the evaluation information and updates the weights w i The weight update unit 11 updates the weight w i The method for updating is as described above.

[0058] (Prediction process S12) In the prediction process S12, the prediction unit 12 executes the prediction process by inputting prediction target information related to the prediction target into the model f_1. More specifically, in the prediction process S12, the prediction unit 12 executes a prediction target information acquisition process S121 and a calculation process S122.

[0059] (Prediction target information acquisition process S121) In the prediction target information acquisition process S121, the prediction unit 12 acquires prediction target information.

[0060] (Calculation process S122) In the calculation process S122, the prediction unit 12 inputs the prediction target information into the model f_1 to calculate a prediction result. As described above, the prediction unit 12 calculates the output f of each model. iWeights about w i The prediction result is calculated as a weighted linear sum using

[0061] (Display information generation process S15) In the display information generation process S15, the display information generation unit 15 generates display information. The display information generation unit 15 supplies the display information to the input / output unit 16. The input / output unit 16 displays the display information. Examples of the display information displayed by the input / output unit 16 will be described with reference to Figs. 5 and 6. Fig. 5 is a diagram showing display example A1 and display example A2. Fig. 6 is a diagram showing display example A3 and display example A4.

[0062] As described above, the display information generating unit 15 calculates the weights w of the models included in the model pool MP. i As an example, the display information generating unit 15 generates display information for displaying a graph with time on the horizontal axis and weight on the vertical axis, as shown in display example A1 of FIG. 5. Display example A1 shows that a weight w1 for the output f1 of model f_1, a weight w2 for the output f2 of model f_2, and a weight w3 for the output f3 of model f_3 change over time. In other words, display example A1 visualizes the time series of the weights. For example, display example A1 shows that after a certain time has passed, the value of weight w1 decreases, the value of weight w2 increases, and the value of weight w3 does not change significantly over time.

[0063] With this configuration, the display information generating unit 15 can present to the user what kind of distribution changes have occurred over time and which models from the model pool MP are being used.

[0064] The display information may also include modulation information indicating how a modulated output is generated by a second model among the multiple models included in the model pool MP. For example, in display example A1, the display information includes modulation information RI indicating how a modulated output f2 is generated by the second model, model f_2 (f2=f1+10). In other words, the modulation information can also be referred to as relationship information indicating the relationship between the output of the first model and the output of the second model.

[0065] As described above, the modulated output is a modulated output that can be interpreted as to what value the modulated output is. For example, the second model may be a model that generates the modulated output using a polynomial function that takes the output of the first model as an argument. Therefore, by including the modulation information in the display information, the display information generator 15 can present to the user what models are included in the model pool MP and the relationship between the first model f_1 and the second model.

[0066] As described above, the display information generating unit 15 also uses the weight w i As an example, as shown in a display example A2 in FIG. 5, the display information generating unit 15 generates display information including information obtained from the weight w i As an example, the display information generating unit 15 calculates the shift degree using the following equation (6).

[0067]

number

[0068] That is, the display information generating unit 15 calculates a weighted average of the offsets as the shift degree. i is the normalized value. The unnormalized weight w i In this case, we use the weight w i Just divide by the sum of the above.

[0069] With this configuration, the display information generating unit 15 can present to the user what kind of distribution changes have occurred over time.

[0070] The display information generating unit 15 may generate display information for displaying a bar plot. For example, as shown in a display example A3 in FIG. 6, the display information generating unit 15 may generate display information for displaying a bar plot. i Generate display information for displaying the values ​​of in a bar plot. Display example A3 shows that when t=1 changes to t=2, the weight w1 for the output f1 of model f_1 and the weight w2 for the output f2 of model f_2 change, but the weight w3 for the output f3 does not change significantly.

[0071] Even in this configuration, the display information generation unit 15 can present to the user what distribution changes have occurred over time and which models from the model pool MP are being used.

[0072] Also, as in the above example, the display information may include modulation information RI indicating that the output f2 after modulation by model f_2 is the result of swapping the first and second places in the predicted probability of the output f1 of model f_1, as shown in display example A3.

[0073] Even in this configuration, the display information generating unit 15 can present to the user what kind of models are included in the model pool MP and the relationship between the first model f_1 and the second model.

[0074] In addition, the display information generating unit 15 uses the weight w i The display information generating unit 15 may generate display information for displaying the flip degree, which is information obtained from the above, as shown in display example A4 of Fig. 6. As an example, the display information generating unit 15 calculates the flip degree using the following equation (7).

[0075]

number

[0076] That is, the display information generating unit 15 uses the weight w of the second model as the flip degree. i The display information generating unit 15 may generate display information that displays the time series of the flip degree. i is the normalized value. The unnormalized weight w i In this case, we use the weight w i Just divide by the sum of the above.

[0077] Even in this configuration, the display information generating unit 15 can present to the user how much the distribution has changed since the first model f_1 was learned.

[0078] (Effect of prediction device 1A) The prediction device 1A configured as described above generates various second models (model f_2, ..., model f_Nmodel) without training, similar to the prediction device 1 according to the first exemplary embodiment, and executes prediction processing using a model pool MP including the second models. Therefore, the prediction device 1A has the advantage of being able to perform ensemble prediction at low cost.

[0079] (Application example 1) The following describes application examples of the above-described prediction devices 1 and 1 A. In the following description, application examples of the prediction device 1A will be described, but application examples of the prediction device 1 can also be realized in the same way.

[0080] For example, the prediction device 1A can be applied in the medical field. A specific example of the prediction device 1A applied in the medical field will be described with reference to FIG. 7. FIG. 7 is a schematic diagram showing a specific example of the prediction device 1A applied in the medical field. In this example, the prediction target is "bed occupancy rate one week from now," and the prediction target information is "weather, temperature, day of the week, disease name, latest bed occupancy rate, holiday / weekday label." Here, information about the weather is acquired from a weather providing server via an API (Application Programming Interface) through communication via a communication unit (not shown) provided in the prediction device 1A. Furthermore, information about temperature is acquired from a temperature sensor connected to the prediction device 1A via the input / output unit 16.

[0081] Furthermore, models f_1, f_2, . . . , f_Nmodel are stored in the prediction device 1A. The model f_1 is generated in advance by a learning algorithm using training data.

[0082] Then, the prediction device 1A executes the management process S13, generates models f_2, . . . , model f_Nmodel that generate modulated outputs by referring to the output of the model f_1, and adds them to the model pool MP.

[0083] Furthermore, the prediction device 1A repeats the update process S11, the prediction process S12, and the display information generation process S15. The prediction result derived by the prediction device 1A, "bed occupancy rate one week later," is input to a reservation management system connected to the prediction device 1A and is referenced for optimizing the number of hospital beds to be reserved. With this configuration, the prediction device 1A can perform ensemble prediction with high accuracy even when the distribution of information related to the prediction target changes locally.

[0084] (Application example 2) As another example, the prediction device 1A can be applied to demand forecasting. A specific example of the prediction device 1A applied to demand forecasting will be described with reference to FIG. 8. FIG. 8 is a schematic diagram showing a specific example of the prediction device 1A applied to demand forecasting. In this example, the prediction target is "next day's sales," and the prediction target information is "weather, temperature, day of the week, product classification, moving average, and holiday / weekday label." Here, information about the weather is obtained from a weather providing server via an API (Application Programming Interface) through communication via a communication unit (not shown) provided in the prediction device 1A. Furthermore, information about temperature is obtained from a temperature sensor connected to the prediction device 1A via the input / output unit 16.

[0085] Furthermore, models f_1, f_2, . . . , f_Nmodel are stored in the prediction device 1A. The model f_1 is generated in advance by a learning algorithm using training data.

[0086] Then, the prediction device 1A executes the management process S13, generates models f_2, . . . , model f_Nmodel that generate modulated outputs by referring to the output of the model f_1, and adds them to the model pool MP.

[0087] Furthermore, the prediction device 1A repeats the update process S11, the prediction process S12, and the display information generation process S15. The prediction result "next day's sales" derived by the prediction device 1A is input to a reservation management system connected to the prediction device 1A and is referenced for optimizing the number of orders. With this configuration, ensemble prediction can be performed with high accuracy even when the distribution of information related to the prediction target changes locally.

[0088] Third Exemplary Embodiment A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0089] (Outline of Prediction Device 1B) In addition to the configuration of the prediction device 1 and the prediction device 1A, the prediction device 1B has a configuration for updating some or all of the multiple weight vectors w_i based on the evaluation results obtained by evaluating the performance of each model f_i by referring to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models f_i, and the evaluation information.

[0090] Furthermore, the prediction device 1B outputs a prediction result obtained by integrating the prediction results predicted by each model f_i by referring to model input information included in the prediction target information related to the prediction target, using a weight vector w_i selected from multiple weight vectors w_i based on the prediction target information.

[0091] Here, the weight vector w_i selected based on the prediction target information is not limited to one, and may be multiple. When multiple weight vectors w_i are selected, an integrated weight vector obtained by integrating the selected multiple weight vectors w_i may be used to integrate the prediction results of each model.

[0092] (Configuration of prediction device 1B) The configuration of the prediction device 1B will be described with reference to Fig. 9. Fig. 9 is a block diagram showing the configuration of the prediction device 1B. As shown in Fig. 9, the prediction device 1B includes a weight update unit 11, a prediction unit 12, a model management unit 13, a weight storage unit 14, a display information generation unit 15, and an input / output unit 16. In this exemplary embodiment, the weight update unit 11, the prediction unit 12, the model management unit 13, and the display information generation unit 15 respectively realize an update means, a prediction means, a management means, and a display information generation means.

[0093] The model management unit 13 and the input / output unit 16 have the same configuration as in the above-described exemplary embodiment, and therefore a description thereof will be omitted.

[0094] (Weight storage unit 14) The weight storage unit 14 is configured similarly to the exemplary embodiment described above, and is also configured as follows: The weight storage unit 14 stores Nweight weight vectors w_1, w_2, ..., w_Nweight, where Nweight is a natural number equal to or greater than 2. In other words, the weight storage unit 14 stores a weight set W expressed by the following equation (8).

[0095]

number

[0096] The weight vector w_j is a vector having Nmodel weights w_j_i as elements, and is expressed by the following equation (9).

[0097]

number

[0098] The weight w_j_i represents the weight assigned to the model f_i when the weight vector w_j is selected. The weight vector w_j may be updated by the weight update unit 11, and the initial value of each element is determined arbitrarily. For example, the initial values ​​of each element may all be equal or may be determined randomly.

[0099] Here, each of the multiple weight vectors w_j stored in the weight storage unit 14 is associated with at least one of the multiple conditions that the prediction target information may satisfy and that the evaluation information may satisfy. For example, each of the multiple conditions c_j may be a condition that the model input information may satisfy. As an example, if the model input information includes a weekday / holiday label, the multiple conditions c_1 and c_2 may be set as condition c_1 "weekday" and condition c_2 "holiday." Furthermore, if the evaluation information and the prediction target information include additional information, each of the multiple conditions c_j may be a condition that the additional information may satisfy, or may be a condition that both the model input information and the additional information may satisfy.

[0100] The additional information is information included in the prediction target information and the evaluation information, and is not input to the model f_1. For example, when the model input information includes a store surrounding image, examples of the additional information include the resolution of the store surrounding image, the time of shooting, the type of shooting device, the photographer, or a combination thereof. However, examples of the model input information and the additional information are not limited to these. Note that the prediction target information and the evaluation information only need to include at least the model input information, and do not necessarily need to include the additional information. However, when multiple conditions, which will be described later, are determined with reference to the additional information, both the prediction target information and the evaluation information include the model input information and the additional information.

[0101] Furthermore, although the weight vectors and conditions do not necessarily correspond one-to-one, an example of one-to-one correspondence will be mainly described here, and a condition that corresponds one-to-one with a weight vector w_j will be referred to as a condition c_j. In other words, the weight vector w_j is associated with the condition c_j. When the weight vector w_j and the condition c_j correspond one-to-one, the number of the multiple conditions c_j is equal to the number Nweight of weight vectors.

[0102] (Prediction Section 12) The prediction unit 12 is configured in the same manner as the above-described exemplary embodiment, and in addition, outputs a prediction result obtained by integrating the prediction results predicted by each of the multiple models f_i by referring to model input information included in prediction target information related to the prediction target, using a weight vector w_j selected from the multiple weight vectors w_j based on the prediction target information.

[0103] More specifically, the prediction unit 12 selects a weight vector w_j associated with a condition c_j satisfied by the prediction target information from among a plurality of conditions c_j. The prediction unit 12 also outputs a prediction result obtained by integrating prediction results y_i output from each of a plurality of models f_i for model input information included in the prediction target information using the selected weight vector w_j. For example, the process of calculating the integrated prediction result is expressed by the following equation (10).

[0104]

number

[0105] In equation (10), the left side (hereinafter referred to as y^) indicates the integrated prediction result. x indicates the model input information included in the prediction target information. f_i(x) indicates the prediction result y_i by model f_i. w_j_i is the element corresponding to model f_i among the elements of weight vector w_j associated with condition c_j satisfied by the model input information.

[0106] In addition, in equation (10), if the condition c_j(x) is True, that is, if x satisfies the condition c_j, the prediction unit 12 calculates an integrated prediction result using the weight w_j corresponding to the condition c_j. In addition, the weight w_j is a normalized value.

[0107] Furthermore, when additional information is included in the prediction target information, the prediction unit 12 may select one of the weight vectors w_j based on the model input information and the additional information included in the prediction target information. In this case, w_j_i in equation (10) may be an element corresponding to the model f_i among elements of the weight vector w_j associated with the condition c_j satisfied by one or both of the model input information and the additional information.

[0108] For example, an example will be described in which the prediction target information includes store surroundings images and weekday / holiday labels as model input information, and the resolution of the store surroundings images as additional information. In this example, condition c_1 may be "weekday and high resolution," condition c_2 may be "weekday and low resolution," condition c_3 may be "holiday and high resolution," and condition c_4 may be "holiday and low resolution." In this case, the number of weight vectors Nweight may be 4, which is the number of conditions.

[0109] Furthermore, for example, a set of prediction target information may be input to the prediction unit 12. Such a set X is expressed by the following equation (11).

[0110]

number

[0111] In equation (11), x_k indicates the k-th model input information included in the N input pieces of prediction target information. When additional information is included in the prediction target information, equation (11) can be similarly explained by replacing "x_k" with "x_k, v_k." v_k indicates the additional information included in the k-th prediction target information.

[0112] In this case, the prediction unit 12 outputs a set Y of integrated prediction results corresponding to the set X. Such a set Y is expressed by the following equation (12).

[0113]

number

[0114] In equation (12), y^_k represents the integrated prediction result corresponding to the k-th prediction target information.

[0115] (Weight update unit 11) In addition to being configured in the same manner as the exemplary embodiment described above, the weight update unit 11 updates some or all of the multiple weight vectors w_j based on the evaluation results obtained by evaluating the performance of each of the multiple models f_i included in the model pool MP by referring to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models f_i.

[0116] More specifically, the weight update unit 11 updates the weight vector w_j associated with the condition c_j that is satisfied by the evaluation information among the multiple conditions c_j. For example, if the evaluation model input information included in the evaluation information includes a weekday / holiday label indicating a weekday, the evaluation information satisfies the condition c_1 "weekday." Therefore, the weight update unit 11 updates the weight vector w_1 associated with the condition c_1.

[0117] Here, a case will be described in which model input information included in prediction target information referenced by the prediction unit 12 in a past prediction process is applied as the model input information for evaluation to be included in the evaluation information. For example, the weight update unit 11 may use model input information referenced by the prediction unit 12 in the past (e.g., store surroundings image and weekday / holiday label for a target day) as model input information for evaluation in response to obtaining a true value (e.g., actual sales value for a target day) corresponding to the model input information. In this case, the weight update unit 11 may acquire the evaluation information including the model input information for evaluation and the true value. However, the model input information for evaluation to be included in the evaluation information is not limited to the above-mentioned example as long as it is information corresponding to the elapsed time since the start of operation of the multiple models f_i.

[0118] Furthermore, the weight update unit 11 may update some or all of the multiple weight vectors w_j based on multiple pieces of evaluation information and evaluation results of the performance of each model f_i with reference to the multiple pieces of evaluation information. The evaluation results of the performance of each model f_i with reference to the multiple pieces of evaluation information may be, for example, statistical values ​​(e.g., average value, maximum value, minimum value, etc.) of the evaluation results of the performance of each model f_i with reference to each piece of evaluation information.

[0119] In addition, if the evaluation information includes additional evaluation information, the weight update unit 11 may update some or all of the multiple weight vectors w_k based on the evaluation results of the performance of each model f_i using the evaluation information, as well as the evaluation model input information and additional evaluation information included in the evaluation information.

[0120] For example, the set of evaluation information Deval input to the weight update unit 11 is expressed by the following equation (13).

[0121]

number

[0122] In equation (13), x_l indicates the model input information included in the l-th evaluation information among the Neval pieces of evaluation information, and y_l indicates the true value included in the l-th evaluation information. When the evaluation information includes additional evaluation information, equation (13) can be similarly explained by replacing "x_l, y_l" with "x_l, y_l, v_l." v_l indicates the additional evaluation information included in the l-th evaluation information.

[0123] For example, the weight update unit 11 may extract, for at least one of the multiple conditions c_j, one or more pieces of evaluation information that satisfy the condition c_j from the multiple pieces of evaluation information. The process of extracting one or more pieces of evaluation information that satisfy the condition c_j is expressed, for example, by the following equation (14).

[0124]

number

[0125] In equation (14), the left side (hereinafter also referred to as Deval_j) represents a set of evaluation information that is a subset of Deval and satisfies the condition c_j.

[0126] Furthermore, the weight update unit 11 updates the weight vector w_j associated with the condition c_j based on the evaluation results obtained by evaluating the performance of each model f_i using one or more pieces of extracted evaluation information (Deval_j). When the evaluation information includes additional evaluation information, equation (14) can be similarly explained by replacing c_j(x) with c_j(x,v). c_j(x,v) is true if the evaluation model input information x and the additional evaluation information v included in the evaluation information satisfy the condition c_j, and false if they do not.

[0127] For example, suppose the set Deval contains five pieces of evaluation information, three of which contain the weekday / holiday label "weekday," and the remaining two contain the weekday / holiday label "holiday." In this case, the first three pieces of evaluation information that satisfy the condition c_1 "weekday" are extracted as subset Deval_1. The latter two pieces of evaluation information that satisfy the condition c_2 "holiday" are extracted as subset Deval_2.

[0128] For example, when the evaluation result of the performance of each model f_i is expressed as a numerical value, the weight update unit 11 may directly adopt the numerical value of the evaluation result of the performance as the element w_j_i of the weight vector w_j to be updated. This process is expressed, for example, by the following equation (15).

[0129]

number

[0130] The right side of equation (15) represents the evaluation result obtained by evaluating the performance of each model f_i using the above-mentioned subset Deval_j. As described above, the evaluation result may be, for example, a statistical value of the evaluation result obtained by evaluating the model f_i using each piece of evaluation information included in the subset Deval_j.

[0131] As a specific example, suppose the multiple pieces of evaluation information include evaluation information that satisfies the condition c_1 "weekday" and evaluation information that satisfies the condition c_2 "holiday." In this case, the weight update unit 11 may update the evaluation result, obtained by evaluating the performance of each model f_i with reference to a subset Deval_1 of the evaluation information that satisfies the condition c_1 "weekday," as an element w_1_i of the weight vector w_1 associated with the condition c_1. Furthermore, the weight update unit 11 may update the evaluation result, obtained by evaluating the performance of each model f_i with reference to a subset Deval_2 of the evaluation information that satisfies the condition c_2 "holiday," as an element w_2_i of the weight vector w_2 associated with the condition c_2.

[0132] (Display information generation unit 15) The display information generator 15 is configured similarly to the exemplary embodiment described above, and is also configured as follows. The display information generator 15 generates display information including at least one of a plurality of conditions c_j, weights w_j_i of a plurality of models f_i in each of the plurality of conditions c_j, and information obtained from the weights w_j_i. Examples of the display information generated by the display information generator 15 will be described later.

[0133] (Prediction method flow) The prediction method S1B executed by the prediction device 1B configured as above will be described in substantially the same manner as the prediction method S1A described with reference to Fig. 4. However, details of the weight update process S11, the prediction process S12, and the display information generation process S15 are different.

[0134] (Detailed flow of weight update process) First, a detailed flow of the weight update process S11 will be described with reference to Fig. 10. Fig. 10 is a flow diagram illustrating an example of a detailed flow of the weight update process S11. As shown in Fig. 10, the weight update process S11 includes steps S111B to S115B.

[0135] In step S111B, the weight update unit 11 acquires a set Deval of evaluation information. For example, when the prediction method S1B has been executed in the past, the weight update unit 11 may acquire evaluation information including, as model input information for evaluation, model input information included in the prediction target information used in the past prediction process S12. Furthermore, when the number of unprocessed evaluation information pieces among the acquired evaluation information reaches a predetermined number, the weight update unit 11 may execute the processing from the next step S112B onwards using the set Deval of the predetermined number of evaluation information pieces. Note that the set Deval of evaluation information is only required to include at least one piece of evaluation information, and is not limited to including multiple pieces of evaluation information. An example of the set Deval is as described with reference to formula (13).

[0136] In step S112B, the weight update unit 11 extracts a subset Deval_j of evaluation information that satisfies a certain condition c_j from the set Deval. An example of the subset Deval_j of evaluation information has been described with reference to equation (14).

[0137] In step S113B, the weight update unit 11 evaluates the performance of each model f_i using the subset Deval_j of the evaluation information.

[0138] In step S114B, the weight update unit 11 updates the weight vector w_j associated with the condition c_j based on the evaluation result of the performance of each model f_i using the subset Deval_j. An example of the process of updating the weight vector w_j has been described with reference to equation (15).

[0139] In step S115B, the weight update unit 11 determines whether or not evaluation information satisfying other conditions c_j that have not yet been processed is included in the set of evaluation information Deval. If evaluation information satisfying other conditions c_j is included (Yes in step S115B), the processing from step S112B is repeated. If evaluation information satisfying other conditions c_j is not included, the weight update processing S11 ends.

[0140] Then, the next prediction process S12 is executed using the weight vectors w_j that have been partially or entirely updated in the weight update process S11.

[0141] (Detailed flow of prediction process) Next, details of the prediction process S12 will be described with reference to Fig. 11. Fig. 11 is a flow diagram showing an example of the detailed flow of the prediction process S12. As shown in Fig. 11, the prediction process S12 includes steps S121B to S126B.

[0142] In step S121B, the prediction unit 12 acquires a set X of prediction target information. The prediction unit 12 may acquire the set X of prediction target information stored in a memory included in the prediction device 1B, or may acquire the set X of prediction target information received via a network. The set X of prediction target information may include at least one piece of prediction target information, and is not limited to including a plurality of pieces of prediction target information. An example of the set X is as described with reference to formula (11).

[0143] In step S122B, the prediction unit 12 selects a weight vector w_j associated with the condition c_j satisfied by the information to be predicted.

[0144] In step S123B, the prediction unit 12 inputs the model input information included in the prediction target information to each model f_i, thereby obtaining a prediction result y_i output from each model f_i.

[0145] In step S124B, the prediction unit 12 calculates the prediction result y^ by integrating the prediction results y_i of each model f_i using the selected weight vector w_j. An example of the calculation process of y^ has been described with reference to equation (10).

[0146] In step S125B, the prediction unit 12 determines whether the set X includes other prediction target information for which an integrated prediction result has not yet been calculated. If other prediction target information is included (Yes in step S125B), the processing from step S122B is repeated for the other prediction target information. If other prediction target information is not included, the processing in the next step S126B is executed.

[0147] In step S126B, the prediction unit 12 outputs a set Y of integrated prediction results corresponding to each prediction target information. An example of the set Y has been described with reference to equation (12). Note that the next weight update process S11 may be performed using evaluation information that includes model input information included in the prediction target information used in the prediction process S12 as model input information for evaluation. However, as in exemplary embodiment 2, the weight update process S11 and the prediction process S12 may be performed independently of each other, and the order and timing of execution of each process are not specified.

[0148] (Example of display information generation process S15) Examples of display information generated by the display information generating unit 15 when it executes the display information generating process S15 will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a diagram showing display examples B1 to B3. Fig. 13 is a diagram showing display example B4.

[0149] As described above, the display information generating unit 15 generates display information including a plurality of conditions c_j. As an example, in the case of condition c_1 "sunny on a holiday," condition c_2 "rainy on a holiday," condition c_3 "sunny on a weekday," and condition c_4 "rainy on a weekday," the display information generating unit 15 generates display information including a list of conditions c_1 to c_4, as shown in display example B1 in FIG. 12 .

[0150] The input / output unit 16 may also be configured to display a cursor CSR that can be operated by the user, allowing the user to select a condition. In this configuration, the display information generation unit 15 may be configured to generate display information including additional information to be presented to the user based on an input from the user. Examples of the additional information include weights w_j_i of multiple models f_i in the selected condition c_j and information obtained from the weights w_j_i. For example, the display information generation unit 15 generates display information including the shift degree of the above-described display example A2 and the flip degree of the above-described display example A4 as additional information.

[0151] For example, as shown in display example B1, when the user operates the cursor CSR and selects condition c_4 "raining on a weekday", the display information generation unit 15 generates display information including the weights w_j_i of each of the multiple models f_i in the selected condition c_4 "raining on a weekday" and information obtained from the weights w_j_i.

[0152] As another example, the display information generating unit 15 generates display information including a tree structure indicating the branching rules of conditions c_1 to c_4, as shown in display example B2 of Fig. 12. In display example B2, condition c_j is displayed in each of "leaf 1" to "leaf 4" at the lowest level.

[0153] Similarly, in display example B2, for example, when the user operates the cursor CSR and selects the condition "leaf 4", the display information generation unit 15 generates display information including the weights w_j_i of each of the multiple models f_i in the selected condition "leaf 4" and information obtained from the weights w_j_i.

[0154] As yet another example, the display information generation unit 15 generates display information including information obtained by dividing conditions c_1 to c_4 in the feature space, as shown in display example B3 in Fig. 12. In display example B3, whether the feature of the condition is a "sunny weekday," a "rainy weekday," a "sunny holiday," or a "rainy holiday" is displayed in the feature space.

[0155] Similarly, in display example B3, when the user operates the cursor CSR and selects the condition "sunny holiday", the display information generation unit 15 generates display information including the weights w_j_i of each of the multiple models f_i for the selected condition "sunny holiday" and information obtained from the weights w_j_i.

[0156] As yet another example, the display information generation unit 15 generates display information including multiple data points obtained by embedding model input information (explanatory variables) in a low-dimensional space (a two-dimensional space in the case of FIG. 13), as shown in display example B4 of FIG. 13.

[0157] 13, data points indicated by the model input information are indicated using markers of different shapes for each condition satisfied by the model input information. As an example, data points indicated by circular markers indicate data points indicated by model input information (explanatory variables) that satisfy condition c_1, and data points indicated by diamond markers indicate data points indicated by model input information (explanatory variables) that satisfy condition c_2.

[0158] 13, the input / output unit 16 may be configured to display a cursor CSR that can be operated by the user, allowing each data point to be selected. Even in this configuration, the display information generation unit 15 may be configured to generate display information including additional information to be presented to the user based on input from the user. Examples of the additional information include weights w_j_i of each of the multiple models f_i in the selected condition c_j and information obtained from the weights w_j_i. For example, the display information generation unit 15 generates display information including the shift degree of the above-described display example A2 and the flip degree of the above-described display example A4 as additional information. For example, as shown in FIG. 13, when the user operates the cursor CSR to select condition c_4, the display information generation unit 15 generates display information including the shift degree for condition c_4.

[0159] (Effect of prediction device 1B) As described above, prediction device 1B employs a configuration in which the prediction target information includes, in addition to the model input information, additional information not input to each model, and the evaluation information includes, in addition to the model input information for evaluation, the additional information for evaluation. With this configuration, weight update unit 11 updates some or all of the weight vectors based on the evaluation results, the model input information for evaluation, and the additional information for evaluation. Furthermore, prediction unit 12 selects one of the weight vectors based on the model input information and the additional information included in the prediction target information to obtain an integrated prediction result. Therefore, prediction device 1B, in addition to the effects achieved by prediction device 1 and prediction device 1A, can perform accurate predictions even when different distribution changes occur depending on the conditions by referring to the model input information and the additional information.

[0160] Furthermore, the prediction device 1B generates display information including at least one of a plurality of conditions c_j, weights w_j_i of the plurality of models f_i in each of the plurality of conditions c_j, and information obtained from the weights w_j_i. Therefore, the prediction device 1B can present to the user what kind of distribution changes have occurred in each of the plurality of conditions c_j over time.

[0161] Fourth Exemplary Embodiment A fourth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0162] (Outline of prediction device 1C) The prediction device 1C evaluates the performance of each of the plurality of models f_i by referring to evaluation information including model input information for evaluation, and calculates a plurality of first weight vectors w based on the evaluation information. (1) some or all of the j, and a plurality of second weight vectors w (2) Update some or all of the second weight vector w (2) _k is the first weight vector w (1) _j is a vector whose components are the weights assigned to each of the

[0163] Furthermore, the prediction device 1C calculates the prediction results predicted by each model f_i by referring to model input information included in prediction target information related to the prediction target, using a plurality of first weight vectors w (1) _j, and a plurality of second weight vectors w (2) _k based on the prediction target information, and outputs the integrated prediction result.

[0164] Here, the second weight vector w is selected based on the prediction target information. (2) _k is, for example, a vector. The selected second weight vector w (2) The first weight vector w is selected depending on k. (1) _j may be a single vector or multiple vectors.

[0165] (Configuration of prediction device 1C) The configuration of the prediction device 1C will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the configuration of the prediction device 1C. As shown in Fig. 14, the prediction device 1C includes a weight update unit 11, a prediction unit 12, a model management unit 13, a weight storage unit 14, a display information generation unit 15, and an input / output unit 16. In this exemplary embodiment, the weight update unit 11, the prediction unit 12, the model management unit 13, and the display information generation unit 15 respectively realize an update means, a prediction means, a management means, and a display information generation means.

[0166] The model management unit 13 and the input / output unit 16 have the same configuration as in the above-described exemplary embodiment, and therefore a description thereof will be omitted.

[0167] (Weight storage unit 14) The weight storage unit 14 includes: Nweight(1) first weight vectors w (1) _1, w (1) _2, ..., w (1) _Nweight(1) and Nweight(2) second weight vectors w (2) _1, w (2) _2, ..., w (2) _Nweight(2) and Here, Nweight(1) and Nweight(2) are, for example, natural numbers greater than or equal to 2. (1) _j, w (1) j Or write it as w (2) _k, w (2) k In other words, the weight storage unit 14 stores the first weight set W (1) and the second weight set W (2) I remember that.

[0168]

number

[0169]

number

[0170] (first weight vector) The first weight vector w (1) _j is the weight w of Nmodels (1) _j_i as element (w (1) ji , or w (1) j,i is a vector that is expressed by the following equation (18).

[0171]

number

[0172] Here, the weight w (1) _j_i is the first weight vector w (1) The weight vector w represents the weight assigned to model f_i when _j is selected. (1) _j may be updated by the weight update unit 11, and the initial value of each element may be determined arbitrarily. For example, the initial values ​​of each element may all be the same, or may be determined randomly.

[0173] In addition, the plurality of first weight vectors w stored in the weight storage unit 14 (1) Each of the first conditions c_j is a plurality of first conditions that can be satisfied by the prediction target information, and is associated with at least one of a plurality of first conditions that can be satisfied by the evaluation information. (1) There may be multiple first conditions c (1) Each of the first conditions c_j may be a condition that the model input information can satisfy. For example, if the model input information includes a weekday / holiday label and a weather label, a plurality of first conditions c_j may be (1) _1, c (1) _2, condition c (1) _1 "Sunny weekday" and condition c (1) In addition, when the evaluation information and the prediction target information include additional information, a plurality of first conditions c (1) Each of the j's may be a condition that the additional information may satisfy, or may be a condition that both the model input information and the additional information may satisfy.

[0174] Here, the first weight vector and the first condition do not necessarily correspond one-to-one. However, the following description focuses on an example in which there is a one-to-one correspondence, and the first weight vector w (1) The first condition that corresponds one-to-one with _j is the first condition c(1) In other words, the first weight vector w (1) _j is the first condition c (1) The first weight vector w (1) _j and the first condition c (1) If there is a one-to-one correspondence between _j and multiple first conditions c (1) The number of _j is equal to the number of the first weight vectors Nweight(1). Note that the first condition c (1) _j is c (1) j It can also be written as:

[0175] (second weight vector) On the other hand, the second weight vector w (2) _k is Nweight(1) weights w (2) _k_j as element (w (2) kj , or w (2) k,j is a vector that is expressed by the following equation (19):

[0176]

number

[0177] Here, the weight w (2) _k_j is the second weight vector w (1) If _k is selected, the first weight vector w (1) The weight vector w represents the weight assigned to _j. (2) _k may be updated by the weight update unit 11, and the initial value of each element may be determined arbitrarily. For example, the initial values ​​of each element may all be the same, or may be determined randomly.

[0178] In addition, the plurality of second weight vectors w stored in the weight storage unit 14 (2)Each of the second conditions c_k is a plurality of second conditions that can be satisfied by the prediction target information, and is associated with at least one of the plurality of second conditions that can be satisfied by the evaluation information. (2) It is preferable that the number of second conditions c_k is set to one. (2) Each of the second conditions c_k may be a condition that the model input information can satisfy. (2) _k satisfies the first condition c (1) For example, if the model input information includes a weekday / holiday label and a weather label, multiple first conditions c (2) _1, c (2) _2, condition c (2) _1 "Sunny weekday" and condition c (2) In addition, when additional information is included in the evaluation information and the prediction target information, a plurality of second conditions c (2) Each of the .sub.k may be a condition that the additional information may satisfy, or may be a condition that both the model input information and the additional information may satisfy.

[0179] Here, the second weight vector and the second condition do not necessarily correspond one-to-one. However, the following description focuses on an example in which there is a one-to-one correspondence, and the second weight vector w (2) The second condition that corresponds one-to-one with _k is the second condition c (2) In other words, the second weight vector w (2) _k is the second condition c (2) The second weight vector w (2) _k and the second condition c (2) If there is a one-to-one correspondence between _k and multiple second conditions c (2) The number of _k is equal to the number of second weight vectors, Nweight(2). Note that the second condition c (2) _k is c (2) k It can also be written as:

[0180] The second weight vector w (2) _k is the first weight vector w as described above. (1) The weight assigned to _j is the component w (2) _k_j. Therefore, the second weight vector w (2) _k is a weight vector used in the prediction process. (1) It can also be expressed as a weight vector for softly determining which of the first weight vectors w_j is to be used. Here, "softly determining" means, for example, (1) This refers to the use of a combination of _j using multi-stage coefficients.

[0181] In this exemplary embodiment, the plurality of first weight vectors w (1) _j has multiple first conditions c (1) _j may include a weight vector associated with a condition corresponding to the logical sum of any two or more conditions included in j.

[0182] (Prediction Section 12) The prediction unit 12 is configured in the same manner as the above-described exemplary embodiment, and in addition, outputs a prediction result obtained by integrating the prediction results predicted by each model f_i by referring to model input information included in prediction target information related to the prediction target, using a weight vector selected from a plurality of first weight vectors and a plurality of second weight vectors based on the prediction target information.

[0183] More specifically, the prediction unit 12 predicts a plurality of second conditions c (2) _k, the second condition c that the prediction target information satisfies (2) The second weight vector w associated with k (2) The prediction unit 12 also selects prediction results y_i output from each of the multiple models f_i for model input information included in the prediction target information, as follows: The selected second weight vector w (2) _k, and The selected second weight vector w (2) The first weight vector w is indirectly selected by k(1) _j combination For example, in the case of a regression task, the process of calculating the integrated prediction result is expressed by the following equation (20).

[0184]

number

[0185] In equation (20), the left side (hereinafter referred to as y^) indicates the integrated prediction result. x indicates the model input information included in the prediction target information. f i (x) is the model f i Prediction result by y i Indicates w (2) k,j is the second condition c that the model input information satisfies. (2) k The second weight vector w associated with (2) k The first weight vector w (1) j Also, w (1) j,i is the first weight vector w (1) j Among the elements of the model f i is the element corresponding to

[0186] In addition, in equation (20), each weight vector is

[0187]

number

[0188] It is assumed that it is normalized so that

[0189] For classification tasks, for example, i(x) represents a vector of dimension equal to the number of classes, and f i(x)The predicted probability for each class label i is expressed by: Then, the prediction unit 12 can calculate the predicted probability after integration using an equation similar to the above equation (20). When finally determining a label as a predicted value, the prediction unit 12 determines the class label with the highest probability as the label as the predicted value.

[0190] Furthermore, when the prediction target information includes additional information, the prediction unit 12 calculates a plurality of second weight vectors w based on the model input information and the additional information included in the prediction target information. (2) k In this case, w in equation (20) (2) k,j is the second condition c that is satisfied by either or both of the model input information and the additional information. (2) k The second weight vector w associated with (2) k The first weight vector w (1) j It may also be an element associated with

[0191] For example, an example will be described in which the prediction target information includes store surroundings images and weekday / holiday labels as model input information, and the resolution of the store surroundings images as additional information. In this example, the second condition c (2) 1 is "weekday and high resolution", and the second condition c (2) 2 is "weekday and low resolution", and the second condition c (2) 3 is "holiday and high resolution", and the second condition c (2) 4 may be “holiday and low resolution.” In this case, the number of second weight vectors Nweight(2) may be 4, which is the number of conditions.

[0192] Furthermore, for example, a set of prediction target information may be input to the prediction unit 12. Such a set X is expressed by the following equation (22).

[0193]

number

[0194] In equation (22), x m indicates the m-th model input information included in the Ninput prediction target information. m indicates the m-th additional information included in the Ninput pieces of prediction target information.

[0195] In this case, the prediction unit 12 outputs a set Y of integrated prediction results corresponding to the set X. Such a set Y is expressed by the following equation (23).

[0196]

number

[0197] In equation (23), y^ m indicates the integrated prediction result corresponding to the m-th prediction target information.

[0198] (Weight update unit 11) The weight update unit 11 is configured in the same manner as the exemplary embodiment described above, and in addition updates some or all of the multiple weight vectors based on the evaluation results obtained by evaluating the performance of each of the multiple models f_i included in the model pool MP by referring to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models f_i, and on the evaluation information.

[0199] More specifically, the weight update unit 11 determines whether the weights are equal to or smaller than the first plurality of conditions c (1) _j, the first condition c that the evaluation information satisfies (1) The first weight vector w associated with _j (1) For example, if the model input information for evaluation included in the evaluation information includes a weekday / holiday label indicating a weekday and a weather label indicating fine weather, the evaluation information satisfies the first condition c (1) Therefore, the weight update unit 11 satisfies the first condition c (1) The first weight vector w associated with _1 (1) _1 is the item to be updated.

[0200] The weight update unit 11 also updates the weights based on a second plurality of conditions c (2) _k, the second condition c that the evaluation information satisfies (2) The second weight vector w associated with k (2) For example, if the model input information for evaluation included in the evaluation information includes a weekday / holiday label indicating a weekday and a weather label indicating fine weather, the evaluation information satisfies the second condition c (2) Therefore, the weight update unit 11 satisfies the second condition c (2) The second weight vector w associated with _1 (2) _1 is the item to be updated.

[0201] Furthermore, the weight update unit 11 updates the first plurality of weight vectors w based on the plurality of pieces of evaluation information and the evaluation results of the performance of each model f_i with reference to the plurality of pieces of evaluation information. (1) some or all of the j, and a second plurality of weight vectors w (2) _k may be updated in part or in whole. The evaluation result of the performance of each model f_i with reference to a plurality of pieces of evaluation information may be, for example, a statistical value (e.g., average value, maximum value, minimum value, etc.) of the evaluation result of the performance of each model f_i with reference to each piece of evaluation information.

[0202] Furthermore, when the evaluation information includes additional information for evaluation, the weight update unit 11 updates the first plurality of weight vectors w based on the evaluation results of the performance of each model f_i with reference to the evaluation information, and the model input information for evaluation and the additional information for evaluation included in the evaluation information. (1) some or all of the j, and a second plurality of weight vectors w (2) Some or all of _k may be updated.

[0203] For example, the set of evaluation information Deval input to the weight update unit 11 is expressed by the following equation (24).

[0204]

number

[0205] In equation (24), x n indicates the model input information included in the n-th evaluation information among Neval evaluation information, and y n indicates the true value contained in the n-th evaluation information. n indicates additional evaluation information included in the n-th piece of evaluation information among the Neval pieces of evaluation information. Note that the value of Neval may be 1.

[0206] For example, the weight update unit 11 may update the weights of a plurality of first conditions c (1) For at least one of the first conditions c, the first condition c is selected from the plurality of pieces of evaluation information. (1) One or more pieces of evaluation information that satisfy the first condition c may be extracted. (1) The process of extracting one or more pieces of evaluation information that satisfy j is expressed by, for example, the following equation (25).

[0207]

number

[0208] In equation (25), the left side (hereinafter also referred to as D(1)eval_j) is a subset of Deval, and satisfies the first condition c (1) _j indicates a set of evaluation information that satisfies the condition.

[0209] Furthermore, the weight update unit 11 evaluates the performance of each model f_i using the extracted one or more pieces of evaluation information (D(1)eval_j), and based on the evaluation result, updates the first condition c (1) The first weight vector w associated with _j (1) Update _j. Equation (25) shows the case where the evaluation information includes additional evaluation information, and c (1) _j(x,v) is the model input information x for evaluation and the additional information v for evaluation contained in the evaluation information, and the first condition c (1) If _j is satisfied, it is true, and if not, it is false.

[0210] For example, suppose that the set Deval contains five pieces of evaluation information, three of which contain the weekday / holiday label "weekday" and the remaining two contain the weekday / holiday label "holiday." In this case, the first condition c (1) The first three pieces of evaluation information that satisfy the first condition c are extracted as a subset D(1)eval_1. (1) The latter two pieces of evaluation information that satisfy _2 "holiday" are extracted as the subset D(1)eval_2.

[0211] Similarly, the weight update unit 11 updates a plurality of second conditions c (2) For at least one of the second conditions c_k, the second condition c is selected from the plurality of pieces of evaluation information. (2) One or more pieces of evaluation information that satisfy the second condition c may be extracted. (2) The process of extracting one or more pieces of evaluation information that satisfy _k is expressed by, for example, the following equation (26).

[0212]

number

[0213] In equation (26), the left side (hereinafter also referred to as D(2)eval_k) is a subset of Deval and satisfies the second condition c (2) _k indicates a set of evaluation information that satisfies the condition.

[0214] Furthermore, the weight update unit 11 evaluates the first weight vector using the extracted one or more pieces of evaluation information (D(2)eval_k), and based on the evaluation result, updates the second condition c (2) The second weight vector w associated with k (2) Equation (26) shows the case where the evaluation information includes additional evaluation information, and c (2) _k(x,v) is the model input information x for evaluation and the additional information v for evaluation contained in the evaluation information, and the second condition c (2) If _k is satisfied, it is true, and if not, it is false.

[0215] For example, suppose that the set Deval contains five pieces of evaluation information, three of which contain the weekday / holiday label "weekday" and the remaining two contain the weekday / holiday label "holiday." In this case, the second condition c (2) The first three pieces of evaluation information that satisfy the condition "weekday" are extracted as the subset D(2)eval_1. (2) The latter two pieces of evaluation information that satisfy _2 "holiday" are extracted as a subset D(2)eval_2.

[0216] (Display information generation unit 15) The display information generation unit 15 is configured similarly to the exemplary embodiment described above, and generates display information by referring to the integrated prediction result derived by the prediction unit 12, the model pool MP, and the first and second weight vectors. Examples of the display information generated by the display information generation unit 15 will be described later.

[0217] (Prediction method flow) The prediction method S1C executed by the prediction device 1C configured as above will be described in substantially the same manner as the prediction method S1A described with reference to Fig. 4. However, details of the weight update process S11, the prediction process S12, and the display information generation process S15 are different.

[0218] (Detailed flow of weight update process) First, a detailed flow of the weight update process S11 will be described with reference to Fig. 15. Fig. 15 is a flow diagram illustrating an example of a detailed flow of the weight update process S11. As shown in Fig. 15, the weight update process S11 includes steps S111C to S116C.

[0219] (Step S111C) In step S111C, the weight update unit 11 acquires evaluation information. For example, when the prediction method S1A has been executed in the past, the weight update unit 11 may acquire evaluation information including, as model input information for evaluation, model input information included in the prediction target information used in the past prediction process S12. Furthermore, when the number of unprocessed evaluation information pieces among the acquired evaluation information pieces reaches a predetermined number, the weight update unit 11 may execute the processes from the next step S112C onwards using the predetermined number of evaluation information pieces.

[0220] (Step S112C) Subsequently, in step S112C, the weight update unit 11 selects, from the plurality of first conditions, the first condition c that is satisfied by the evaluation information acquired in step S111C. (1) j As an example, the weight update unit 11 extracts a plurality of first conditions c (1) j (e.g., c (1) 1 "Sunny Weekday", c (1) 5 "Sunny", c (1) 7 "all samples", etc.) are extracted from the evaluation information set Deval. (1) j Here, an example of the evaluation information subset D(1)eval_j is as described with reference to equation (25).

[0221] (Step S113C) Subsequently, in step S113C, the weight update unit 11 calculates the weight of the first condition c extracted in step S112C. (1) j The first weight vector w corresponding to (1) j As an example, the weight update unit 11 updates a plurality of first conditions c (1) 1 "Sunny Weekday", c (1) 5 "Sunny", c (1)7. Multiple first weight vectors w corresponding to "all samples" (1) 1. W (1) 5. W (1) 7 is updated. For example, the subsets D(1)eval_1, D(1)eval_5, and D(1)eval_7 of the evaluation information extracted in step S112C are used in the update process of these first weight vectors. However, this is not intended to limit the present example.

[0222] More specifically, the weight update unit 11 ·Predicted value y^ of each model i =f i The evaluation results for each model are derived by referring to (x)(i=1,...,Nmodel) and the true value (correct value) y. The first weight vector w is calculated based on the derived evaluation result. (1) j element of the model f i The element w corresponding to (1) j,i Update Here, the specific update algorithm does not limit the present exemplary embodiment, but as an example, the Hedge algorithm

[0223]

number

[0224] By the element w (1) j,i may be updated, where

[0225]

number

[0226] is the predicted value y^ i and the true value y, and η represents the learning rate.

[0227] Note that if D(1)eval_j contains multiple samples, the evaluation result of the model is the average value of the loss function.

[0228]

number

[0229] In this case, l in equation (27) i , l i mean Instead of the average value of the loss function, a statistical quantity such as the maximum or minimum value of the loss function may be used.

[0230] (Step S114C) Subsequently, in step S114C, the weight update unit 11 selects, from the plurality of second conditions, the second condition c that is satisfied by the evaluation information acquired in step S111C. (2) k As an example, the weight update unit 11 extracts one second condition c that the evaluation information satisfies. (2) k (For example, the second condition c (2) 1) is extracted from the evaluation information set Deval. (2) j Here, an example of the evaluation information subset D(2)eval_k is as described with reference to equation (26).

[0231] (Step S115C) Subsequently, in step S115C, the weight update unit 11 calculates the weight of the second condition c extracted in step S114C. (2) k The second weight vector w corresponding to (2) k As an example, the weight update unit updates one second condition c (1) 1. One second weight vector w corresponding to "sunny weekday" (2)1 is updated. For example, the evaluation information subset D(2)eval_1 extracted in step S114C is used in the update process of the second weight vector. However, this is not intended to limit the present example.

[0232] More specifically, the weight update unit 11 updates the first weight vector w (1) j Using the above, the predicted value for the evaluation information is calculated based on the first condition c (1) j (j=1, . . . , Nweight(1)). In other words, the weight update unit 11 calculates

[0233]

number

[0234] By this, the predicted value y^ (1) j Then, the weight update unit 11 calculates: Each predicted value y^ (1) j and the true value (correct value) y, to derive an evaluation result of each first weight vector; The second weight vector w is calculated based on the derived evaluation result. (2) j element, each first condition c (1) j Each element corresponding to (in other words, the first weight vector w (1) j each element corresponding to w (2) k,j Update Here, the specific update algorithm does not limit the present exemplary embodiment, but as an example, the Hedge algorithm may be used as in step S113C.

[0235] (Step S116C) In step S116C, the weight update unit 11 determines whether or not there is other evaluation information that has not yet been processed. If there is other evaluation information that has not yet been processed (YES in step S116C), the processing from step S112C is repeated. If not (NO in step S116C), the processing ends.

[0236] (Detailed flow of prediction process) Next, details of the prediction process S12 will be described with reference to Fig. 16. Fig. 16 is a flow diagram showing an example of the detailed flow of the prediction process S12. As shown in Fig. 16, the prediction process S12 includes steps S121C to S126C.

[0237] (Step S121C) First, in step S121C, the prediction unit 12 acquires a set X of prediction target information. The prediction unit 12 may acquire the set X of prediction target information stored in a memory included in the prediction device 1C, or may acquire the set X of prediction target information received via a network. Furthermore, the set X of prediction target information may include at least one piece of prediction target information, and is not limited to including a plurality of pieces of prediction target information. An example of the set X has been described with reference to equation (22) etc.

[0238] (Step S122C) Subsequently, in step S122C, the prediction unit 12 selects a second condition c that is satisfied by the prediction target information from the plurality of second conditions. (2) As an example, the prediction unit 12 extracts one second condition c_k determined by the prediction target information from the plurality of second conditions. (2) Extract _k.

[0239] (Step S123C) Subsequently, in step S123C, the prediction unit 12 calculates the second condition c extracted in step S122. (2) The second weight vector w associated with k (2) Select _k.

[0240] (Step S124C) Subsequently, in step S124C, the prediction unit 12 calculates the prediction results f i Calculate (x).

[0241] (Step S125C) Subsequently, in step S125C, the prediction unit 12 calculates the second weight vector w selected in step S123C. (2) Using _k, the integrated prediction result y^ is

[0242]

number

[0243] The above example shows an example of a regression task. In addition, in equation (31), each weight vector is calculated as follows:

[0244]

number

[0245] It is assumed that it is normalized so that

[0246] For classification tasks, for example, i (x) represents a vector of the number of classes, and f i The predicted probability for each class label i is expressed by (x). Then, the prediction unit 12 can calculate the predicted probability after integration using an equation similar to the above equation (31). When finally determining a label as a predicted value, the prediction unit 12 determines the class label with the highest probability as the label as the predicted value.

[0247] (Step S126C) In step S126C, the prediction unit 12 determines whether the set X includes other prediction target information for which an integrated prediction result has not yet been calculated. If other prediction target information is included (YES in step S126C), the process from step S122C is repeated for the other prediction target information. If other prediction target information is not included, the integrated prediction result y^ calculated in step S125C is output, and the process ends. Note that the next weight update process S11 may be performed using evaluation information that includes model input information included in the prediction target information used in the prediction process S12 as model input information for evaluation. However, as in the above-described exemplary embodiment, the weight update process S11 and the prediction process S12 may be performed independently of each other, and the order and timing of execution of each process are not specified.

[0248] (Example of display information generation process S15) An example of display information generated by the prediction device 1B when it executes the display information generation process S15 will be described with reference to Fig. 17. Fig. 17 is a diagram showing a display example C1.

[0249] Fig. 17 shows an example of display information generated by the display information generating unit 15 and displayed via the input / output unit 16. As shown in Fig. 17, the display information includes, as an example, a plurality of data points obtained by embedding model input information (explanatory variables) in a low-dimensional space (a two-dimensional space in the case of Fig. 17).

[0250] In the display example C1 shown in Fig. 17, a data point indicated by the model input information is indicated by a marker of a different shape for each condition that the model input information satisfies. For example, the data point indicated by a circular marker satisfies the second condition c (2) The data points indicated by the model input information (explanatory variables) satisfying the second condition c are shown. (2) The data points indicated by the model input information (explanatory variables) satisfying 2 are shown.

[0251] The prediction unit 12 may refer to the second weight vector to identify a set of conditions among the multiple conditions that are causing changes in conjunction with each other, and reflect this in the display information. The example in Fig. 17 shows a case where the prediction unit 12 finds that the conditions satisfied by the model input information indicated by the circular data points and the conditions satisfied by the model input information indicated by the diamond data points are linked, and the display information generation unit 15 includes a boundary line CONT surrounding these data points in the display information.

[0252] 17, the input / output unit 16 may be configured to display a cursor CSR that can be operated by the user, allowing each data point to be selected. In this configuration, the display information generation unit 15 may also be configured to generate display information including additional information to be presented to the user based on an input from the user. The display information generation unit 15 may also be configured to generate additional information to be presented to the user based on an input from the user. An example of the additional information is the selected condition c. (1) j and c (2) k 17, the user may operate the cursor CSR to set the first condition c (1) 4 and the second condition c (2) When selecting 4, the display information generating unit 15 performs the first condition c (1) 4 and the second condition c (2) 4. As an example, the display information generating unit 15 generates display information including the shift degree in the weight w i is expressed as the following equation (33), and the shift degree is calculated using the above equation (6).

[0253]

number

[0254] (Effect of prediction device 1C) According to the prediction device 1C configured as described above, some or all of the multiple first weight vectors and some or all of the multiple second weight vectors can be updated with high accuracy based on the evaluation results of each model evaluated with reference to the evaluation information, as well as based on the evaluation information referenced to obtain the evaluation results. Furthermore, when making predictions, weight vectors selected from the multiple first weight vectors and multiple second weight vectors updated in this way based on the prediction target information are used, thereby achieving the effect of enabling highly accurate ensemble predictions even when the distribution of model input information included in the prediction target information changes locally.

[0255] In addition, in the prediction device 1C, a plurality of first conditions c (1) j , multiple conditions c (2) k and generating display information including at least one of the first weight and the second weight of each of the plurality of models f_i in each of the plurality of models f_i in the plurality of models f_i, and at least one of the information obtained from the at least one of the first weight and the second weight. (1) j , multiple conditions c (2) k In addition, the prediction device 1C can present to the user information about the conditions under which the distribution change occurs in conjunction with the second weight vector information.

[0256] [Software implementation example] Some or all of the functions of the prediction devices 1, 1A, 1B, and 1C (hereinafter also referred to as "each of the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0257] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 18. Figure 18 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0258] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0259] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0260] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0261] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0262] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0263] (Appendix A1) a management means for managing a model pool including a first model and one or more second models that generate one or more modulated outputs by referring to an output of the first model; a prediction means for executing a prediction process using a plurality of models included in the model pool; A prediction device comprising:

[0264] (Appendix A2) the prediction means generates a prediction result by referring to the output of each of a plurality of models included in the model pool and the weight of each of the plurality of models; The method further includes updating means for updating the weights. 10. The prediction device according to claim A1.

[0265] (Appendix A3) The system further includes a display information generating means for generating display information including at least one of the weights of each of the plurality of models included in the model pool and information obtained from the weights. 10. The prediction device according to claim A2.

[0266] (Appendix A4) The display information includes modulation information indicating how the modulated output is generated by the second model among the plurality of models included in the model pool. 10. The prediction device according to claim 9, wherein the first and second components are selected from the group consisting of:

[0267] (Appendix A5) The second model generates the modulated output using a polynomial function that takes the output of the first model as an argument. 10. The prediction device according to claim 1, wherein the first and second inputs are input to the prediction device.

[0268] (Appendix A6) The first model is a machine learning model 10. The prediction device according to any one of appendices A1 to A5.

[0269] (Appendix A7) The update means updating some or all of the weight vectors based on evaluation results obtained by evaluating the performance of each of the multiple models included in the model pool with reference to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models, and based on the evaluation information; The prediction means The prediction result is output by integrating the prediction results predicted by each of the plurality of models with reference to model input information included in prediction target information related to the prediction target, using a weight vector selected from the plurality of weight vectors based on the prediction target information. 10. The prediction device according to claim 9, wherein the first and second inputs are input to the prediction device.

[0270] (Appendix A8) The update means updating some or all of the plurality of first weight vectors and some or all of the plurality of second weight vectors based on an evaluation result obtained by evaluating the performance of each of the plurality of models with reference to evaluation information including model input information for evaluation and true values ​​corresponding to the model input information, and the evaluation information; The prediction means The prediction results predicted by each model with reference to model input information included in prediction target information related to the prediction target are integrated using a weight vector selected from the plurality of first weight vectors and the plurality of second weight vectors based on the prediction target information, and the integrated prediction result is output. 10. The prediction device according to claim 9, wherein the first and second inputs are input to the prediction device.

[0271] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0272] (Appendix B1) a management process in which at least one processor manages a model pool including a first model and one or more second models that reference an output of the first model to generate one or more modulated outputs; a prediction process in which the at least one processor executes a prediction process using a plurality of models included in the model pool; A forecasting method comprising:

[0273] (Appendix B2) In the prediction process, the at least one processor generates a prediction result by referring to an output of each of a plurality of models included in the model pool and a weight of each of the plurality of models; The at least one processor further includes an update process for updating the weights. Forecasting method described in Appendix B1.

[0274] (Appendix B3) The at least one processor further includes a display information generation process for generating display information including at least one of weights of each of a plurality of models included in the model pool and information obtained from the weights. Forecasting method described in Appendix B2.

[0275] (Appendix B4) The display information includes modulation information indicating how the modulated output is generated by the second model among the plurality of models included in the model pool. A prediction method as described in Appendix B2 or B3.

[0276] (Appendix B5) The second model generates the modulated output using a polynomial function that takes the output of the first model as an argument. 10. The prediction method of any one of Appendices B1 to B4.

[0277] (Appendix B6) The first model is a machine learning model 10. The prediction method of any one of Appendices B1 to B5.

[0278] (Appendix B7) In the update process, the at least one processor: updating some or all of the weight vectors based on evaluation results obtained by evaluating the performance of each of the multiple models included in the model pool with reference to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models, and based on the evaluation information; In the prediction process, the at least one processor Output a prediction result obtained by integrating prediction results predicted by each of the plurality of models by referring to model input information included in the prediction target information regarding the prediction target, using a weight vector selected based on the prediction target information from among the plurality of weight vectors The prediction method according to any one of Appendices B2 to B4

[0279] (Appendix B8) In the update process, the at least one processor An evaluation result obtained by evaluating the performance of each of a plurality of models by referring to evaluation information including model input information for evaluation and a true value corresponding to the model input information, and based on the evaluation information, updating part or all of a plurality of first weight vectors and part or all of a plurality of second weight vectors In the prediction process, the at least one processor Output a prediction result obtained by integrating prediction results predicted by each model by referring to model input information included in the prediction target information regarding the prediction target, using the plurality of first weight vectors and a weight vector selected based on the prediction target information from among the plurality of second weight vectors The prediction method according to any one of Appendices B2 to B4

[0280] [Appendix C] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope shown in the claims

[0281] (Appendix C1) A program for causing a computer to function as a prediction device, The computer Management means for managing a model pool including a first model and one or more second models that generate one or more modulated outputs by referring to the output of the first model Prediction means for executing a prediction process using a plurality of models included in the model pool A prediction program for causing the computer to function as such

[0282] (Appendix C2) the prediction means generates a prediction result by referring to the output of each of a plurality of models included in the model pool and the weight of each of the plurality of models; The computer and further functioning as an update means for updating the weights. The prediction program described in Appendix C1.

[0283] (Appendix C3) The computer The model pool further functions as a display information generating means for generating display information including at least one of the weights of each of the plurality of models included in the model pool and information obtained from the weights. 1. The prediction program described in Appendix C2.

[0284] (Appendix C4) The display information includes modulation information indicating how the modulated output is generated by the second model among the plurality of models included in the model pool. A prediction program as described in appendix C2 or C3.

[0285] (Appendix C5) The second model generates the modulated output using a polynomial function that takes the output of the first model as an argument. 10. The prediction program according to any one of appendices C1 to C4.

[0286] (Appendix C6) The first model is a machine learning model 10. The prediction program of claim 9, wherein the prediction program is a program for predicting a prediction of a target object.

[0287] (Appendix C7) The update means updating some or all of the weight vectors based on evaluation results obtained by evaluating the performance of each of the multiple models included in the model pool with reference to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models, and based on the evaluation information; The prediction means The prediction result is output by integrating the prediction results predicted by each of the plurality of models with reference to model input information included in prediction target information related to the prediction target, using a weight vector selected from the plurality of weight vectors based on the prediction target information. 10. The prediction program of claim 9, wherein the prediction program is a program for predicting a prediction state of a target object.

[0288] (Appendix C8) The update means updating some or all of the plurality of first weight vectors and some or all of the plurality of second weight vectors based on an evaluation result obtained by evaluating the performance of each of the plurality of models with reference to evaluation information including model input information for evaluation and true values ​​corresponding to the model input information, and the evaluation information; The prediction means The prediction results predicted by each model with reference to model input information included in prediction target information related to the prediction target are integrated using a weight vector selected from the plurality of first weight vectors and the plurality of second weight vectors based on the prediction target information, and the integrated prediction result is output. 10. The prediction program of claim 9, wherein the prediction program is a program for predicting a prediction state of a target object.

[0289] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0290] (Appendix D1) at least one processor, a management process for managing a model pool including a first model and one or more second models that reference the output of the first model to generate one or more modulated outputs; a prediction process that executes a prediction process using a plurality of models included in the model pool; A prediction device that performs the above.

[0291] The prediction device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0292] (Appendix D2) In the prediction process, the at least one processor generates a prediction result by referring to an output of each of a plurality of models included in the model pool and a weight of each of the plurality of models; the at least one processor: Further, an update process for updating the weights is performed. 10. The prediction device according to claim D1.

[0293] (Appendix D3) the at least one processor: and further executing a display information generation process for generating display information including at least one of the weights of each of the plurality of models included in the model pool and information obtained from the weights. 10. The prediction device according to claim D2.

[0294] (Appendix D4) The display information includes modulation information indicating how the modulated output is generated by the second model among the plurality of models included in the model pool. 10. The prediction device of claim 9, wherein the first and second sub-units are connected to a first and second sub-unit

[0295] (Appendix D5) The second model generates the modulated output using a polynomial function that takes the output of the first model as an argument. 10. The prediction device according to claim 9, wherein the first and second sub-routines are identical to those described in claim 1.

[0296] (Appendix D6) The first model is a machine learning model 10. The prediction device according to any one of appendices D1 to D5.

[0297] (Appendix D7) In the update process, the at least one processor: updating some or all of the weight vectors based on evaluation results obtained by evaluating the performance of each of the multiple models included in the model pool with reference to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models, and based on the evaluation information; In the prediction process, the at least one processor The prediction result is output by integrating the prediction results predicted by each of the plurality of models with reference to model input information included in prediction target information related to the prediction target, using a weight vector selected from the plurality of weight vectors based on the prediction target information. 10. The prediction device according to any one of appendices D2 to D4.

[0298] (Appendix D8) In the update process, the at least one processor: updating some or all of the plurality of first weight vectors and some or all of the plurality of second weight vectors based on an evaluation result obtained by evaluating the performance of each of the plurality of models with reference to evaluation information including model input information for evaluation and true values ​​corresponding to the model input information, and the evaluation information; In the prediction process, the at least one processor The prediction results predicted by each model with reference to model input information included in prediction target information related to the prediction target are integrated using a weight vector selected from the plurality of first weight vectors and the plurality of second weight vectors based on the prediction target information, and the integrated prediction result is output. 10. The prediction device according to any one of appendices D2 to D4.

[0299] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0300] (Appendix E1) A program that causes a computer to function as a prediction device, The computer, a management process for managing a model pool including a first model and one or more second models that reference the output of the first model to generate one or more modulated outputs; a prediction process that executes a prediction process using a plurality of models included in the model pool; A non-transitory recording medium on which a prediction program for executing the above is recorded. [Explanation of symbols]

[0301] 1, 1A, 1B, 1C Prediction Device 11 Weight update unit 12 Prediction Department 13 Management Department, Model Management Department 14 Weight storage unit 15 Display information generation section 16 Input / output section C1 processor C2 Memory

Claims

1. a management means for managing a model pool including a first model and one or more second models that generate one or more modulated outputs by referring to an output of the first model; a prediction means for executing a prediction process using a plurality of models included in the model pool; A prediction device comprising:

2. the prediction means generates a prediction result by referring to the output of each of a plurality of models included in the model pool and the weight of each of the plurality of models; The method further includes updating means for updating the weights. The prediction device according to claim 1 .

3. The system further includes a display information generating means for generating display information including at least one of the weights of each of the plurality of models included in the model pool and information obtained from the weights. The prediction device according to claim 2 .

4. The display information includes modulation information indicating how the modulated output is generated by the second model among the plurality of models included in the model pool. The prediction device according to claim 3 .

5. The second model generates the modulated output using a polynomial function that takes the output of the first model as an argument. The prediction device according to claim 1 .

6. The first model is a machine learning model. The prediction device according to claim 1 .

7. The update means updating some or all of the weight vectors based on evaluation results obtained by evaluating the performance of each of the multiple models included in the model pool with reference to evaluation information including model input information for evaluation that is obtained over time after the start of operation of the multiple models, and based on the evaluation information; The prediction means The prediction result is output by integrating the prediction results predicted by each of the plurality of models with reference to model input information included in prediction target information related to the prediction target, using a weight vector selected from the plurality of weight vectors based on the prediction target information. The prediction device according to claim 2 .

8. The update means updating some or all of the plurality of first weight vectors and some or all of the plurality of second weight vectors based on an evaluation result obtained by evaluating the performance of each of the plurality of models with reference to evaluation information including model input information for evaluation and true values ​​corresponding to the model input information, and the evaluation information; The prediction means The prediction results predicted by each model with reference to model input information included in prediction target information related to the prediction target are integrated using a weight vector selected from the plurality of first weight vectors and the plurality of second weight vectors based on the prediction target information, and the integrated prediction result is output. The prediction device according to claim 2 .

9. a management process in which at least one processor manages a model pool including a first model and one or more second models that reference an output of the first model to generate one or more modulated outputs; a prediction process in which the at least one processor executes a prediction process using a plurality of models included in the model pool; A forecasting method comprising:

10. A program that causes a computer to function as a prediction device, The computer a management means for managing a model pool including a first model and one or more second models that generate one or more modulated outputs by referring to an output of the first model; a prediction means for executing a prediction process using a plurality of models included in the model pool; A prediction program that functions as a

Images