Learning device, control device, learning method, and recording medium

Abstract

This learning device determines an input value for use in learning a model, learns a determination rule for the input value using an evaluation function including a subexpression that shows a better evaluation as the difference between an output value of a simulator and an output value of the model for the same input value is larger, and learns the model using the input value determined by the input value determination means and the output value of the simulator with respect to the input value.

Description

Learning device, control device, learning method, and recording medium

[0001] The present invention relates to a learning device, a control device, a learning method, and a recording medium.

[0002] Model learning, such as a model that simulates the operation of a plant, may be performed. For example, a data processing device described in Patent Literature 1 performs machine learning using input values ​​for a process, such as manufacturing, in a plant and actual measured output values ​​of the process to generate an output prediction model for the process. The data processing device then determines control parameters for a control device that controls the process so as to reduce the difference between a predicted output value based on the output prediction model and a target value.

[0003] Japanese Patent Application Laid-Open No. 2019-021186

[0004] When learning a model, it is preferable that the learning be carried out efficiently.

[0005] An example of an object of the present disclosure is to provide a learning device, a control device, a learning method, and a recording medium that can solve the above-mentioned problems.

[0006] According to a first aspect of the present disclosure, a learning device includes an input value determination means for determining input values ​​to be used in learning a model, a decision rule learning means for learning a decision rule for the input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the output value of a simulator and the output value of the model for the same input value, and a model learning means for learning the model using the input value determined by the input value determination means and the output value of the simulator for that input value.

[0007] According to a second aspect of the present disclosure, the control device includes a control means for controlling an object simulated by a model obtained by learning using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value, and the model is obtained by learning using input values ​​determined using a determination rule for the input values.

[0008] According to a third aspect of the present disclosure, a learning method includes a computer determining input values ​​to be used in learning a model, learning a rule for determining the input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value, and learning the model using the determined input values ​​and the simulator's output value for those input values.

[0009] According to a fourth aspect of the present disclosure, a recording medium stores a program that causes a computer to determine input values ​​to be used in model training; learn a rule for determining the input values ​​using an evaluation function that includes a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value; and train the model using the determined input values ​​and the simulator's output value for those input values.

[0010] According to one aspect of the present disclosure, model training can be performed relatively efficiently.

[0011] FIG. 1 is a diagram illustrating an example of the configuration of a learning system according to at least one embodiment; FIG. 2 is a diagram illustrating an example of input and output of data in a learning system according to at least one embodiment; FIG. 3 is a diagram illustrating an example of a change in training error according to at least one embodiment; FIG. 4 is a diagram illustrating an example of a processing procedure by which a learning system according to at least one embodiment trains a predictive model; FIG. 5 is a diagram illustrating an example of the configuration of a learning system according to at least one embodiment; FIG. 6 is a diagram illustrating an example of the configuration of a learning device according to at least one embodiment; FIG. 7 is a diagram illustrating an example of the configuration of a control system according to at least one embodiment; FIG. 8 is a diagram illustrating an example of the configuration of a learning device according to at least one embodiment; FIG. 9 is a diagram illustrating an example of the configuration of a control device according to at least one embodiment; FIG. 10 is a diagram illustrating an example of the processing procedure of a learning method according to at least one embodiment; FIG. 11 is a diagram illustrating an example of the configuration of a computer according to at least one embodiment;

[0012] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Below, letters with circumflexes may be indicated by adding a "^" after the letter. For example, a circumflex x may also be written as x^.

[0013] First Embodiment Fig. 1 is a diagram illustrating an example of the configuration of a learning system according to at least one embodiment. In the configuration illustrated in Fig. 1, the learning system 1 includes a simulator device 100, a predictive model device 200, a learning device 300, and a communication network 400. The simulator device 100 includes a simulator 110. The predictive model device 200 includes a predictive model 210. The learning device 300 includes an input value determination unit 310, a decision rule learning unit 320, and a model learning unit 330.

[0014] The learning system 1 learns the prediction model 210. Model learning here refers to adjusting the parameter values ​​of the model based on training data. Model learning can also be called model training.

[0015] The simulator device 100 executes calculations using a simulator 110. The simulator device 100 may be configured using a computer. The simulator 110 simulates the behavior of an object. Specifically, the simulator 110 receives input values ​​indicating input to the object, such as operation input to the object, and calculates and outputs the state of the operating environment of the object when the object operates according to the input values. The state of the operating environment of the object referred to here may include the state of the object itself.

[0016] The target here is not limited to a specific one, and can be any one for which a simulation can be performed. For example, the target may be a device such as a robot, a machine tool, or a mobile object. Alternatively, the target may be a facility that includes multiple devices, such as a factory or a power plant.

[0017] The input value to the simulator 110 may be a combination of multiple values, for example, represented by a vector or a matrix. The output value of the simulator 110 (the value output from the simulator 110) may also be a combination of multiple values, for example, represented by a vector or a matrix.

[0018] The predictive model device 200 executes calculations using a predictive model 210. The predictive model device 200 may be configured using a computer. The predictive model 210, like the simulator 110, simulates the behavior of a target. Meanwhile, while the predictive model 210 is a learning target, the simulator 110 is not a learning target.

[0019] The prediction model 210 receives input values ​​indicating inputs to the target, such as operational inputs to the target, and calculates and outputs the state of the target's operating environment when the target operates according to the input values. The input values ​​to the prediction model 210 may be a combination of multiple values, for example, represented by a vector or a matrix. The output value of the prediction model may also be a combination of multiple values, for example, represented by a vector or a matrix.

[0020] The prediction model 210 may be configured to include a differentiable model such as a neural ordinary differential equation (Neural ODE). The differentiable model here is a model that is capable of calculating the time derivative of the output value of the model and the partial derivative of the output value of the model with respect to the input value to the model. For example, when the prediction model 210 is configured to include a neural ordinary differential equation, the neural ordinary differential equation f is expressed as shown in Equation (1).

[0021]

[0022] In equation (1), x^ denotes the internal state of the prediction model 210. The internal state x^ of the prediction model 210 can be considered as a predicted value of the state of the target operating environment by the prediction model 210. dx^ / dt denotes the time derivative of the internal state x^ of the prediction model 210. In equation (1), u denotes an input value to the prediction model 210.

[0023] Equation (1) indicates that the neural ordinary differential equation f outputs the time derivative of the internal state of the prediction model 210, i.e., the time derivative of the state of the target operating environment, which is determined based on the internal state x^ of the prediction model 210 and the input value u to the prediction model 210. In the neural ordinary differential equation, the derivative may be approximately represented using a difference.

[0024] The partial derivative of the output value f(x^, u) of the neural ordinary differential equation f with respect to the input value u is expressed as ∂(x, u) / ∂u. The ability to calculate this partial derivative makes it possible to use a gradient method, for example, when controlling an object using the prediction model 210, and it is expected that a control command value can be calculated in a relatively short time. The simulation of the object's behavior using the prediction model 210 is expressed as in Equation (2).

[0025]

[0026] Here, time is represented by time steps, and time t+1 indicates the time when one time step has elapsed since time t. Equation (2) shows that the output value f(x^, u) of the neural ordinary differential equation is integrated over time to calculate a predicted value x^(t+1) of the state of the target operating environment at time t+1. The prediction model 210 may perform the integral calculation using a numerical integration technique.

[0027] The learning device 300 learns the prediction model 210. Specifically, the learning device 300 adjusts parameter values ​​of the prediction model 210 based on training data. The learning device 300 may be configured using a computer.

[0028] The input value determination unit 310 determines input values ​​to the simulator 110 in order to generate training data for the prediction model 210. The input value determination unit 310 corresponds to an example of an input value determination means. The input value determination unit 310 generates data indicating input to an object, such as an operation input to the object, as input values ​​to the simulator 110. The input value determination unit 310 may generate data indicating boundary conditions in the simulation performed by the simulator 110, such as conditions related to the operating environment of the object, in addition to the input to the object, as input values ​​to the simulator 110. The input value determination unit 310 outputs the determined input values ​​to the simulator 110 and the prediction model 210.

[0029] The decision rule learning unit 320 learns a decision rule used by the input value determination unit 310 to determine an input value. In particular, the decision rule learning unit 320 learns the decision rule for the input value using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the output value of the simulator 110 and the output value of the prediction model 210 for the same input value. The decision rule learning unit 320 learns the decision rule for the input value so that the difference between the output value of the simulator 110 and the output value of the prediction model 210 for the same input value becomes greater. The decision rule learning unit 320 corresponds to an example of a decision rule learning means.

[0030] The following describes an example in which the decision rule learning unit 320 learns decision rules for input values ​​using a reinforcement learning technique. Reinforcement learning here is machine learning that learns a policy, which is an action rule for an agent that takes an action in an environment, based on the state in the environment and a reward that represents an evaluation of the state or action.

[0031] In the case of the learning system 1, the operating environment of the target in the simulation performed by the simulator 110 is the environment in reinforcement learning. Furthermore, the input values ​​from the input value determination unit 310 to the simulator 110 and the prediction model are the actions in reinforcement learning. Furthermore, the function by which the input value determination unit 310 determines the input values ​​is the agent in reinforcement learning, and the decision rules by which the input value determination unit 310 determines the input values ​​are the policies in reinforcement learning. Furthermore, the evaluation function used by the decision rule learning unit 320 to learn the decision rules for input values ​​is the reward function in reinforcement learning.

[0032] Hereinafter, the input value determined by the input value determination unit 310 will also be referred to as an action. The output value of the simulator 110 will also be referred to as a correct value of the state or a state. The output value of the prediction model 210 will also be referred to as a predicted value of the state or a state. However, the method by which the decision rule learning unit 320 learns the decision rule for the input values ​​is not limited to a specific method. For example, the decision rule learning unit 320 may learn the decision rule for the input values ​​by machine learning other than reinforcement learning.

[0033] The prediction error of the prediction model 210 may be used as a sub-expression of the evaluation function used by the decision rule learning unit 320, which indicates a better evaluation the greater the difference between the output value of the simulator 110 and the output value of the prediction model 210 for the same input value. The prediction error here refers to the difference between the output value of the prediction model 210 and the output value of the simulator 110.

[0034] Alternatively, this sub-formula may be such that, when the magnitude of the prediction error is larger than that of the previous step in reinforcement learning, an evaluation function value is added as a reward value. This sub-formula can be regarded as a sub-formula that adds points to the evaluation when the magnitude of the prediction error becomes larger.

[0035] Alternatively, learning device 300 may store the prediction error for each episode or step in reinforcement learning. Then, as a subexpression of the evaluation function used by decision rule learning unit 320, a subexpression may be used that adds a reward value to the final reward of the episode or the reward of the step when the prediction error is larger than that of an episode or step that is the same as an episode in past learning or an episode or step that is similar to an episode in past learning.

[0036] As a result, if the magnitude of the prediction error for a learned episode or step is larger than in previous learning, it is expected that it will be easier to learn that episode or step again, or that it will be easier to learn an episode or step similar to that episode or step.

[0037] For example, the state (output value of the simulator 110) and the action (input value determined by the input value determination unit 310) in each episode or step may be represented by a vector, and the distance between the vectors may be used as the similarity between episodes or steps. The criterion for determining whether episodes or steps are similar may be whether the similarity indicates a similarity equal to or greater than a predetermined threshold.

[0038] The model learning unit 330 learns the prediction model 210 using training data that is a combination of input values ​​acquired by the input value determination unit 310 and output values ​​of the simulator 110 corresponding to those input values. The model learning unit 330 learns the prediction model 210 so that the output values ​​of the prediction model 210 become closer to the output values ​​of the simulator 110. The model learning unit 330 corresponds to an example of a learning means. As a learning method for the prediction model 210 by the model learning unit 330, supervised learning may be used in which the output values ​​of the simulator 110 are used as correct answers (teaching data).

[0039] The trained prediction model 210 can be used, for example, for model predictive control of an object. In model predictive control, an optimization calculation is performed in which the control plan for the object is repeatedly updated and the predicted value is calculated so as to minimize the residual between a target value, such as a target state, and a predicted value that would be obtained if planned control were performed on the object.

[0040] In model predictive control, by calculating a predicted value using the prediction model 210, the amount of calculation is expected to be smaller and the calculation can be completed in a relatively short time compared to when using the simulator 110. Furthermore, by using a differentiable model as the prediction model 210, a method using differentiation, such as a gradient method, can be applied when determining a control plan that reduces the residual between the target value and the predicted value, and it is expected that the calculation can be completed in a relatively short time.

[0041] In particular, in model predictive control, it is necessary to repeatedly update the control plan and calculate the predicted value in real time. By using the predictive model 210 in model predictive control, it is expected that the control plan can be updated and the predicted value can be calculated in a realistic time. Furthermore, by using the predictive model 210 in model predictive control, the number of repetitions of updating the control plan and calculating the predicted value can be relatively increased, thereby improving the accuracy of control of the target. However, the use of the predictive model 210 is not limited to a specific use.

[0042] 2 is a diagram showing an example of data input / output in the learning system 1. In the example of FIG. 2, the input value determination unit 310 determines an input value and outputs the determined input value to the simulator 110 and the prediction model 210.

[0043] The simulator 110 outputs an output value corresponding to an input value. The output value of the simulator 110 is used as a correct answer value for a state after a state transition corresponding to the action (input value) determined by the input value determination unit 310. The simulator 110 outputs the output value to the model learning unit 330, the input value determination unit 310, and the decision rule learning unit 320.

[0044] The prediction model 210 also outputs an output value corresponding to the input value. The output value of the prediction model is used as a predicted value by the prediction model 210 of a state after a state transition corresponding to the action determined by the input value determination unit 310. The prediction model 210 outputs the output value to the model learning unit 330 and the decision rule learning unit 320.

[0045] A series of processes in which the input value determination unit 310 determines input values ​​and the simulator 110 and the prediction model 210 each output output values ​​corresponding to the input values, or the time period for such processes, is also referred to as a step. One step is also referred to as one step. Repetition of steps is also referred to as an episode. One episode is also referred to as one episode.

[0046] At the start of an episode, the input value determination unit 310 determines and outputs input values ​​according to a preset initial state for that episode. From the second step onwards in the episode, the input value determination unit 310 determines and outputs input values ​​according to the output values ​​of the simulator 110.

[0047] The decision rule learning unit 320 updates the parameter values ​​of the decision rules for the behavioral data each time an episode ends. Based on the output values ​​of the simulator 110 and the prediction model 210 for each step, the decision rule learning unit 320 updates the parameter values ​​of the decision rules for input values ​​so as to increase the magnitude of the difference between the output values ​​of the simulator 110 and the prediction model 210. For example, the decision rule learning unit 320 may learn the decision rules for input values ​​using the evaluation function r shown in Equation (3) so as to increase the evaluation function value.

[0048]

[0049] i is an index that identifies a step in an episode, where 1≦i≦i ｍａｘ It takes an integer value of i ｍａｘ denotes the number of steps in an episode. ｍａｘ may have the same value for all episodes, or may have a different value for each episode.

[0050] x ｉdenotes the output value of the simulator 110 at the i-th step of the episode. ｉ denotes the output value of the prediction model 210 at the i-th step of the episode. x denotes the output value x of the simulator 110 at the episode. １ , x ２ , ..., x ｉｍａｘ x^ is the output value x^ of the prediction model 210 in the episode. １ , x^ ２ , ..., x^ ｉｍａｘ The evaluation function r in formula (3) indicates a better evaluation as the evaluation function value is larger. ｉ -x^ ｉ ) ２ is the output value x of the simulator 110 for the same input value. ｉ and the output value x^ of the prediction model 210 ｉ The larger the difference between the two, the better the subexpression is evaluated.

[0051] Each time an episode ends, the model learning unit 330 updates the parameter values ​​of the prediction model 210. Based on the output value of the simulator 110 and the output value of the prediction model 210 for each step, the model learning unit 330 updates the parameter values ​​of the prediction model 210 so that the output value of the prediction model 210 approaches the output value of the simulator 110.

[0052] For example, the model learning unit 330 may use the state x at the start of the step. ｉ , input value u ｉ , next state x ｉ＋１ The model learning unit 330 uses training data that is a combination of the state x ｉ When the input value u ｉ The output value x^ of the prediction model 210 when receiving the input ｉ＋１ is the next state x ｉ＋１ The parameter values ​​of the prediction model 210 are updated so that the

[0053] An upper limit value for evaluation may be set for the evaluation function used by decision rule learning unit 320. This reduces the possibility that the magnitude of the difference between the output value of simulator 110 and the output value of prediction model 210 will suddenly change, causing instability in the learning of prediction model 210.

[0054] The evaluation function used by the decision rule learning unit 320 may include a subexpression that indicates a worse evaluation the greater the difference in input values ​​between steps. The evaluation value indicated by this subexpression is also referred to as an action difference penalty. By including such a subexpression in the evaluation function, the decision rule learning unit 320 learns the decision rule for input values ​​so as to avoid sudden changes in the input values. This is expected to prevent a sudden change in the input values ​​to the simulator 110, which would otherwise cause a decrease in the accuracy of the simulator 110. Furthermore, by being able to avoid a decrease in the accuracy of the simulator 110 and by being able to avoid sudden changes in the input values ​​to the prediction model 210, it is possible to stabilize the learning of the prediction model 210.

[0055] Furthermore, the learning rate for the learning performed by model learning unit 330 and the learning rate for the learning performed by decision rule learning unit 320 may be set so that the learning performed by model learning unit 330 is slower than the learning performed by decision rule learning unit 320. Here, even with the same teacher data, the magnitude of the training error may differ as the learning of the prediction model progresses. Here, the output value of simulator 110 corresponds to the teacher data, and the difference between the output value of prediction model 210 and the output value of simulator 110 corresponds to the training error.

[0056] FIG. 3 is a diagram showing an example of the change in training error. FIG. 3 shows an example in which a state is represented by the values ​​of two state variables a and b. The horizontal axis of the graph in FIG. 3 shows the value of state variable a, and the vertical axis shows the value of state variable b. In the example in FIG. 3, x ０ denotes the initial state in the episode. The time series of the output values ​​of the simulator 110 is １ , x ２ , x ３ ,...

[0057] Furthermore, the time series of the output values ​​of the prediction model 210 is expressed as x^１ , x^ ２ , x^ ３ ,... and x^ １’ , x^ ２’ , x^ ３’ , ... are shown. Both of these time series are based on the initial state x ０ 10 shows an example of a time series of output values ​​of the prediction model 210 that has received the same time series of input values ​​as the time series of input values ​​to the simulator 110.

[0058] Also, the time series x^ １’ , x^ ２’ , x^ ３’ , ... is the time series x^ １ , x^ ２ , x^ ３ , ..., the time series when the learning of the prediction model 210 is progressing, and the magnitude of the training error is smaller. For example, the training error x^ before the learning progresses in the second step of the scenario ２ -x ２ The training error x^ after learning has progressed is ２’ -x ２ The magnitude of the error is smaller.

[0059] In this way, it is conceivable that the magnitude of the training error for the same initial state and the same time series of input values ​​changes as the learning of the prediction model 210 progresses, and therefore the input value to be output by the input value determination unit 310 changes. For example, if the magnitude of the training error is |x^ ２’ -x ２ If there is an input value that is greater than |, it is considered that the input value determination unit 310 should select that input value. Also, it is considered that the decision rule learning unit 320 should learn the decision rule for the input value in this way.

[0060] It is conceivable that, from the same state, a change in the input value to be output by input value determination unit 310 may cause instability in the learning by decision rule learning unit 320. Therefore, as described above, it is conceivable to set the learning rate in the learning performed by model learning unit 330 and the learning rate in the learning performed by decision rule learning unit 320 so that the learning performed by model learning unit 330 is slower than the learning performed by decision rule learning unit 320.

[0061] For example, if the prediction model 210 and the decision rule used by the input value determination unit 310 to determine input values ​​are constructed using models with similar structures, such as neural networks with similar structures, and the learning of these is performed using a similar learning algorithm, the learning rate in the learning performed by the model learning unit 330 may be set to a value smaller than the learning rate in the learning performed by the decision rule learning unit 320.

[0062] This reduces the rate of change of the prediction model 210 compared to the rate of change of the decision rule for input values, and allows the input values ​​that the input value determination unit 310 should output from the same state to be approximately considered to be the same. This is expected to stabilize learning by the decision rule learning unit 320. This can be seen as bringing a learning environment that is not a Markov Decision Process (MDP) closer to a Markov Decision Process.

[0063] In addition to or instead of setting the learning rate as described above, the state of reinforcement learning may be expressed as a combination of the output value of the simulator 110 and an index value related to the progress of learning of the prediction model 210. This allows situations in which the input value determination unit 310 should output different input values ​​from the same state to be reinterpreted as situations in which different input values ​​should be output from different states. This is expected to stabilize learning by the decision rule learning unit 320. This can also be seen as bringing a learning environment that is not a Markov decision process closer to a Markov decision process.

[0064] As an index value relating to the progress of learning of the prediction model 210, the model learning unit 330 may count the number of times that learning of the prediction model 210 has been performed for each state. Here, even if the state to be learned does not completely match the learned state, if it is a similar state, the results of past learning are reflected in the state to be learned. To reflect this, the model learning unit 330 may cumulatively count the number of times that learning of the prediction model 210 has been performed for similar states, for example, by rounding the values ​​of the state variables by rounding them off.

[0065] Alternatively, the model learning unit 330 may count the number of steps in which the prediction model 210 has been learned (the total number of states in which the prediction model 210 has been learned) as an index value relating to the progress of learning of the prediction model 210. This allows each state to be treated as a separate state for each repetition when learning is repeated for each individual state.

[0066] 4 is a diagram showing an example of the processing procedure by which the learning system 1 learns the prediction model 210. In the processing of FIG. 4, the learning device 300 performs initial setting of the episode (step S101). In particular, the input value determination unit sets the state set as the initial state of the episode to be executed as the state in reinforcement learning.

[0067] Next, the learning system 1 executes processing for one step in the episode (step S111). In particular, the input value determination unit 310 determines input values ​​based on the state. The simulator 110 outputs an output value corresponding to the state stored in the simulator 110 itself and the input value determined by the input value determination unit 310. The prediction model 210 outputs an output value corresponding to the state stored in the prediction model 210 itself and the input value determined by the input value determination unit 310.

[0068] Next, the decision rule learning unit 320 determines whether the episode termination condition is satisfied (step S112). The episode termination condition here is not limited to a specific condition. For example, the episode termination condition may be a condition that step S has been repeated a predetermined number of times. Alternatively, a condition related to the state may be set as the episode termination condition. Alternatively, a condition related to the evaluation function value may be set as the episode termination condition.

[0069] If the decision rule learning unit 320 determines that the episode termination condition is not met (step S112: NO), the process returns to step S111. On the other hand, if the decision rule learning unit 320 determines that the episode termination condition is met (step S112: YES), the decision rule learning unit 320 updates the parameter values ​​of the decision rule for the input value (step S121). Furthermore, the model learning unit 330 updates the parameter values ​​of the prediction model (step S122).

[0070] Next, the model learning unit 330 determines whether a termination condition for the learning of the prediction model is met (step S123). The termination condition for the learning of the prediction model here is not limited to a specific condition. For example, the termination condition for the learning of the prediction model may be a condition that the number of times the episode is executed reaches a predetermined number. Alternatively, a condition related to the accuracy of the prediction model 210 may be set as the termination condition for the learning of the prediction model.

[0071] If the model training unit 330 determines that the termination condition for the prediction model training is not met (step S123: NO), the decision rule training unit 320 selects the next episode to be executed (step S131). After step S131, the process returns to step S101. On the other hand, if the model training unit 330 determines in step S123 that the termination condition for the prediction model training is met (step S123: YES), the learning system 1 ends the process of FIG. 4.

[0072] As described above, the input value determination unit 310 determines input values ​​to be used in model training. The decision rule learning unit 320 learns the input value decision rules using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the prediction model's output value for the same input value. The model training unit 330 trains the prediction model 210 using the input values ​​determined by the input value determination unit 310 and the simulator's output value for that input value.

[0073] According to the learning device 300, it is expected that the input value determination unit 310 will determine input values ​​that will increase the difference between the output value of the simulator 110 and the output value of the prediction model 210, and these input values ​​can be used to train the prediction model 210. According to the learning device 300, the prediction model 210 can be trained using input values ​​that will increase the error, and in this respect, the prediction model 210 can be trained relatively efficiently.

[0074] Furthermore, the decision rule learning unit 320 learns the input value decision rule through reinforcement learning, using the output value of the simulator 110 as the state, the input value as the action, and the evaluation function as the reward function. The learning device 300 has a high degree of freedom in timing the acquisition of training data for learning the input value decision rule, since the input value decision rule is learned through reinforcement learning. In this respect, the learning device 300 can perform these learning phases without having to be aware of the distinction between the input value decision rule learning phase and the prediction model 210 learning phase. Furthermore, the learning device 300 can use a reinforcement learning algorithm to learn the input value decision rule. In this respect, the burden on the designer designing the learning device 300 is relatively light.

[0075] Furthermore, the decision rule learning unit 320 uses a combination of the output value of the simulator 110 and an index value related to the progress of learning of the prediction model 210 as the state of reinforcement learning. Here, if the output value of the simulator 110 is the state and the input value is the action, as the learning of the prediction model 210 progresses, it is conceivable that the magnitude of the difference between the output value of the simulator 110 and the output value of the prediction model 210 will differ for the same state and the same action. This may result in different evaluations indicated by the evaluation function, making the learning of the decision rule for the input value unstable.

[0076] In contrast, in the learning device 300, even if the output value of the simulator 110 is the same, if the degree of learning progress of the prediction model 210 is different, the state is treated as being different. In this respect, the learning device 300 can stabilize the learning of the input value determination rule.

[0077] Furthermore, the learning rate of the model learning means and the learning rate of the decision rule learning means are set so that the learning performed by the model learning unit 330 is slower than the learning performed by the decision rule learning unit 320. The learning device 300 can prevent a sudden change in the evaluation function in learning the decision rule for input values. In this respect, the learning device 300 is expected to be able to stably learn the decision rule for input values.

[0078] Furthermore, the decision rule learning unit 320 uses an evaluation function with an upper limit set for the evaluation. The learning device 300 can reduce the possibility that the magnitude of the difference between the output value of the simulator 110 and the output value of the prediction model 210 will suddenly change, causing the learning of the prediction model 210 to become unstable.

[0079] Furthermore, the decision rule learning unit 320 uses an evaluation function that includes a subexpression that indicates a worse evaluation the greater the difference in input values ​​between steps. This allows the decision rule learning unit 320 to learn the decision rule for input values ​​so as to avoid sudden changes in the input values. In this respect, the learning device 300 is expected to be able to avoid a decrease in the accuracy of the simulator 110 due to a sudden change in the input values ​​to the simulator 110. Furthermore, the learning device 300 can avoid a decrease in the accuracy of the simulator 110 and avoid sudden changes in the input values ​​to the prediction model 210, thereby stabilizing the learning of the prediction model 210.

[0080] Second Embodiment An autoencoder may be used to calculate an index value relating to the degree of progress of learning of the prediction model 210. In the second embodiment, an example of this case will be described.

[0081] 5 is a diagram illustrating an example of the configuration of a learning system according to at least one embodiment. In the configuration illustrated in FIG. 5, the learning system 2 includes a simulator device 100, a predictive model device 200, a learning device 301, and a communication network 400. The simulator device 100 includes a simulator 110. The predictive model device 200 includes a predictive model 210. The learning device 301 includes an input value determination unit 310, a decision rule learning unit 320, a model learning unit 330, an autoencoder 340, and an autoencoder learning unit 350.

[0082] 5, parts having the same functions as those in FIG. 1 are given the same reference numerals (100, 110, 200, 210, 310, 320, 330, 400), and detailed descriptions thereof will be omitted here. Learning system 2 differs from learning system 1 in that learning device 301 further includes an autoencoder 340 and an autoencoder learning unit 350 in addition to the configuration of learning device 300 in learning system 1. In other respects, learning system 2 is similar to learning system 1.

[0083] The autoencoder 340 performs dimensional compression and reconstruction of data. The autoencoder learning unit 350 trains the autoencoder 340 in accordance with the progress of learning of the prediction model 210. The autoencoder learning unit 350 is an example of an autoencoder learning means. In the learning system 2, the model learning unit 330 uses an index value that uses a reconstruction error with which the autoencoder 340 reconstructs the output value of the simulator 110 as an index value related to the progress of learning of the prediction model 210.

[0084] 6 is a diagram showing an example of data input / output in the learning system 2. In the example of Fig. 6, the simulator 110 outputs output values ​​to the model learning unit 330, the input value determination unit 310, the decision rule learning unit 320, the autoencoder 340, and the autoencoder learning unit 350.

[0085] The autoencoder 340 performs dimensional compression and reconstruction on the output values ​​of the simulator 110, and outputs the reconstructed data to the autoencoder learning unit 350. Here, the reconstructed data is data obtained by dimensionally compressing and reconstructing the output values ​​of the simulator 110. The autoencoder learning unit 350 trains the autoencoder 340 in accordance with the progress of training of the prediction model 210 by the model training unit 330. In training the autoencoder 340, the autoencoder learning unit 350 updates the parameter values ​​of the autoencoder 340 so that the reconstructed data approaches the original data (the output values ​​of the simulator 110).

[0086] Furthermore, the autoencoder learning unit 350 outputs the reconstruction error to the model learning unit 330. Here, the reconstruction error is the error of the reconstruction data with respect to the output value of the simulator 110. The model learning unit 330 learns the prediction model 210 using an index value that uses the reconstruction error as an index value related to the progress of learning of the prediction model 210. The model learning unit 330 may use the reconstruction error as the index value, or may use a value obtained by performing a calculation on the reconstruction error as the index value.

[0087] The smaller the magnitude of the reconstruction error, the more progress has been made in learning the prediction model 210 with respect to the output value of the simulator 110 corresponding to the state in reinforcement learning. In other respects, the input and output of data in the example of Fig. 6 is the same as the input and output of data in the example of Fig. 2.

[0088] 7 is a diagram showing an example of a processing procedure in which the learning system 2 learns the prediction model 210. Steps S201 to S212 in Fig. 7 are the same as steps S101 to S112 in Fig. 4 .

[0089] If the decision rule learning unit 320 determines in step S212 that the episode termination condition is not met (step S212: NO), the process returns to step S211. On the other hand, if the decision rule learning unit 320 determines that the episode termination condition is met (step S212: YES), the process proceeds to step S221.

[0090] Step S221 is the same as step S121 in Fig. 4. After step S221, the model learning unit 330 updates the parameter values ​​of the prediction model 210 based on the output value of the simulator 110, the output value of the prediction model 210, and the reconstruction error obtained by the autoencoder 340 reconstructing the output value of the simulator 110 (step S222). Next, the autoencoder learning unit 350 updates the parameter values ​​of the autoencoder 340 so as to reduce the magnitude of the reconstruction error (step S223).

[0091] After step S223, the process proceeds to step S224. Step S224 is the same as step S123 in Fig. 4. If the model learning unit 330 determines in step S224 that the condition for ending learning of the prediction model is not met (step S224: NO), the process proceeds to step S231. Step S231 is the same as step S131 in Fig. 4. After step S231, the process returns to step S201.

[0092] On the other hand, if the model learning unit 330 determines in step S224 that the condition for ending learning of the prediction model is met (step S224: YES), the learning system 2 ends the processing of FIG.

[0093] As described above, the autoencoder learning unit 350 learns the autoencoder 340 in accordance with the progress of learning of the prediction model 210. The model learning unit 330 uses an index value that uses a reconstruction error with which the autoencoder 340 reconstructs the output value of the simulator.

[0094] In the learning device 300, it is considered that the magnitude of the reconstruction error decreases as the learning of the prediction model 210 and the learning of the autoencoder 340 progresses. In this way, in the learning device 300, the degree of progress of the learning of the prediction model 210 can be represented by an index value using the reconstruction error of the autoencoder 340.

[0095] Third Embodiment In the third embodiment, control of an object will be described. Fig. 8 is a diagram showing an example of the configuration of a control system according to at least one embodiment. In the configuration shown in Fig. 8, the control system 3 includes a control device 500 and an object 910. The control device 500 includes a prediction model 210 and a control unit 510.

[0096] The target 910 is a target of control in the control system 3. As described above regarding the target, the target 910 is not limited to a specific one. The control device 500 controls the target 910. The control device 500 may be configured using a computer.

[0097] The prediction model 210 simulates the behavior of the target 910. In the example of Fig. 8, a prediction model 210 that has been trained by the learning system 1 or the learning system 2 is used as the prediction model 210. The control unit 510 uses the prediction model 210 to perform model prediction control on the target 910. The control unit 510 corresponds to an example of a control means.

[0098] In model predictive control, the control unit 510 acquires an initial value of a control plan for the object 910. Then, the control unit 510 performs optimization calculations, repeatedly updating the control plan for the object 910 and calculating predicted values, so as to minimize the residual between a set target value and a predicted value when planned control for the object 910 is performed. For each time step in the control of the object 910, the control unit 510 controls the object 910 based on the control plan at that time.

[0099] The control device 500 may include a control unit 510 in addition to the respective units of the learning device 300. The control device 500 may then learn the prediction model 210 and control the object 910 using the trained prediction model 210. Alternatively, the control device 500 may include a control unit 510 in addition to the respective units of the learning device 301. The control device 500 may then learn the prediction model 210 and control the object 910 using the trained prediction model 210.

[0100] Alternatively, the control device 500 may be configured as a device separate from the learning device 300 or the learning device 301. The control device 500 may then acquire the prediction model 210 that has been trained by the learning device 300 or the learning device 301, and perform control over the target 910.

[0101] As described above, the control unit 510 controls the object 910 simulated by the prediction model 210 using the prediction model 210 obtained by learning using input values ​​determined using input value determination rules, which are obtained by learning using an evaluation function including a subexpression that indicates a better evaluation the larger the difference between the simulator's output value and the prediction model 210's output value for the same input value.

[0102] According to the control device 500, it is expected that the prediction model 210 has been trained using input values ​​that increase the difference between the output value of the simulator and the output value of the prediction model 210. In this respect, it is expected that the learning of the prediction model 210 has been performed relatively efficiently according to the control device 500.

[0103] 9 is a diagram illustrating an example of the configuration of a learning device according to at least one embodiment. In the configuration illustrated in FIG. 9, a learning device 610 includes an input value determination unit 611, a decision rule learning unit 612, and a model learning unit 613.

[0104] With this configuration, the input value determination unit 611 determines input values ​​to be used in model training. The decision rule learning unit 612 learns decision rules for input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value. The model training unit 613 trains the model using the input values ​​determined by the input value determination means and the simulator's output values ​​for those input values. The input value determination unit 611 is an example of input value determination means. The decision rule learning unit 612 is an example of decision rule learning means. The model training unit 613 is an example of model training means.

[0105] With the learning device 610, it is expected that the input value determination unit 611 will determine input values ​​that will increase the difference between the simulator output value and the model output value, and the model can be trained using those input values. With the learning device 610, the model can be trained using input values ​​that will increase the error, and in this respect, the model can be trained relatively efficiently.

[0106] The input value determination unit 611 can be realized, for example, using the functions of the input value determination unit 310 in Fig. 1. The decision rule learning unit 612 can be realized, for example, using the functions of the decision rule learning unit 320 in Fig. 1. The model learning unit 613 can be realized, for example, using the functions of the model learning unit 330 in Fig. 1.

[0107] Fifth Embodiment Fig. 10 is a diagram showing an example of the configuration of a control device according to at least one embodiment. In the configuration shown in Fig. 10, a control device 620 includes a control unit 621.

[0108] In this configuration, the control unit 621 controls the object simulated by the model, using a model obtained by learning using input values ​​determined using a decision rule for input values, which is obtained by learning using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the output value of the simulator and the output value of the model for the same input value. The control unit 621 is an example of a control means.

[0109] According to the control device 620, it is expected that the model has been trained using input values ​​that increase the difference between the output values ​​of the simulator and the output values ​​of the model. In this respect, it is expected that the model has been trained relatively efficiently according to the control device 620. The control unit 621 can be realized, for example, using the functions of the control unit 510 in FIG. 8 or the like.

[0110] Sixth Embodiment Fig. 11 is a diagram showing an example of a processing procedure in a learning method according to at least one embodiment. The learning method shown in Fig. 11 includes determining input values ​​(step S611), learning a decision rule (step S612), and learning a model (step S613).

[0111] In determining input values ​​(step S611), the computer determines input values ​​to be used in model training. In training decision rules (step S612), the computer trains decision rules for input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator output value and the model output value for the same input value. In training the model (step S613), the computer trains the model using the determined input values ​​and the simulator output values ​​for those input values.

[0112] According to the learning method shown in Fig. 11, it is expected that the input values ​​to be used in model learning will be determined to be input values ​​that will increase the difference between the simulator's output value and the model's output value, and the model can be learned using those input values. According to the learning method shown in Fig. 11, the model can be learned using input values ​​that will increase the error, and in this respect, the model can be learned relatively efficiently.

[0113] 12 is a diagram illustrating an example of a computer configuration according to at least one embodiment. In the configuration shown in FIG. 12, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0114] One or more of the above-described simulator device 100, predictive model device 200, learning device 300, learning device 301, control device 500, learning device 610, and control device 620, or a portion thereof, may be implemented in a computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is performed by an interface 740 having a communication function and performing communication under the control of the CPU 710. The interface 740 also has a port for a non-volatile storage medium 750, and reads information from the non-volatile storage medium 750 and writes information to the non-volatile storage medium 750.

[0115] When the simulator device 100 is implemented in the computer 700, the operation of the simulator 110 is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0116] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the simulator device 100 to perform processing in accordance with the program. Communication between the simulator device 100 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the simulator device 100 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0117] When the predictive model device 200 is implemented in a computer 700, the operation of the predictive model 210 is stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0118] Furthermore, the CPU 710, in accordance with the program, allocates a storage area in the main storage device 720 for the predictive model device 200 to perform processing. Communication between the predictive model device 200 and other devices is achieved by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the predictive model device 200 and a user is achieved by the interface 740 having an input device and an output device, which presents information to the user via the output device under the control of the CPU 710 and accepts user operations via the input device.

[0119] When the learning device 300 is implemented in a computer 700, the operations of the input value determination unit 310, the decision rule learning unit 320, and the model learning unit 330 are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0120] Furthermore, CPU 710 allocates a storage area in main memory 720 for learning device 300 to perform processing in accordance with the program. Communication between learning device 300 and other devices is performed by interface 740, which has a communication function and operates under the control of CPU 710. Interaction between learning device 300 and a user is performed by interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of CPU 710 and accepting user operations via the input device.

[0121] When the learning device 301 is implemented in a computer 700, the operations of the input value determination unit 310, the decision rule learning unit 320, the model learning unit 330, the autoencoder 340, and the autoencoder learning unit 350 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0122] Furthermore, the CPU 710 allocates a storage area in the main memory device 720 for the learning device 301 to perform processing in accordance with the program. Communication between the learning device 301 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the learning device 301 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0123] When the control device 500 is implemented in a computer 700, the operations of the prediction model 210 and the control unit 510 are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0124] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the control device 500 to perform processing in accordance with the program. Communication between the control device 500 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the control device 500 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0125] When the learning device 610 is implemented in the computer 700, the operations of the input value determination unit 611, the decision rule learning unit 612, and the model learning unit 613 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0126] Furthermore, CPU 710 allocates a storage area in main memory 720 for learning device 610 to perform processing in accordance with the program. Communication between learning device 610 and other devices is performed by interface 740, which has a communication function and operates under the control of CPU 710. Interaction between learning device 610 and a user is performed by interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of CPU 710 and accepting user operations via the input device.

[0127] When the control device 620 is implemented in the computer 700, the operation of the control unit 510 is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0128] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the control device 620 to perform processing in accordance with the program. Communication between the control device 620 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the control device 620 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0129] One or more of the above-described programs may be recorded on nonvolatile recording medium 750. In this case, interface 740 may read the programs from nonvolatile recording medium 750. Then, CPU 710 may directly execute the programs read by interface 740, or may temporarily store the programs in main storage device 720 or auxiliary storage device 730 and then execute them.

[0130] Note that programs for executing all or part of the processing performed by the simulator device 100, the predictive model device 200, the learning device 300, the learning device 301, the control device 500, the learning device 610, and the control device 620 may be recorded on a computer-readable recording medium, and the programs may be loaded into a computer system and executed to perform the processing of each unit. Note that the term "computer system" herein includes hardware such as an operating system (OS) and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, read-only memories (ROMs), and compact disc read-only memories (CD-ROMs), as well as storage devices such as hard disks built into the computer system. The programs may be programs for implementing part of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already recorded on the computer system.

[0131] Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs within the scope of the present invention. Furthermore, the above-described embodiments may be combined with other embodiments as appropriate.

[0132] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0133] (Supplementary Note 1) A learning device comprising: an input value determination means for determining input values ​​to be used in model learning; a decision rule learning means for learning a decision rule for the input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the output value of a simulator for the same input value and the output value of the model; and a model learning means for learning the model using the input value determined by the input value determination means and the output value of the simulator for that input value.

[0134] (Supplementary Note 2) The learning device according to Supplementary Note 1, wherein the decision rule learning means learns the decision rule for the input value by reinforcement learning using an output value of the simulator as a state, an input value as an action, and an evaluation function as a reward function.

[0135] (Supplementary Note 3) The learning device according to Supplementary Note 2, wherein the decision rule learning means uses a combination of an output value of the simulator and an index value relating to a degree of progress of learning of the model as the state of the reinforcement learning.

[0136] (Supplementary Note 4) The learning device according to Supplementary Note 3, further comprising an autoencoder learning means for learning an autoencoder that reconstructs the output values ​​of the simulator in accordance with the progress of learning of the model, wherein the model learning means uses the index value that uses a reconstruction error by which the autoencoder reconstructs the output values ​​of the simulator.

[0137] (Supplementary Note 5) The learning device according to any one of Supplementary Notes 1 to 4, wherein a learning rate in the learning performed by the model learning means and a learning rate in the learning performed by the decision rule learning means are set so that the learning performed by the model learning means is slower than the learning performed by the decision rule learning means.

[0138] (Supplementary Note 6) The learning device according to any one of Supplementary Notes 1 to 5, wherein the decision rule learning means uses the evaluation function in which an upper limit value of evaluation is set.

[0139] (Supplementary Note 7) The learning device according to any one of Supplementary Notes 1 to 6, wherein the decision rule learning means uses the evaluation function including a subexpression that indicates a worse evaluation as the magnitude of the difference in the input value between steps increases.

[0140] (Supplementary Note 8) A control device comprising: a control means for controlling an object simulated by a model obtained by learning using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the output value of a simulator and the output value of a model for the same input value, the model being obtained by learning using input values ​​determined using a determination rule for the input values.

[0141] (Supplementary Note 9) The control device according to Supplementary Note 8, wherein the decision rule is learned by reinforcement learning using an output value of the simulator as a state, an input value as an action, and the evaluation function as a reward function.

[0142] (Supplementary Note 10) The control device according to Supplementary Note 9, wherein the decision rule is learned by reinforcement learning using, as a state, a combination of an output value of the simulator and an index value relating to a degree of progress of learning of the model.

[0143] (Supplementary Note 11) The control device according to Supplementary Note 10, wherein an autoencoder that is trained in accordance with the progress of training of the model and that reconstructs the output values ​​of the simulator uses the index value, which uses a reconstruction error that reconstructs the output values ​​of the simulator, for training the model.

[0144] (Supplementary Note 12) The control device according to any one of Supplementary Notes 8 to 11, wherein a learning rate for training the model and a learning rate for training the decision rule are set so that the model is trained slower than the decision rule.

[0145] (Supplementary Note 13) The control device according to any one of Supplementary Notes 8 to 12, wherein the learning of the decision rule is performed using the evaluation function in which an upper limit value of evaluation is set.

[0146] (Supplementary Note 14) The control device according to any one of Supplementary Notes 8 to 13, wherein the learning of the decision rule is performed using the evaluation function including a subexpression that indicates a worse evaluation as the magnitude of the difference in the input value between steps increases.

[0147] (Supplementary Note 15) A learning method comprising: a computer determining input values ​​to be used in learning a model; learning a rule for determining the input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the output value of a simulator and the output value of the model for the same input value; and learning the model using the determined input values ​​and the output value of the simulator for those input values.

[0148] (Supplementary Note 16) The learning method according to Supplementary Note 15, wherein learning the decision rule includes the computer learning the decision rule for the input value through reinforcement learning using the evaluation function as a reward function.

[0149] (Supplementary Note 17) The learning method according to Supplementary Note 16, wherein the learning of the decision rule includes the computer using, as the state of the reinforcement learning, a combination of an output value of the simulator and an index value relating to a degree of progress of learning of the model.

[0150] (Supplementary Note 18) The learning method according to Supplementary Note 17, wherein the model learning is performed by the computer in accordance with the progress of learning of the model, and an autoencoder that reconstructs the output values ​​of the simulator uses the index value that uses a reconstruction error that reconstructs the output values ​​of the simulator.

[0151] (Supplementary Note 19) The learning method according to any one of Supplementary Notes 15 to 18, wherein a learning rate for training the model and a learning rate for training the decision rule are set so that the model is trained slower than the decision rule.

[0152] (Supplementary Note 20) The learning method according to any one of Supplementary Notes 15 to 19, wherein the learning of the decision rule includes the computer using the evaluation function in which an upper limit value of evaluation is set.

[0153] (Supplementary Note 21) The learning method according to any one of Supplementary Notes 15 to 20, wherein the learning of the decision rule includes the computer using the evaluation function including a subexpression that indicates a worse evaluation the greater the magnitude of the difference in the input value between steps.

[0154] (Supplementary Note 22) A program that causes a computer to perform the following steps: determine input values ​​to be used in model training; train a rule for determining the input values ​​using an evaluation function that includes a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value; and train the model using the determined input values ​​and the simulator's output value for those input values.

[0155] (Supplementary Note 23) The program according to Supplementary Note 22, wherein in learning the decision rule, the program causes the computer to execute learning of the decision rule for the input value through reinforcement learning using the evaluation function as a reward function.

[0156] (Supplementary Note 24) The program according to Supplementary Note 23, wherein in learning the decision rule, the program causes the computer to use a combination of an output value of the simulator and an index value relating to a degree of progress of learning of the model as a state of the reinforcement learning.

[0157] (Supplementary Note 25) The program according to Supplementary Note 24, wherein in the model learning, the program causes the computer to execute a process in which the autoencoder that is learned in accordance with the progress of the model learning and that reconstructs the output values ​​of the simulator uses the index value that uses a reconstruction error that reconstructs the output values ​​of the simulator.

[0158] (Supplementary Note 26) The program according to any one of Supplementary Notes 22 to 25, wherein a learning rate for training the model and a learning rate for training the decision rule are set so that the model is trained slower than the decision rule.

[0159] (Supplementary Note 27) The program according to any one of Supplementary Notes 22 to 26, wherein in learning the decision rule, the program causes the computer to use the evaluation function in which an upper limit value of evaluation is set.

[0160] (Supplementary Note 28) The program according to any one of Supplementary Notes 22 to 27, wherein in learning the decision rule, the program causes the computer to use the evaluation function including a subexpression that indicates a worse evaluation the greater the magnitude of the difference in the input value between steps.

[0161] This application claims priority based on Japanese Patent Application No. 2024-088232, filed May 30, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0162] The present disclosure may be applied to a learning device, a control device, a learning method, and a program recording medium.

[0163] 1, 2 Learning system 3 Control system 100 Simulator device 110 Simulator 200 Prediction model device 210 Prediction model 300, 301, 610 Learning device 310, 611 Input value determination unit 320, 612 Decision rule learning unit 330, 613 Model learning unit 340 Autoencoder 350 Autoencoder learning unit 400 Communication network 500, 620 Control device 510, 621 Control unit 910 Object

Claims

1. A learning device comprising: an input value determination means for determining input values ​​to be used in model learning; a decision rule learning means for learning decision rules for the input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value; and a model learning means for learning the model using the input values ​​determined by the input value determination means and the simulator's output value for that input value.

2. The learning device according to claim 1, wherein the decision rule learning means learns the decision rule for the input value through reinforcement learning using the output value of the simulator as a state, the input value as an action, and the evaluation function as a reward function.

3. The learning device according to claim 2, wherein the decision rule learning means uses a combination of an output value of the simulator and an index value relating to the progress of learning of the model as the state of the reinforcement learning.

4. The learning device according to claim 3, further comprising an autoencoder learning means for learning an autoencoder that reconstructs the output values ​​of the simulator in accordance with the progress of learning of the model, wherein the model learning means uses the index value obtained by using the reconstruction error with which the autoencoder reconstructs the output values ​​of the simulator.

5. A learning device according to any one of claims 1 to 4, wherein a learning rate in the learning performed by the model learning means and a learning rate in the learning performed by the decision rule learning means are set so that the learning performed by the model learning means is slower than the learning performed by the decision rule learning means.

6. The learning device according to any one of claims 1 to 5, wherein the decision rule learning means uses the evaluation function in which an upper limit value for evaluation is set.

7. The learning device according to any one of claims 1 to 6, wherein the decision rule learning means uses the evaluation function including a subexpression that indicates a worse evaluation the greater the magnitude of the difference in the input value between steps.

8. A control device having a control means for controlling an object simulated by a model obtained by learning using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value, and the model is obtained by learning using input values ​​determined using a decision rule for the input values.

9. The control device according to claim 8, wherein the decision rule is learned by reinforcement learning using the output value of the simulator as a state, the input value as an action, and the evaluation function as a reward function.

10. The control device according to claim 9, wherein the decision rule is learned by reinforcement learning using a combination of an output value of the simulator and an index value relating to the progress of learning of the model as a state.

11. The control device according to claim 10, wherein an autoencoder that learns in accordance with the progress of learning of the model and reconstructs the output values ​​of the simulator uses the index value, which uses a reconstruction error that reconstructs the output values ​​of the simulator, for learning the model.

12. The control device according to any one of claims 8 to 11, wherein a learning rate for training the model and a learning rate for training the decision rule are set so that the model is trained slower than the decision rule.

13. The control device according to any one of claims 8 to 12, wherein the learning of the decision rule is performed using the evaluation function in which an upper limit value of the evaluation is set.

14. A control device according to any one of claims 8 to 13, wherein the learning of the decision rule is performed using the evaluation function including a subexpression that indicates a worse evaluation the greater the magnitude of the difference in the input value between steps.

15. A learning method comprising: a computer determining input values ​​to be used in model learning; learning a rule for determining the input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value; and learning the model using the determined input values ​​and the simulator's output value for those input values.

16. The learning method according to claim 15, wherein the learning of the decision rule includes the computer learning the decision rule for the input value through reinforcement learning using the evaluation function as a reward function.

17. The learning method according to claim 16, wherein the learning of the decision rule includes the computer using a combination of an output value of the simulator and an index value relating to the progress of learning of the model as the state of the reinforcement learning.

18. The learning method according to claim 17, wherein the model learning is performed by the computer in accordance with the progress of the model learning, and an autoencoder that reconstructs the output values ​​of the simulator uses the index value that uses a reconstruction error that reconstructs the output values ​​of the simulator.

19. A learning method according to any one of claims 15 to 18, wherein a learning rate for training the model and a learning rate for training the decision rule are set so that the model is trained slower than the decision rule.

20. A recording medium storing a program that causes a computer to execute the following steps: determine input values ​​to be used in model training; train a rule for determining the input values ​​using an evaluation function including a subexpression that indicates a better evaluation the greater the difference between the simulator's output value and the model's output value for the same input value; and train the model using the determined input values ​​and the simulator's output value for those input values.

Images