Learning device, control system, learning method and program

Claims

1. A policy generation means that generates a policy, which is a rule for determining the behavior, based on a reward value, which is a value indicating an evaluation of the controlled behavior in a state where time steps are reversed within an episode representing the learning period. Action determination means for determining the action of the controlled object based on the policy, A learning device equipped with the following features.

2. A reset determination means determines that a predetermined condition for interrupting the simulation of the controlled object's behavior has been met, and interrupts the simulation. Furthermore, The policy generation means generates the policy based on the reward value calculated based on the simulation. The learning device according to claim 1.

3. The reset determination means updates the threshold value, which is the cumulative value of instantaneous reward values, so that the magnitude of deterioration in evaluation indicated by the threshold value increases as the simulation progresses. If it determines that the evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold, it decides to interrupt the simulation. The learning device according to claim 2.

4. The reset determination means updates the threshold value, which is the predicted value of the value function value (which is the cumulative value of instantaneous reward values), so that the magnitude of deterioration in evaluation indicated by the threshold value increases as the simulation progresses. If it determines that the evaluation indicated by the value function value has deteriorated below a predetermined threshold, it decides to interrupt the simulation. The learning device according to claim 2.

5. Restart state determination means for determining the state of the environment in which the controlled object acts when resuming an interrupted simulation. The learning device according to claim 2, further comprising:

6. The restart state determination means selects one of the reset destination candidates based on a probability distribution set for a reset destination candidate, which is an executed time step in the episode that was running up to the time the simulation was interrupted, in which the state is stored as a time step in which the simulation can be restarted. The learning device according to claim 5.

7. The restart state determination means selects one of several reset destination candidates according to a uniformly distributed probability distribution. The learning device according to claim 6.

8. Equipped with a learning device and a control device, The learning device is A policy generation means that generates a policy, which is a rule for determining the behavior, based on a reward value, which is a value indicating an evaluation of the controlled behavior in a state where time steps are reversed within an episode representing the learning period. Action determination means for determining the action of the controlled object based on the policy, Equipped with, The control device performs control on the controlled object based on the strategy obtained using the learning device. Control system.

9. Computers Based on the reward value, which is a value indicating the evaluation of the controlled behavior in the time step backward within the episode representing the learning period, a policy, which is the rule for determining the behavior, is generated. The action of the controlled object is determined based on the policy. A learning method that includes this.

10. On the computer, Based on the reward value, which is a value indicating the evaluation of the controlled behavior in the time step backward within the episode representing the learning period, a policy, which is a rule for determining the behavior, is generated. The action of the controlled object is determined based on the policy, A program to execute.