Reinforcement learning apparatus, control apparatus, and reinforcement learning method

Abstract

It is possible to perform robot motor learning in a quick and stable manner using a reinforcement learning apparatus including: a first-type environment parameter obtaining unit that obtains a value of one or more first-type environment parameters; a control parameter value calculation unit that calculates a value of one or more control parameters maximizing a reward by using the value of the one or more first-type environment parameters; a control parameter value output unit that outputs the value of the one or more control parameters to the control object; a second-type environment parameter obtaining unit that obtains a value of one or more second-type environment parameters; a virtual external force calculation unit that calculates the virtual external force by using the value of the one or more second-type environment parameters; and a virtual external force output unit that outputs the virtual external force to the control object.

Description

[0001]This application claims priority under 35 U.S.C. §119 to Japanese Patent Application No. 2011-074694, filed Mar. 30, 2011, which is incorporated by reference.BACKGROUND OF THE INVENTION[0002]1. Field of the Invention[0003]The present invention relates to a reinforcement learning apparatus and the like that perform robot motor learning.[0004]2. Description of Related Art[0005]Reinforcement learning has been widely used as a robot motor learning technique because it can be implemented even if the dynamics of the control object or the environment are unknown and it autonomously performs learning by simply setting a reward function according to the task (see, for example, JP 2007-66242A).[0006]Conventional techniques, however, have a problem in that the reward function for a complex motion trajectory is often expressed by the sum of various terms, and a trade-off occurring between the terms impedes learning (this is called the “trade-off problem”). For example, the reward function...

AI Technical Summary

Benefits of technology

[0008]With this configuration, the requirements for obstacle avoidance are solved with a virtual external force, and therefore it is possible to perform robot motor learning with a simple reward function in a quick and stable manner.

[0025]With a reinforcement learning apparatus according to the present invention, it is possible to perform robot motor learning in a quick and stable manner.

Problems solved by technology

Conventional techniques, however, have a problem in that the reward function for a complex motion trajectory is often expressed by the sum of various terms, and a trade-off occurring between the terms impedes learning (this is called the “trade-off problem”).

If the ratio of these two elements is not set appropriately, then the speed of learning results is extremely increased or decreased, resulting in undesirable motion trajectories.

This trade-off problem becomes more challenging if requirements such as obstacle avoidance, in addition to a reaching movement, are further imposed.

Too small a negative reward given upon contact with an obstacle results in collision of the robot arm with the obstacle, and too large a negative reward leads to a learning result in which the robot arm does not move from the starting point.

When the reward function has become too complex, the designer has to empirically adjust the balance between the elements, compromising the advantage of reinforcement learning.

Images