Multi-target robot control method based on dynamic model and after-event experience playback

Abstract

The invention discloses a multi-target robot control method based on a dynamic model and after-event experience playback. According to the method, a strategy for completing the whole target space canbe learned, and the method is more advantageous in generalization than a conventional method. According to the method, the data utilization efficiency in multi-target reinforcement learning is improved through model-based value function estimation and after-event experience playback. Compared with other off-line value function estimation methods without off-line deviation, although a model error exists, the method balances the model error and the learning speed through weighted summation of single-step and multi-step value function estimation and has better performance.

Description

technical field [0001] The invention relates to the technical field of robot control, in particular to a multi-objective robot control method based on dynamic models and post-event experience playback. Background technique [0002] Reinforcement learning: A class of methods in machine learning, mainly composed of two parts: agent and environment. According to the current state, the agent executes actions according to a certain strategy (policy) and acts on the environment. After the environment receives the action, it will return a new state and a reward (reward). [0003] Deep reinforcement learning: The combination of deep neural network and reinforcement learning enables reinforcement learning to effectively solve complex problems in large state spaces and even continuous state spaces. Robot control is a continuous state space control problem. [0004] Multi-objective reinforcement learning: The usual reinforcement learning is to accomplish a specific goal, but there are...

AI Technical Summary

Problems solved by technology

The problem with the dynamic model-based method is that when the dynamic model does not fit the real environment enough, there will be model errors, and as the number of interaction steps accumulates, it may also bring harm to the training of the agent.

[0012] (1) The existing technology usually needs to train a policy network to complete a specific task, and its generalization is not strong. However, our multi-objective reinforcement learning technology can train a large number of targets at the same time, and one model can complete all tasks in a certain target space. ;

[0013] (2) Existing technologies do not utilize post-event experience replay information, and often cannot learn from failure data, while our technology uses post-event experience replay to improve the utilization of failure data and accelerate the training of robot tasks;

[0014] (3) The existing technology does not use the value function expansion method based on the model, and often uses a single-step temporal difference method to learn, while our technology can accelerate the learning of the value function and the training of the agent;

[0015] (4) The multi-step value function estimation method in the prior art has a large offline deviation in the case of an offline strategy, and the method of this patent does not have an offline deviation due to the use of a model-based value function expansion, but there is a certain model error

Images