Robot imitation learning method based on virtual scene training

Abstract

The invention discloses a robot imitation learning method based on virtual scene training. The method comprises the following steps: designing a robot model and a virtual interaction environment according to a specific task; collecting and arranging an expert data set; determining a state value space S and an action value space A according to the specific task, and determining structures of the network of a strategy generator and the network of a discriminator according to the state value space S and the action value space A; sampling data from the strategy generator, designing a parameter updating strategy, and alternately training the strategy generator and the discriminator by combining the expert data set and adopting an adversarial training method until the discriminator converges toa saddle point; and testing a network model composed of the strategy generator and the discriminator obtained by training, and taking a real environment state as input of the strategy generator so asto obtain action output. According to the method, a value return function is judged and learned; a large number of complex intermediate steps of inverse reinforcement learning with high calculation amount are bypassed; and the learning process is simpler and more efficient.

Description

technical field [0001] The invention belongs to the technical field of imitation learning and artificial intelligence, in particular to a robot imitation learning method based on virtual scene training. Background technique [0002] In traditional reinforcement learning tasks, the optimal policy is usually learned by calculating cumulative rewards. This method is simple and straightforward, and has better performance when more training data is available. However, in the multi-step decision task (sequential decision), the learner cannot be rewarded frequently, and there is a huge search space based on the cumulative reward and learning method. At the same time, reinforcement learning requires a suitable reward function to solve the optimal action strategy, but in many cases it is not easy to design a sufficiently comprehensive and excellent reward function, especially in some complex application scenarios, such as automatic driving collision It is difficult to have a reasona...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to solve complex skill learning tasks that require multi-step decision-making. In these tasks, the learner cannot be rewarded frequently. Common reinforcement learning algorithms need a suitable reward function to solve the optimal action strategy. In many cases It is extremely complicated and unrealistic to design a sufficiently comprehensive and excellent reward function

Therefore, the present invention uses imitation learning and inverse reinforcement learning to solve such problems. In order to avoid the common behavior cloning method and inverse reinforcement learning algorithm with large amount of calculation, difficult network design, and complicated intermediate calculation process, the present invention introduces The idea of ​​GAN directly learns the distribution of the reward function, bypassing many intermediate steps in reverse reinforcement learning, especially repeated reinforcement learning calculations, thus reducing the amount of calculation.

Images