Robot reality migration system and method based on reinforcement learning and residual error modeling

Abstract

The invention discloses a robot reality migration system and method based on reinforcement learning and residual modeling, and the method employs offline data and a deviation simulator to construct an optimal simulator under the condition that there is only a small amount of mixed offline data and a deviation simulator. A robot behavior strategy is trained through a reinforcement learning algorithm based on the optimal simulator, and the robot can be enabled to adapt to a real environment autonomously. The self-adaptability enables the behavior strategy to be popularized to the real environment more effectively. The invention provides a robot training framework, and provides an innovative method for reducing the deviation of robot control from simulator environment application to real scenes. According to the method, modeling is performed on a residual error of a simulator environment state space and a real environment state space based on offline data, and an original simulator is corrected by using a learned residual error model. And finally, a strategy learned by the robot in the correction simulator is migrated to a real environment.

Description

technical field [0001] The invention relates to a method for a robot to migrate from a simulation environment to a real environment, in particular to a robot simulation and reality migration method and system based on reinforcement learning and residual modeling, belonging to the technical field of robot control. Background technique [0002] In recent years, with the wide application of deep learning in many fields, reinforcement learning has achieved remarkable success in the simulation environment. Different from supervised learning, unsupervised learning, and semi-supervised learning, reinforcement learning mainly solves serialized decision-making problems. Through continuous interaction with the environment, the behavior strategy is continuously updated to obtain the maximum cumulative reward. Reinforcement learning environments are generally modeled as Markov decision processes ( Markov Decision Processes ), the Markov decision model is defined as < S , A , R , ...

AI Technical Summary

Problems solved by technology

These two methods have their own limitations: imitation learning needs to collect enough expert demonstration data, and it is difficult to collect high-performance expert data in reality; training reinforcement learning algorithms in simulators requires building a high-fidelity model that completely restores the real environment. True Simulators, Laborious and Expensive

Another serious and practical challenge is that there are often discrepancies between simulation and the real world, leading to strategies that work well in simulation but perform poorly in the real world

[0005] When using reinforcement learning to train robots applied in actual scenes, when the robot directly interacts with the environment and samples in the real environment, the following two serious problems will appear: low sampling efficiency and safety issues

However, a serious and practical challenge is that there are often discrepancies between simulation and the real world, leading to strategies that work well in simulation but perform poorly in the real world

Images