Robot Reality Transfer Method Based on Reinforcement Learning and Residual Modeling

Abstract

The invention discloses a robot reality migration method based on reinforcement learning and residual modeling. Under the condition of only a small amount of mixed offline data and a simulator with deviation, the offline data and the deviation simulator are used to construct an optimal simulator. Based on the optimal simulator, the robot behavior strategy is trained through the reinforcement learning algorithm and the robot can adapt to the real environment autonomously. This adaptability enables the behavior strategy to be more effectively extended to the real environment. The invention proposes a robot training framework, and provides an innovative method for reducing the deviation of robot control from the simulator environment application to the real scene. The method models the residuals of the simulator environment state space and the real environment state space based on offline data, and uses the learned residual model to modify the original simulator. Finally, the strategy learned by the robot in the correction simulator is transferred to the real environment.

Description

technical field [0001] The invention relates to a method for a robot to migrate from a simulated environment to a real environment, in particular to a method and system for robot simulation and real migration based on reinforcement learning and residual modeling, belonging to the technical field of robot control. Background technique [0002] In recent years, with the wide application of deep learning in many fields, reinforcement learning has achieved remarkable success in the simulation environment. Different from supervised learning, unsupervised learning, and semi-supervised learning, reinforcement learning mainly solves the problem of serialized decision-making. By constantly interacting with the environment, constantly updating behavior strategies and obtaining the maximum cumulative reward. Reinforcement learning environments are generally modeled as Markov decision processes ( Markov Decision Processes ), the Markov decision model is defined as < S , A , R , ...

AI Technical Summary

Problems solved by technology

These two methods have their own limitations: imitation learning needs to collect enough expert demonstration data, and it is difficult to collect high-performance expert data in reality; training reinforcement learning algorithms in simulators requires building a high-fidelity model that completely restores the real environment. True Simulators, Laborious and Expensive

Another serious and practical challenge is that there are often discrepancies between simulation and the real world, leading to strategies that work well in simulation but perform poorly in the real world

[0005] When using reinforcement learning to train robots applied in actual scenes, when the robot directly interacts with the environment and samples in the real environment, the following two serious problems will appear: low sampling efficiency and safety issues

However, a serious and practical challenge is that there are often discrepancies between simulation and the real world, leading to strategies that work well in simulation but perform poorly in the real world

Images