Unified reinforcement learning method based on curiosity driving

Abstract

The invention discloses a unified curiosity driving-based reinforcement learning method, which is used for enabling an intelligent agent to quickly and effectively learn a strategy under the conditionof sparse awards. The method specifically comprises the following steps of 1) obtaining feature expression with a reliable state through an attention module; 2) estimating the state and the exploration degree of a state action pair by using state novelty estimation and forward dynamic prediction, i.e., preliminarily estimated internal rewards; 3) smoothing the estimated internal rewards by usinga plurality of samples in the state space; 4) fusing different types of internal rewards to obtain more accurate and robust internal rewards; and 5) the intelligent agent learning the strategy by using experience data generated by interaction with the environment and the estimated internal rewards. The method is suitable for a sparse reward problem in the field of reinforcement learning, and the strategy can be learned quickly and effectively under the condition that the external rewards are sparse or do not exist.

Description

technical field [0001] The invention belongs to the field of reinforcement learning and is a branch of the field of machine learning, in particular to a unified curiosity-driven reinforcement learning method. Background technique [0002] The reward function is an important factor in the reinforcement learning process, and the agent learns the policy by maximizing the cumulative reward. However, in many scenarios, rewards are often sparse. For example, in a Go game, only the result of winning or losing can be received at the end, and many actions in the middle cannot be rewarded in time, which brings great challenges to reinforcement learning. . The traditional method generally combines specific tasks and manually designs the corresponding reward function, but this method requires high knowledge in the professional field, and requires cumbersome debugging, and it is difficult to transfer between different tasks. [0003] Existing curiosity-driven methods mainly estimate th...

AI Technical Summary

Problems solved by technology

However, this method does not fundamentally solve three problems: 1. Different curiosity-driven methods focus on different objects, making the estimated internal rewards unable to fully and effectively guide the agent to explore and learn; 2. Due to the state The space is very large, and there is a lot of background information irrelevant to the learning task, which will affect the agent's ability to perceive the environment, thereby affecting its learning strategy; 3. Since the value function is continuous in the state space, only one sample is used for the corresponding Estimating the degree of exploration of the state may make the estimated result inaccurate, thus affecting the efficiency of learning

Images