Reinforcement learning method and device for sparse reward environment, equipment and medium

Abstract

The invention discloses a reinforcement learning method, device and equipment for a sparse reward environment and a medium; the method comprises the steps: carrying out the interaction of an action with a plurality of current environment states, and obtaining a plurality of environment states at a next moment; calculating the similarity of the environment state at the next moment to obtain a similarity matrix; judging whether the current environment state is influenced by random noise or not according to the similarity matrix; if the current environment state is influenced by the random noise, calculating an internal reward value through a preset environment familiarity model; and performing strategy learning according to empirical data generated by interaction with the environment and the calculated internal reward value. According to the reinforcement learning method provided by the embodiment of the invention, the strategy can be quickly and effectively learned under the condition that the external rewards are relatively sparse or do not exist.

Description

technical field [0001] The present invention relates to the technical field of artificial intelligence, in particular to a reinforcement learning method, device, equipment and medium for sparse reward environments. Background technique [0002] Everything that interacts with the agent in reinforcement learning is called the environment. The environment can provide the agent with a state, the agent makes decisions based on the state, and the environment feeds back to the agent for rewards. However, in realistic reinforcement learning tasks, many rewards are sparse, and the environment cannot provide timely feedback to the reward value according to each decision of the agent. What's more, rewards can only be obtained in the final state, such as Go, Montezuma Revenge and other games. [0003] The sparse reward problem will cause the reinforcement learning algorithm to iterate slowly, and even difficult to converge. At present, for such tasks with sparse rewards, the methods a...

AI Technical Summary

Problems solved by technology

However, reward reshaping and reconstructing the reward value is not universal. Experience replay is only suitable for offline algorithms. There are curiosity-based methods in exploration and utilization. Curiosity-based exploration methods construct a state prediction model, and predict the next The difference between the state and the actual next state is used to measure the curiosity of the environment, which is regarded as an intrinsic reward, but the model will lose its meaning due to the influence of random noise, making the next state unpredictable and irrelevant to the decision of the agent

Images