Offline reinforcement learning method and device based on state offset correction

Abstract

The invention discloses an off-line reinforcement learning method and device based on state offset correction, and the method comprises the steps: randomly sampling a sample of a current time step from a data set of off-line reinforcement learning, and training an environment model and a state transition model according to the maximum likelihood estimation; constructing a disturbed state based on preset noise, obtaining a first state, obtained by the trained environment model based on the action, of the next moment, and obtaining a second state, obtained by the trained state transition model based on the current state, of the next moment; and updating the strategy network by using the distance between the first state and the second state, learning and training an action value network through a conservative action value function until a convergence condition is met, and generating an offline reinforcement learning model based on state offset correction. Therefore, the technical problem that the state distribution of the training strategy and the data set strategy is not matched due to the fact that the comprehensiveness of the coverage state of the data set is ignored by adopting a limiting strategy method in the related technology is solved.

Description

technical field [0001] The present application relates to the technical field of offline reinforcement learning, and in particular, to an offline reinforcement learning method and device based on state offset correction. Background technique [0002] Reinforcement learning mainly studies how the agent obtains the maximum reward, that is, it learns the optimal strategy to solve the problem under a given task. Reinforcement learning has received extensive attention because it can more intuitively model sequential decision-making problems. In recent years, with the rise of deep learning and large-scale datasets, due to the strong generalization ability of deep neural networks as function approximators, reinforcement learning uses neural networks to deal with more complex scenarios. Deep reinforcement learning has made rapid progress in video games, Go, recommender systems, and robotics. [0003] Compared with general reinforcement learning, offline reinforcement learning only...

AI Technical Summary

Problems solved by technology

[0008] This application provides an off-line reinforcement learning method and device based on state offset correction to solve the problem of using a restrictive strategy in related technologies, ignoring the comprehensiveness of the data set coverage state, resulting in inconsistent state distribution of the training strategy and the data set strategy. matching technical issues

Images