Reinforcement learning method based on mixed behavior space

Abstract

The invention discloses a reinforcement learning method based on a mixed behavior space, and relates to the field of reinforcement learning, and the invention consists of a plurality of parallel Actornetworks which jointly act to output structured behaviors, and a Critic network which guides the training of the Actor networks. The Actor network comprises a state coding network, a discrete Actor network and a continuous parameter Actor network, the state coding network codes the state and inputs the state into the discrete Actor network and the continuous parameter Actor network, the discreteActor network is used for generating discrete actions, and the continuous parameter Actor network is used for generating continuous parameters corresponding to the discrete actions. According to the invention, the mixed behavior space with continuous actions and discrete actions can be processed, and the method can be expanded to all behavior spaces with hierarchical structures. According to the invention, a reinforcement learning result better than that of a previous mixed behavior space processing method can be obtained, the accuracy of behaviors is not lost, and the problem of over-parameterization is avoided through mask operation.

Description

technical field [0001] The invention relates to the field of reinforcement learning, in particular to a reinforcement learning method based on a mixed behavior space. Background technique [0002] Representation and learning of complex strategies in reinforcement learning refers to the problem of how to represent strategies and learn end-to-end when the strategy is relatively complex in reinforcement learning. The present invention is mainly aimed at the problem of mixed behavior space, that is, the behavior has a part of discrete selection and a part of continuous parameters. For example, in an automatic driving task, whether to turn the steering wheel or brake in this step is a discrete action choice. If you turn the steering wheel, what is the corresponding angle? This is the action selection of continuous values. Most of the current reinforcement learning algorithms are aimed at pure discrete behavior spaces or pure continuous behavior spaces, and there are few algorith...

AI Technical Summary

Problems solved by technology

But in the update process, the goal is to maximize all Q values ​​instead of the largest Q value, which will lead to over-parameterization and cause some unnecessary training

[0008] The following conclusions can be drawn from the analysis of related research at home and abroad: At present, there are some loopholes in the existing methods of reinforcement learning algorithms in mixed behavior spaces, such as loss of accuracy, lack of theoretical support, over-parameterization, etc. A relatively complete and general reinforcement learning algorithm for mixed behavior spaces has not been proposed

Images