A reinforcement learning-based continuous action control method based on discrete-continuous heterogeneous Q-networks

Abstract

This invention relates to a reinforcement learning-based continuous action control method based on discrete-continuous heterogeneous Q-networks, belonging to the field of reinforcement learning technology, and particularly to a reinforcement learning control method based on value functions. The reinforcement learning method designed in this invention, based on a continuous-heterogeneous Q-network architecture, fully utilizes the characteristics of this architecture. While efficiently utilizing empirical samples for offline updates, it employs an ∈-Gaussian composite exploration method to improve the exploration efficiency of reinforcement learning. Compared to commonly used policy-evaluation structure reinforcement learning methods, it has a faster convergence speed and a more stable training process, significantly improving the control effect of reinforcement learning controllers applied to unmanned equipment.

Description

Technical Field

[0001] This invention relates to a reinforcement learning continuous action control method based on discrete-continuous heterogeneous Q-networks, belonging to the field of reinforcement learning technology, and particularly to a reinforcement learning control method based on value functions. Background Technology

[0002] Control technology is one of the most relied-upon technologies in modern human society, permeating every corner of people's lives. From home appliances to industrial equipment, transportation vehicles, power grid systems, and even aerospace technology, control technology is indispensable. In recent years, with the continuous development of artificial intelligence technology, more and more intelligent products have begun to enter society. While these intelligent products bring great convenience to people's lives, they also place higher demands on control technology. The most representative examples are various unmanned devices, such as intelligent self-driving cars, service robots, and bionic robots. The control problems of these devices are often high-dimensional, time-varying, and nonlinear, making it difficult for traditional methods to analyze these systems and provide reasonable control laws. In addition, these intelligent devices have different control requirements in different usage scenarios. For example, when a robotic arm grasps and pushes an object, the control laws used are different due to the different targets. Designing different control laws for different targets using traditional methods would consume a lot of human and material resources.

[0003] Therefore, researchers in related fields have begun to explore various intelligent control methods to replace traditional control methods, aiming to achieve autonomous intelligent design of control laws for unmanned equipment. This aims to save labor costs while improving the control performance and multi-tasking capabilities of unmanned equipment. Related research includes fuzzy control, expert control, model predictive control, neural network control, and reinforcement learning control; among these, reinforcement learning-based intelligent control is one of the most promising methods. This type of method is based on Markov decision processes, fitting an optimal policy by maximizing the objective reward function, thereby achieving stable control of unmanned equipment. Reinforcement learning-based control methods have proven effective in multiple real-world scenarios, such as the robotic arm folding fabric implemented by the School of Computing at Imperial College London in 2018, and the robotic arm solving a Rubik's Cube implemented by the artificial intelligence research institution OpenAI in 2019.

[0004] However, reinforcement learning-based unmanned equipment control methods still face significant challenges in generalization and application. The most difficult aspect is the acquisition of experience from real-world systems. Current reinforcement learning algorithms still suffer from insufficient training efficiency. Even with optimization methods such as simulation training combined with transfer learning to reduce the required number of experience samples, training a single reinforcement learning agent's control policy still requires at least tens of thousands of real-world samplings. This massive demand for real-world experience samples translates to high time and economic costs, severely hindering the widespread adoption of reinforcement learning control technology in real-world unmanned equipment. Furthermore, the policies trained using current reinforcement learning methods remain unstable, making it difficult to guarantee the desired performance in real-world applications.

[0005] To date, researchers in related fields have largely adopted similar design principles for reinforcement learning controllers: selecting a suitable core reinforcement learning method and then applying deep learning techniques to improve and adapt it, enabling the original method to be applied stably and efficiently to the target task. In the field of reinforcement learning, mainstream methods can be divided into three categories: value-based reinforcement learning methods, policy gradient-based reinforcement learning methods, and policy-critic (Actor-Critic)-based reinforcement learning methods. Among these, value-based reinforcement learning methods have mature, stable, and efficient performance, and are relatively well-developed. A representative method is the Deep Q-Network (DQN) algorithm proposed by Google's AI research institute DeepMind in 2012, and its related improved algorithms. This type of algorithm is designed for reinforcement learning tasks with discrete action spaces, such as video games, and cannot be directly applied to continuous action control problems. While policy gradient-based reinforcement learning methods can handle continuous actions in control problems, their use is currently limited due to the low efficiency of utilizing empirical samples.

[0006] Currently, the most suitable reinforcement learning methods for unmanned equipment control tasks are those based on the policy-evaluation framework. The main characteristic of this type of method is that one network fits the value function of reinforcement learning, while another network optimizes the policy based on the evaluation result of this value function. The most representative policy-evaluation methods include the online reinforcement learning method "Proximal Policy Optimization (PPO)" proposed by Schulman et al., and the offline reinforcement learning method "Deep Deterministic Policy Gradient (DDPG)" proposed by Lilicrap et al. DDPG, due to its ability to store and replay experience, is far more efficient than online methods and is therefore more suitable for reinforcement learning control tasks where experience acquisition is difficult. The original version of DDPG has poor stability, often exhibiting non-convergence and training failures in some complex tasks. Several improved versions of this algorithm have mitigated this deficiency to some extent and also improved training efficiency. The most widely used are the dual-delay deep deterministic policy gradient method (TD3) proposed by Fujimoto et al. in 2018, and the flexible policy-evaluation method (SAC) proposed by Haarnoja et al., also in 2018. These two algorithms have shown good performance in reinforcement learning control of some unmanned equipment.

[0007] However, due to some inherent problems in the policy-evaluation framework, the above two algorithms still perform poorly in certain scenarios. Value-based reinforcement learning methods can directly express the policy through value functions, while the policy-evaluation structure requires a policy network to learn the continuous Q-function of the evaluation network through policy gradients. This deep coupling between the two networks significantly reduces the overall stability of the algorithm. Furthermore, since the controller's output has an upper limit, the corresponding policy network must use a Tanh layer as its activation function to restrict actions within a certain range. In actual training, the Tanh layer easily leads to the vanishing gradient problem, affecting the overall training efficiency of reinforcement learning.

[0008] To avoid a series of problems inherent in policy-evaluation algorithms, some reinforcement learning researchers have attempted to improve the DQN method to make it applicable to control tasks in continuous action spaces. The biggest problem with DQN for continuous control lies in the dimensionality explosion caused by action discretization. This problem was illustrated by Lilicrap et al. in their work. Because DQN's discretization of actions is based on the entire action space rather than a specific action dimension, the discretization process of the continuous action space requires traversing and combining the discretized actions of each action dimension. This results in the number of discretized actions corresponding to the continuous action space being exponentially related to the dimensionality, making DQN unsuitable for this scenario. To address this problem, some related works have attempted to discretize each dimension of the action space independently and train DQN independently or sequentially. Among these methods, BDQ proposed by Tavakoli et al. has shown good performance, effectively solving some continuous control tasks, but its efficiency and stability still do not meet the standards for practical applications. Summary of the Invention

[0009] The technical problem solved by this invention is to overcome the shortcomings of existing technologies and propose a reinforcement learning-based continuous action control method based on discrete-continuous heterogeneous Q-networks. This method, based on the independent discretization of the action space in each dimension, designs a discrete-continuous heterogeneous Q-network architecture and its corresponding reinforcement learning training method, solving the problem of mutual coupling between Q-functions in different action dimensions, and significantly improving the training efficiency and final-state performance of value function reinforcement learning for continuous control problems. This architecture includes two different Q-network structures: a continuous evaluation Q-network and a discrete action Q-network. The continuous evaluation Q-network takes the state and continuous actions as input and outputs corresponding state-behavior values, updated using temporal difference, and mainly provides mentor signals to the discrete Q-network. The discrete action Q-network functions similarly to the Actor network in a policy-evaluation structure, taking the state as input and outputting the state-behavior values ​​corresponding to discrete actions in each dimension. It follows the value function of the continuous evaluation Q-network using supervised learning and can output actions through a greedy policy. The reinforcement learning method based on the continuous-heterogeneous Q-network architecture designed in this invention makes full use of the characteristics of this architecture. While efficiently utilizing experience samples for offline updates, it adopts an ∈-Gaussian composite exploration method to improve the exploration efficiency of reinforcement learning. Compared with the commonly used policy-evaluation structure reinforcement learning method, it has a faster convergence speed and a more stable training process, which significantly improves the control effect of reinforcement learning controllers applied to unmanned equipment.

[0010] The technical solution of this invention is:

[0011] A reinforcement learning-based continuous action control method based on discrete-continuous heterogeneous Q-networks, comprising the following steps:

[0012] The first step is to construct a continuous-discrete valued function reinforcement learning model;

[0013] The main method of action selection in value function reinforcement learning is to compare the state-behavior values ​​corresponding to different actions and select the action with the maximum state-behavior value. However, the continuous state space has an infinite number of actions, making comparison impossible. Therefore, discretization is necessary to select the optimal action from a discrete state space with a finite number of actions. For the control problems of robots and other unmanned devices with high dimensionality of continuous actions, if the actions are discretized centrally, the number of actions in the discrete action space is exponentially related to the dimensionality of the continuous actions, making it infeasible to compare the state-behavior values ​​of each discrete action. This invention, however, adopts an independent discretization method for each action dimension, setting the value function and discrete action space for each action dimension separately, effectively solving the problem of excessive number of actions in the discretization of continuous actions.

[0014] This invention employs a continuous-discrete dual-valued function approach to train reinforcement learning. Therefore, reinforcement learning models corresponding to both discrete-valued and continuous-valued functions are constructed. The reinforcement learning model corresponding to the continuous-valued function is defined on the global action and adopts the standard paradigm of a Markov decision process. This model consists of Markov quintuples:

[0015]

[0016] in Let X be the state space, containing all possible state observations in the reinforcement learning control task. For most control tasks, this state space exists in a continuous form; therefore, this invention defines it on an X-dimensional continuous real space, i.e. The state variable is denoted as s(t), which represents the state at time t. This refers to the robot's action space. For reinforcement learning models corresponding to continuous-valued functions, this action space is continuous and defined on an M-dimensional continuous real space, i.e. The continuous action quantity it contains is denoted as a(t), representing the continuous action at time t. p is the state transition probability p(s(t+1)|s(t), a(t)), which represents the state distribution of the current state and action at the next time step, corresponding to the state transition model of the control problem. In the model established in this invention, p is unknown. r is the reward function r(s(t), a(t)) for reinforcement learning, used to evaluate the performance of the reinforcement learning policy within one step; γ is the time discount rate of the reward function. The goal of continuous-valued function reinforcement learning is to find an optimal deterministic policy μ(s(t)) to maximize the following cumulative reward value:

[0017]

[0018] In other words, the optimal deterministic policy should be able to select an action that yields the highest expected cumulative reward. Under the action a(t) = μ(s(t)), the expected cumulative reward achievable by the model is also called the state-behavior value or Q-function value, which is calculated in continuous models using the following formula:

[0019]

[0020] To obtain the highest cumulative reward, the optimal deterministic policy of a continuous Q-function reinforcement learning model is defined as follows: selecting the action corresponding to the maximum state action value.

[0021]

[0022] The reinforcement learning model for discrete Q-functions is derived from the reinforcement learning model for continuous Q-functions, and is defined over each continuous action dimension of the controlled object (controllable units such as motors). For dimension m, the discrete Q-function reinforcement learning model consists of the following Markov quintuple:

[0023]

[0024] in The definition of γ is the same as that of the reinforcement learning model for continuous-valued functions. Defined as the discrete state space after discretizing the continuous action dimension m individually:

[0025]

[0026] in This represents the nth discrete action in the m-th action dimension. All action dimensions have the same discretization precision, each being discretized into N actions.

[0027] p m It is the single-step state transition probability p for dimension m. m (s(t+1)|s(t), a m (t)), using only the global state s(t) and the one-dimensional action a m (t) is the input, representing the strategies for dimensions other than m. Then it is considered as a parameter of the transition probability, which is defined as:

[0028]

[0029] At the transition probability p m Except for dimension m, actions in other dimensions are directly determined by the set of strategies for those dimensions. Given. p mIt is constructed based on the transition probability p of the continuous Q-function reinforcement learning model. The following coupling relationship is defined between the two:

[0030]

[0031] r in the quintuple m For a one-dimensional reward function r m (s(t), a(t)), similarly using As a parameter of the transition probability function. m It is constructed based on the reward function r of the continuous model, and the following coupling relationship is defined between the two in this invention:

[0032]

[0033] Based on the coupling relationship between the two definitions above, in a one-dimensional discrete reinforcement learning model, the reward value and the probability of the next state are only related to the state and the action in dimension m. Actions in other dimensions are determined by the given policy. Automatically generated. Due to the strategies in the two coupling relationships mentioned above. Since it changes over time, the single-dimensional discrete reinforcement learning model also changes over time.

[0034] The goal of the above one-dimensional discrete reinforcement learning is to find an optimal one-dimensional discrete policy μ. m (s(t)), thereby maximizing the following cumulative return:

[0035]

[0036] The state-behavior value function of the discrete model, also known as the Q-function, is calculated as the policy μ in the following form. m (s(t)) is the reference for selecting the optimal action. The Q-function is also the fitting target for the discrete Q-network:

[0037]

[0038] To optimize the cumulative reward of the discrete Q-function reinforcement learning model, a single-dimensional discrete policy μ is used. m (s(t)) selects the action corresponding to the maximum discrete Q-function value:

[0039]

[0040] In the discrete model of this invention, discrete action a m (t) There are only N values ​​for each dimension, hence the corresponding state-behavior values There are only N discrete strategies μ m(s(t)) can be directly chosen from among N Q-functions, selecting the policy corresponding to the largest Q-function value. However, for continuous models, the number of continuous actions a(t) is infinite, making it impossible to select the optimal action through Q-value comparison. The optimal continuous policy cannot actually be given. Furthermore, due to the state transition probability distribution p in a one-dimensional discrete reinforcement learning model… m It will change as the strategy iterates, therefore We cannot directly use previously collected experience samples for iterative fitting because these experiences do not conform to the current p. m This approach cannot accurately reflect state transitions. Therefore, this invention is based on the reward value function r of discrete and continuous models. m The coupling relationship between r and the discrete and continuous state-behavior value functions is given below. This allows the continuous model to apply the optimal strategy of the discrete model to select actions, while the discrete model can select actions based on the state-behavior value function of the continuous model. Upgrade:

[0041]

[0042] Under this coupling relationship, the optimal policy μ(s(t)) of the continuous model can be approximated by the optimal policy of the discrete model, and the discrete policy, after being made continuous, can be used as the approximate optimal policy of the continuous model:

[0043] μ(s(t)):={μ1(s(t)),μ2(s(t)),...,μ M (s(t))}

[0044] Thus, the construction of the continuous-discrete valued function reinforcement learning model is complete. Based on the above model and the coupling and policy transformation relationships between the two models, the neural network model of step 2 can be built. The discrete model can be fitted using a deep Q-network, while the continuous model adopts a policy-critic (Actor-Critic) architecture network model. Because the optimal policy of the continuous model is approximated by the optimal policy of the discrete model, the policy network of the policy-critic architecture is not constructed separately, but is replaced by the deep Q-network of the discrete model.

[0045] 2. Constructing a neural network architecture for reinforcement learning using discrete-continuous heterogeneous Q-networks.

[0046] To efficiently fit the above discrete-continuous valued function reinforcement learning model and obtain the optimal policy required for the control problem, this invention specifically designs the following deep reinforcement learning neural network architecture, which consists of five neural networks and can be divided into two modules: a continuous Q-network and a decomposed discrete Q-network.

[0047] The continuous Q-network module adopts a dual-network structure similar to double DQN, consisting of four networks: two evaluation Q-networks and two target Q-networks, which are defined as follows:

[0048] ①Evaluate Q network 1, using The network parameters are as follows: This network employs a multilayer perceptron structure with two hidden layers. The input layer has X+M neurons (state dimension + continuous action dimension), and the output layer has one neuron. The number of neurons in the hidden layers is an adjustable hyperparameter, with a default of 512. The main purpose of this network is to fit the state-behavior values ​​in a continuous-valued function reinforcement learning model based on the state and action information of the control system.

[0049] ②Evaluate Q-network 2, using The network parameters are as follows: This network has the same multilayer perceptron structure as evaluation network 1. In this invention, the actual value of the state-behavior value estimate is the smaller of the outputs of the two evaluation networks:

[0050]

[0051] In the value function fitting phase of reinforcement learning training, overestimation of the state-behavior value function is a significant factor affecting training speed and final performance. The above-described dual-network output minimization operation can effectively suppress the overestimation problem and improve the performance of the reinforcement learning controller.

[0052] ③Target Q network 1, using The network parameters are as follows: In this invention, the network is the target network of the evaluation Q network 1, with the same structure. Its function is to estimate the target state-behavior value function in the temporal difference, so that the Q function value to be optimized and the target Q function value are decoupled during the evaluation network update process.

[0053] ④Target Q network 2, using The network parameters are as follows: This network serves as the target network for evaluation network 2, and its main function is the same as that of target evaluation network 1. The parameters of both networks are updated softly to follow the parameters of their respective evaluation networks.

[0054]

[0055] Where τ is the target network soft update hyperparameter, with a default value of 0.005. The lower this parameter is, the more stable the reinforcement learning training process will be, but setting it too low will affect training efficiency.

[0056] When evaluating the objective Q-function, this invention also adopts the form of taking the smaller value of the dual-objective network output:

[0057]

[0058] The decomposed discrete Q-network module contains one decomposed discrete Q-network. This network adopts a "decomposition" structure, outputting the discrete state-behavior value function corresponding to each action dimension, and the Q-functions of different dimensions are independent. In this invention, this network uses... It means that the parameter is θ d The discrete Q-network decomposes using a multilayer perceptron network structure. The input layer has X neurons, meaning the input is the state variable. The network has two hidden layers, with the number of neurons in the hidden layers being a hyperparameter (default 512). For control problems with higher dimensions, the number of hidden layer neurons can be increased appropriately. The output layer has M×N neurons, outputting a vector of size M×N (action dimension × number of discrete actions per dimension), which is then reconstructed into an M-row, N-column output matrix. Each row of this matrix contains the state-behavior values ​​of all discrete actions in one action dimension. The element in the m-th row and n-th column is... The state-behavior value corresponding to the nth discrete action in the mth action dimension:

[0059]

[0060] Since the output corresponds to a discrete state-behavior value function, the decomposed discrete Q-network can also be used as a policy network, selected in a greedy manner. The discrete action corresponding to the maximum value in each row:

[0061] a(f)=μ(s(t); θ d )={μ1(s(t); θ d ), μ2(s(t); θ d )…μ M (s(t;θ) d ))

[0062]

[0063] Unlike continuous Q-network modules which include a target network, setting a target network for decomposing discrete Q-networks does not significantly improve performance in practical applications. To conserve computational resources, the discrete Q-network decomposition module does not include a target network, and a single network structure is sufficient to achieve good results. This completes the construction of the neural network architecture for reinforcement learning using discrete-continuous heterogeneous Q-networks.

[0064] 3. Train the neural network from step 2.

[0065] The method of this invention is similar to existing model-free offline policy reinforcement learning methods in its overall training process framework. The main difference lies in the decomposition of supervised learning of discrete Q-networks and the ∈-Gaussian exploration policy in the interaction process. The reinforcement learning method based on discrete-continuous heterogeneous Q-networks mainly includes the following steps in the training process:

[0066] (1) Initialize the continuous Q network parameters, decompose the discrete Q network parameters, learn the hyperparameters such as learning rate, reward discount rate, and exploration rate, and store the experience pool of samples.

[0067] (2) Environment Interaction. The method of this invention mainly involves simultaneous environment interaction and strategy optimization, continuously collecting environment interaction experience samples using new strategies and storing them in an experience pool for continuous Q-network to iteratively update the Q-function. To enable the Q-function of this invention to converge faster, an ∈-Gaussian exploration strategy is adopted during the environment interaction process to improve the efficiency of environment exploration. This strategy is represented in this invention as follows: The interaction with the environment is mainly achieved through the following steps:

[0068] ① Based on the strategy of decomposing the discrete Q-network and maximizing the Q-function value, the discrete action a(t) = μ(s(t)) corresponding to the state s(t) is obtained; θ d );

[0069] ② Apply the ∈-greedy strategy to generate exploration actions:

[0070]

[0071] That is, to randomly select an action with a probability of ∈.

[0072] ③ The exploration actions are made continuous, and a Gaussian noise is superimposed to obtain the exploration actions under the ∈-Gaussian exploration strategy, where σ is the noise standard deviation hyperparameter, which can generally be set to 0.05 or 0.1:

[0073]

[0074] ④ The application is applied to the controller, which interacts with the environment to obtain the next moment's action:

[0075]

[0076]

[0077] After completing the above environmental interaction steps, the single-step state transition experience samples are stored in the experience pool. These experience samples include {s(t), a(t), r}. t , s(t+1)} quadruple.

[0078] (3) Experience Sampling: Once the experience pool contains a sufficient number of experiences (generally set to 1024 in this invention), a batch of experiences is sampled from the experience pool. The number of experiences sampled in each batch can be adjusted according to the training conditions; in this invention, the default value is set to 512. The sampled experience samples are denoted as follows in this invention. After sampling, based on Decompose the discrete Q-network and calculate the corresponding optimal action.

[0079]

[0080] (4) Continuous Q-network update. This update process uses the temporal difference method TD(0) to simultaneously calculate the loss functions of the two evaluation Q-networks:

[0081]

[0082] Subsequently Using a step size, the gradient descent method is used to update the evaluation Q-network parameters:

[0083]

[0084] (5) Decomposing the Discrete Q-Network Update. This invention mainly employs supervised learning to reduce the difference in state-behavior value estimates between the decomposed discrete Q-network and the evaluation Q-network, thereby enabling the decomposed discrete Q-network to accurately estimate the levels of state-behavior values ​​for different discrete actions and select the action with the best control effect. In this step, the action quantity in the sample batch is first... As input, the discrete Q-network is decomposed to output a Q-function matrix. Then, a new estimate is made based on the state-behavior values ​​in the matrix. The optimal action

[0085]

[0086]

[0087] And define its exclusion action. for Exclude actions in all dimensions except the m-th dimension. At the same time, With θ d Decoupling will not be calculated again in subsequent loss function calculations. θ d The gradient.

[0088] After completing the above steps, for each discrete state-behavior value in the Q-function matrix, a corresponding continuous state-behavior value can be output using the continuous evaluation network, and the mean square error between the two can be calculated as a partial loss function for decomposing the discrete Q-function.

[0089]

[0090] After calculating the mean squared error corresponding to each element in the Q-function matrix, these errors are summed to form the overall error function for decomposing the discrete Q-network.

[0091]

[0092] Subsequently, with λ d Using a step size, the parameters of the decomposed Q-network are updated using gradient descent.

[0093]

[0094] (6) Target Q network parameter soft update:

[0095]

[0096] This invention provides a deep Q-network reinforcement learning method applicable to complex unmanned equipment control problems. Based on the independent discretization of the action space in each dimension, this method designs a discrete-continuous heterogeneous Q-network architecture and its corresponding reinforcement learning training method, solving the problem of mutual coupling between Q-functions in different action dimensions, and significantly improving the training efficiency and final-state performance of value function reinforcement learning for continuous control problems. This architecture includes two different Q-network structures: a continuous evaluation Q-network and a discrete action Q-network. The continuous evaluation Q-network takes the state and continuous actions as input and outputs the corresponding state-behavior values, updated using temporal difference, and mainly serves to provide mentor signals for the discrete Q-network. The discrete action Q-network functions similarly to the Actor network in a policy-evaluation structure, taking the state as input and outputting the state-behavior values ​​corresponding to discrete actions in each dimension. It follows the value function of the continuous evaluation Q-network using supervised learning and can output actions through a greedy policy. The reinforcement learning method based on the continuous-heterogeneous Q-network architecture designed in this invention makes full use of the characteristics of this architecture. While efficiently utilizing experience samples for offline updates, it adopts an ∈-Gaussian composite exploration method to improve the exploration efficiency of reinforcement learning. Compared with the commonly used policy-evaluation structure reinforcement learning method, it has a faster convergence speed and a more stable training process, which significantly improves the control effect of reinforcement learning controllers applied to unmanned equipment.

[0097] Beneficial effects

[0098] (1) This invention constructs two different reinforcement learning models and proposes to train the discrete Q function formed by discretizing each continuous action dimension by using the state-behavior value function transformation relationship between the two reinforcement learning models. The value function reinforcement learning method effectively solves the continuous control problem and provides a new reinforcement learning solution for some continuous control tasks that do not perform well in the Actor-Critic algorithm.

[0099] (2) The decomposed discrete Q-function network constructed in this invention realizes stable reinforcement learning control based on discrete actions, which further improves the effect of discrete reinforcement learning on continuous control problems and can effectively promote the development and application of discrete reinforcement learning.

[0100] (3) The method of the present invention has high training stability and training efficiency, which can effectively reduce the time cost of training reinforcement learning controllers and improve the performance of reinforcement learning controllers.

[0101] (4) The method of the present invention uses temporal difference training of continuous evaluation network, so it can effectively reuse experience. Combined with the high training efficiency of the method, it can significantly reduce the number of experience samples required for reinforcement learning training process, reduce the training cost of reinforcement learning deployment on physical control, and promote the deployment of reinforcement learning control on physical systems.

[0102] (5) The decomposed discrete Q-function network constructed in this invention can be used as a policy network. Therefore, the method of this invention has the structural characteristics of both value function reinforcement learning method and Actor-Critic reinforcement learning method. It has the ability to integrate the improvement schemes of the two types of methods and has important reference significance for the further improvement and enhancement of reinforcement learning control methods. Attached Figure Description

[0103] Figure 1 This is a diagram of the overall network architecture for reinforcement learning of the discrete-continuous heterogeneous Q-network provided by this invention.

[0104] Figure 2 This is a schematic diagram illustrating the decomposition of the discrete Q-function network loss function in the reinforcement learning method for discrete-continuous heterogeneous Q-networks provided by this invention. Detailed Implementation

[0105] The present invention will now be described in detail with reference to the accompanying drawings. These drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0106] A reinforcement learning-based continuous action control method based on discrete-continuous heterogeneous Q-networks, comprising the following steps:

[0107] Step 1: Based on the number and characteristics of sensors and motors of the robot and other controlled objects, construct a continuous value function reinforcement learning model and a discrete Q function reinforcement learning model corresponding to the controlled object, and utilize the coupling relationship between the two to further construct a continuous-discrete Q function reinforcement learning model that can learn a discrete Q function based on a continuous Q function.

[0108] Step 2: Based on the continuous-discrete Q-function reinforcement learning model obtained in Step 1, construct the neural network architecture for discrete-continuous heterogeneous Q-network reinforcement learning, including continuous Q-network modules and discrete Q-network modules;

[0109] Step 3: Based on temporal difference and supervised learning, train the neural network architecture constructed in Step 2 to obtain the converged discrete Q network module and its corresponding decomposed discrete Q function.

[0110] Step 4: Decompose the discrete Q-function as the controller, select the action corresponding to the maximum Q-function value at each time step, and apply it to the controlled object such as the robot to enable it to work effectively, thus completing the reinforcement learning continuous motion control method based on discrete-continuous heterogeneous Q-network.

[0111] In step 1, the continuous Q-function reinforcement learning model is a Markov decision process model, i.e., a quintuple. in Let be the state space, containing all possible states s(t), where t is time; Let a be the action space, representing all possible continuous actions a(t) = {a1(t), a2(t), ..., a...}. M (t)}, consisting of each action dimension a m The model is composed of actions, where m represents the dimension; p represents the transition probability p(s(t+1)|s(t), a(t)) to the next state s(t+1) under s(t) and a(t); r is the reward function r(s(t), a(t)); and γ∈[0,1] is the discount rate constant. The model aims to find an optimal deterministic policy μ(s(t))={μ1(s(t)),...,μ... M (s(t))}, this strategy includes the optimal deterministic sub-strategy μ in each dimension. m The expected cumulative reward value (s(t)) is determined by the action a(t) = μ(s(t)), which maximizes the expected reward value of the entire model, i.e., the cumulative value of the reward function in the future, thereby optimizing the control performance of the reinforcement learning controller. Under the action a(t) = μ(s(t)), the expected cumulative reward value that the model can obtain is also called the state-behavior value or Q-function value, which is calculated using the following formula in continuous models:

[0112]

[0113] To obtain the highest cumulative reward and optimize the performance of the controlled object, the optimal deterministic policy of the continuous Q-function reinforcement learning model and the related provisions of the continuous model Q-function are as follows:

[0114]

[0115] The reinforcement learning model for discrete Q-functions is derived from the reinforcement learning model for continuous Q-functions, and is defined over each continuous action dimension of the controlled object (controllable units such as motors). For dimension m, the discrete Q-function reinforcement learning model is constructed using Markov quintuples. Composition, in which, Defined as follows, a discrete state space containing N discrete actions:

[0116]

[0117] p m The single-step state transition probability for dimension m has the following transformation relationship with the transition probability p of a continuous model:

[0118]

[0119]

[0120] in It is a strategy for all dimensions except dimension m.

[0121] r m The reward function for dimension m has the following coupling relationship with the reward function r of the continuous model:

[0122]

[0123] The goal of a discrete Q-function reinforcement learning model for action dimension m is to find an optimal discrete policy μ. m (s(t)) can maximize the following discrete Q function:

[0124]

[0125] Among them, the single-dimensional discrete strategy μ m (s(t)) is related to the discrete Q-function and has the following form:

[0126]

[0127] In the discrete model of this invention, the discrete and continuous state behavior value functions (Q functions) have the following coupling relationship:

[0128]

[0129] Based on this coupling relationship, the optimal policy μ(s(t)) of the continuous model can be approximated by the optimal policy of the discrete model, and the discrete policy, after being made continuous, can be used as the approximate optimal policy of the continuous model:

[0130] μ(s(t)):={μ1(s(t)),μ2(s(t)),...,μ M (s(t))}

[0131] In step 2, the neural network architecture for discrete-continuous heterogeneous Q-network reinforcement learning includes two modules: a continuous Q-network and a discrete Q-network decomposition module. The continuous Q-network module comprises four networks: two evaluation Q-networks and two target Q-networks, used for the evaluation part of fitting the continuous model.

[0132] ①Evaluate Q network 1, using The network parameters are as follows: The network employs a multilayer perceptron structure with two hidden layers. The input layer has X+M neurons, representing the state dimension plus the continuous action dimension, and takes the state and action s(t) and a(t) as inputs. The output layer has one neuron, which outputs the state-action value.

[0133] ②Evaluate Q-network 2, using The network parameters are as follows: The structure is the same as ①. The state-behavior values ​​output by the two networks are minimized to obtain the Q-function estimates actually applied to the algorithm:

[0134]

[0135] ③Target Q network 1, using The network parameters are as follows: The structure is the same as ①;

[0136] ④Target Q network 2, using The network parameters are as follows: The structure is the same as ①; the two target networks are used to evaluate the objective Q-function value of the reinforcement learning temporal difference update, and the network parameters... All of them use a soft update method to follow their corresponding evaluation network parameters:

[0137]

[0138] Where τ is the target network soft update hyperparameter, with a default value of 0.005.

[0139] The Discrete Q-Network Decomposition module contains one Discrete Q-Network Decomposition module, which... It means that the parameter is θ dThe network employs a multilayer perceptron structure with X neurons in the input layer, and the input is the state variable s(t). The network has two hidden layers, and the output layer has M×N neurons, outputting a vector of size M×N, which is the action dimension × the number of discrete actions per dimension. The output vector is then reconstructed into an M-row N-column output matrix. Each row of this matrix contains the state-behavior values ​​of all discrete actions in one action dimension. The element in the m-th row and n-th column is... Corresponding to the nth discrete action in the mth action dimension State-behavior values:

[0140]

[0141] When decomposing discrete Q-networks and using them as policy networks in continuous models, the selection... The discrete action corresponding to the maximum value in each row:

[0142] a(t)=μ(s(t); θ d ): = {μ1(s(t); θ d ), μ2(s(t); θ d )···μ M (s(t;θ) d )}

[0143]

[0144] In step 3, the method for training the deep reinforcement learning network framework is as follows:

[0145] (1) Initialize the continuous Q network parameters Decomposition of discrete Q-network parameters θ d Assign random initial values ​​that conform to a Gaussian distribution; initialize hyperparameters such as learning rate, reward discount rate, and exploration rate with constant values; initialize the experience pool for storing samples, which contains no experience samples by default;

[0146] (2) An ∈-Gaussian exploration strategy is used to interact with the environment and collect experience {s(t), a(t), r}. t , s(t+1)}, that is, the state, action, reward at time t and the state at the next time t+1, and store them in the experience pool;

[0147] (3) Once the experience pool contains a sufficient amount of experience, a batch of experience is sampled from the experience pool, denoted as . in, This represents the time when the sample was collected; after sampling, based on... Decompose the discrete Q-network and calculate the corresponding optimal action.

[0148]

[0149] (6) Continuous Q-network update. This update process uses the temporal difference method (TD(0)) to simultaneously calculate the loss functions of the two evaluation Q-networks:

[0150]

[0151] Then A Q To update the step size (learning rate), gradient descent is used to update the evaluation Q-network parameters:

[0152]

[0153] (7) Decompose the discrete Q-network update. This involves considering the Q-function matrix. Each discrete state-behavior value Both methods utilize a continuous evaluation network to output a corresponding continuous state-behavior value, and calculate the mean squared error between the two as a partial loss function for decomposing the discrete Q-function.

[0154]

[0155] After calculating the mean squared error corresponding to each element in the Q-function matrix, these errors are summed to form the overall error function for decomposing the discrete Q-network.

[0156]

[0157] Subsequently, with λ d Using a step size, the parameters of the decomposed Q-network are updated using gradient descent.

[0158]

[0159] (6) Target Q network parameter soft update:

[0160] (7) Repeat steps 2 to 6 until the robot control strategy converges.

[0161] The ∈-Gaussian exploration strategy described in step (2) is mainly implemented through the following methods:

[0162] ① Based on the decomposition of the discrete Q-network, the discrete actions corresponding to state s(t) are obtained:

[0163] a(t)=μ(s(t); θ d );

[0164] ② Apply the ∈-greedy strategy to generate exploration actions, that is, randomly select actions with a probability of ∈:

[0165]

[0166] ③ The exploration actions are made continuous, and a Gaussian noise is superimposed to obtain the exploration actions under the ∈-Gaussian exploration strategy, where σ is the variance of the Gaussian exploration noise:

[0167]

[0168] In step (5), the parameters θ of the discrete Q-network are iteratively updated, mainly by minimizing the following loss function using gradient descent:

[0169]

[0170] in, The partial loss function is calculated as follows:

[0171]

[0172] Example

[0173] This invention provides a reinforcement learning control method based on continuous-discrete heterogeneous Q-networks, which solves the continuous action space control problem that is difficult to address by DQN equivalent function reinforcement learning methods by learning two different Q-functions. Compared with the Actor-Critic framework algorithm commonly used to solve complex reinforcement learning control, the method of this invention has higher training efficiency and more stable training performance, thus taking the effect of reinforcement learning control to a new level.

[0174] To address the control problems of robots and other unmanned devices using value function reinforcement learning, this method first transforms the continuous action space corresponding to the control problem into a discrete action space. This facilitates the comparison of discrete value functions of different actions by the value function reinforcement learning method, thereby selecting the optimal control variable. The method of this invention discretizes each continuous action dimension into N discrete actions, which are uniformly distributed within the range [-1, 1].

[0175]

[0176]

[0177] While discretizing the action space, it is necessary to transform the state space into an x-dimensional real space, that is... If the state observations of the control system have a low dimension (approximately 500 dimensions or less), the domain of the state observations can be directly used as the state space. If the state observations include high-dimensional information such as images, encoders such as convolutional neural networks, recurrent neural networks, and single-layer / multilayer perceptrons can be used to encode the state observations into x-dimensional vectors, and then the domain of these vectors can be used as the state space.

[0178] After constructing the state and action spaces, based on their corresponding discrete Q-function reinforcement learning models and continuous Q-function reinforcement learning models, the reinforcement learning neural network model in step 3 of the method described in this invention is constructed. This model includes two modules: a continuous Q-network module and a discrete Q-network module, as follows: Figure 1 As shown.

[0179] The continuous Q-network module includes four networks: evaluation Q-networks 1 and 2, and target Q-networks 1 and 2. The definitions of these four networks were given in step 2 above. For control problems of general complexity, the network structure can be directly set as a double-hidden-layer fully connected network with x + M (state dimension + action dimension) neurons in the input layer, 512 neurons in the hidden layer, and 1 neuron in the output layer. For problems with higher complexity, the network structure can be adjusted according to actual needs. The discrete Q-network module contains a decomposed discrete Q-network, which also adopts a double-hidden-layer fully connected network structure with x neurons in the input layer, 512 neurons in the hidden layer, and M × N (state dimension multiplied by the number of states per dimension) neurons in the output layer. For complex problems, the structure can be adjusted as needed. After the neural network is built, the neural network parameters are randomly initialized using a Gaussian distribution.

[0180] After completing the above neural network construction steps, the reinforcement learning training process begins. The training process is as described in step 3 above. First, the first step of the process is executed to initialize the hyperparameters and experience pool required for reinforcement learning. After initialization, steps 2 to 6 are looped to continuously update the state and behavior values ​​of the Q-network; step 2 is the environment interaction step, and steps 3 to 6 are the neural network training steps; the fifth step, which decomposes the update of the discrete Q-network, is relatively complex, so an illustration is provided. Figure 2 For reference, this figure shows a partial loss function. The calculation diagram is shown below. Training stops once the policy of the discrete Q-network converges.

[0181] After training, the discrete Q-network is deployed as a controller. In practical applications, this network uses a strategy of decomposing the maximum Q-function value to provide control actions. That is, based on the system state information, the discrete action corresponding to the maximum state-behavior value is selected in each action dimension, and then continuousized and output as the control quantity. It is important to note that in the method provided by this invention, the action space of reinforcement learning is restricted to the range [-1, 1]. Therefore, when converting actions into control quantities, it is necessary to map the action quantities to the original control quantities. Assume that the upper and lower bounds of the i-th dimension control quantity are respectively... The specific mapping rules are as follows:

[0182] a(t)=μ(s(t); θ d )

[0183]

[0184] u(t)=u1(t),u2(t),…,u M (t)

[0185] By continuously applying the control actions obtained by the above method to the unmanned system at each time step, a good reinforcement learning control effect can be obtained.

[0186] In summary, the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for reinforcement learning continuous action control based on discrete-continuous heterogeneous Q-network, characterized in that The steps of the method include: Step 1, constructing a continuous Q function reinforcement learning model and a discrete Q function reinforcement learning model, and constructing a continuous-discrete Q function reinforcement learning model according to the coupling relationship between the constructed continuous Q function reinforcement learning model and the discrete Q function reinforcement learning model; Step 2, constructing a neural network architecture of a discrete-continuous heterogeneous Q network reinforcement learning according to the continuous-discrete Q function reinforcement learning model constructed in step 1; Step 3, training the neural network architecture constructed in step 2 based on time difference and supervised learning to obtain a converged discrete Q network module and a corresponding decomposed discrete Q function network; Step 4, taking the decomposed discrete Q function after training in step 3 as a controller, selecting the action corresponding to the maximum Q function value at each time step, and applying it to the controlled object to complete the reinforcement learning continuous action control method based on the discrete-continuous heterogeneous Q network; The continuous Q function reinforcement learning model in step 1 is a Markov decision process model, that is, a five-tuple Wherein, S is a state space, State transition probability, Reward value function, and γ is a discount rate constant; The all possible states , time all possible continuous actions , where each action dimension is represented by an action, where m represents the dimension. The representing in , to the next state transition probability ; The To S ( t ), a ( t ) is a function of ; ； The continuous Q-function reinforcement learning model aims to find an optimal deterministic policy, The optimal deterministic policy includes an optimal deterministic sub-policy in each dimension So that the entire continuous Q-function reinforcement learning model can obtain the maximum expected return value, i.e. the cumulative value of the reward value function in the future, thereby optimizing the control performance of the reinforcement learning controller, in the action The expected cumulative return value that the continuous Q-function reinforcement learning model can obtain is also called the state-action value or Q-function value, which is calculated by the following formula: In order to obtain the highest cumulative return value and optimize the performance of the controlled object, the optimal deterministic policy of the continuous Q function reinforcement learning model and the continuous Q function are as follows: 。 2. The reinforcement learning continuous action control method based on the discrete-continuous heterogeneous Q network according to claim 1, characterized in that: The reinforcement learning model of the discrete Q-function is derived from a reinforcement learning model of a continuous Q-function for a dimension The reinforcement learning model of the discrete Q-function is derived from a reinforcement learning model of a continuous Q-function for a dimension The reinforcement learning model of the discrete Q-function is derived from a reinforcement learning model of a continuous Q-function for a dimension The discrete state space is defined as follows, containing N discrete actions: is the single-step state transition probability for dimension m, and the transition probability p of the continuous Q-function reinforcement learning model has the following transformation relationship: wherein is a strategy for other dimensions than dimension m; is a reward function for dimension m and the reward function of the continuous Q function reinforcement learning model has the following coupling relationship: 。 3. The reinforcement learning continuous action control method based on the discrete-continuous heterogeneous Q network according to claim 2, characterized in that: The discrete Q-function reinforcement learning model for the action dimension m aims to find an optimal discrete policy that can maximize the following discrete Q-function: wherein the single-dimension discrete policy is of the form: 。 4. The reinforcement learning continuous action control method based on the discrete-continuous heterogeneous Q network according to claim 3, characterized in that: The coupling relationship between the continuous Q function reinforcement learning model and the discrete Q function reinforcement learning model is: Optimal policy of a continuous Q-function reinforcement learning model approximated by a discrete Q-function reinforcement learning model, the discrete policy is then continuous after the approximation of the optimal policy of the continuous model: 。 5. The reinforcement learning continuous action control method based on the discrete-continuous heterogeneous Q network according to claim 3, characterized in that: In step 2, the constructed neural network architecture includes a continuous Q network module and a discrete Q network module, wherein the continuous Q network module has four networks, including two evaluation Q networks and two target Q networks, for fitting the evaluation part of the continuous model; the two evaluation Q networks are evaluation Q network 1 and evaluation Q network 2; the two target Q networks are target Q network 1 and target Q network 2 respectively; The evaluation Q network 1 employs is represented by a network parameter ; the network employs a multi-layer perceptron structure with two hidden layers, the input layer has i.e. state dimension + continuous action dimension neurons, the input state, action ; the output layer has 1 neuron, output state-action value ; The evaluation Q-network 2, with is represented by a network parameter , has the same structure as the evaluation Q-network 1, and the state-action values output by both networks are combined by a minimization operation to obtain an estimate of the actual Q-function: The target Q-network 1, adopts to express, network parameters are , the structure and the evaluation Q-network 1 are same; The target Q network 2 adopts The network parameters are expressed as The structure is the same as that of the evaluation Q network 1; the two target networks are used to evaluate the target Q function value of the reinforcement learning time difference update, and the network parameters , Both of them follow the corresponding evaluation network parameters in a soft update manner: wherein, target_network_soft_update_hyperparameter, with a default value of 0.005; The discrete Q-network module contains one decomposed discrete Q-network, so that... It means that the parameter is It adopts a multilayer perceptron network structure, with the input layer... There are 1 neuron, and the input is the state variable. The network has 2 hidden layers and the network output layer has 10 hidden layers. Each neuron outputs a value with a size equal to the action dimension. A vector of discrete action numbers for each dimension is output, and then reconstructed into a vector. OK Column output matrix Each row of this matrix contains the state-behavior values ​​of all discrete actions in one action dimension. The element in the m-th row and n-th column is... Corresponding to the nth discrete action in the mth action dimension State-behavior values: When the decomposed discrete Q-network is used as a policy network for a continuous model, the discrete action corresponding to the maximum value of each row is selected the discrete action corresponding to the maximum value of each row 。 6. The reinforcement learning continuous action control method based on the discrete-continuous heterogeneous Q network according to claim 1, characterized in that: In step 3, the constructed neural network architecture is trained based on time difference and supervised learning, and the method for training the neural network architecture is: Step (1) initialize continuous Q-network parameters and decomposed discrete Q-network parameters , give random initial values conforming to Gaussian distribution; initialize learning rate, reward value discount rate, exploration rate hyperparameters, give fixed values; initialize experience pool storing samples, by default, there are no experience samples in it; Step (2) employs - the Gaussian exploration policy interacts with the environment, collecting experiences i.e. the state at time t, action, reward, and next state at time t+1, and stores them into the experience pool; Step (3) When there are enough experiences in the experience pool, sample a batch of experiences from the experience pool, denoted as wherein, represents the time when the sample is collected; after sampling, based on and the decomposed discrete Q network, the corresponding optimal action is calculated : Step (4) continuous Q network update, the loss function of the two evaluation Q networks is calculated synchronously using the time difference method in the update process: Subsequently, with The evaluation Q network parameters are updated using the gradient descent method with an update step size Step (5) decomposes the discrete Q-network update, for each discrete state-action value in the Q-function matrix , with the continuous evaluation network to output a corresponding continuous state-action value, and computes the mean squared error between the two as a partial loss function for decomposing the discrete Q-function: After calculating the mean square error corresponding to each element in the Q function matrix, the sum of these errors is taken as the total error function of the decomposed discrete Q network: Subsequently, with The parameters of the decomposition Q-network are updated using the gradient descent method with a step size of Step (6) target Q network parameter soft update: Step (7) repeat steps (2)~(6) until the control policy converges.

7. The reinforcement learning continuous action control method based on the discrete-continuous heterogeneous Q network according to claim 6, characterized in that: Said step (2) in, - a Gaussian exploration policy, mainly implemented by the following method: Based on the decomposition discrete Q-network, the state Corresponding discrete action: ； ② Application The strategy generates an exploration action, i.e. selects a random action with a probability of : iii. The exploration action is made continuous and a Gaussian noise is added to it, resulting in - the exploration action under the Gaussian exploration policy, where is the variance of the Gaussian exploration noise: In the step (5), the parameters of the discrete Q network are decomposed The iterative update is mainly achieved by minimizing the following loss function through the gradient descent method: wherein, is a partial loss function, calculated as follows: 。

Images