A safe reinforcement learning method for continuous control problems

Abstract

The application discloses a safety reinforcement learning method for continuous control problems. A probabilistic ensemble model is used to approximate the dynamic environment, representing the cognitive uncertainty and accidental uncertainty of the environment. A safety policy optimization method is designed with a dangerous punishment mechanism as the core, which increases the cost of executing unsafe actions so that the value of safe actions is higher than that of unsafe actions. The safety policy optimization method is combined with model-based reinforcement learning, and the convergence and safety of the algorithm are theoretically proved. A model predictive shielding algorithm is designed, which uses model predictive control as a potential shielding layer during the interaction between the agent and the environment, filters high-risk actions, and selects high-value actions. Simulation experiments show that the safety reinforcement learning method for continuous control problems proposed by the application can achieve good performance while maintaining safety.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of safety-critical intelligent software, and particularly relates to a safety reinforcement learning method for continuous control problems. BACKGROUND

[0002] In recent years, traditional industrial control systems have gradually become autonomous, intelligent and unmanned. Reinforcement learning, as an intelligent control algorithm, has accumulated numerous research results in complex fields such as robots, autonomous driving and biological medicine. Reinforcement learning is an intelligent agent that obtains a reward signal and learns an optimal action policy in the process of interacting with the environment, so that the intelligent agent obtains the maximum cumulative reward in the environment. Reinforcement learning algorithms can be divided into two categories: model-free and model-based. Model-free reinforcement learning algorithms directly interact with the environment to obtain experience data without relying on the environment model, and use the collected data to optimize the policy. Model-based reinforcement learning algorithms learn in the environment while generating an environment model using experience data, and use the environment model for planning. Although reinforcement learning algorithms have been extensively and deeply researched, the process of moving from the laboratory to the industry is cautious and slow. Reinforcement learning algorithms involve a lot of trial and error and dangerous actions in the process of large-scale interaction with the environment, and the application of such algorithms in the actual environment may cause equipment damage or even personal injury, such as improper operation of a robot. In addition, for some application scenarios with strict safety requirements, such as autonomous driving and chemical production, general reinforcement learning algorithms do not consider risk control, which may cause the intelligent agent to enter a dangerous state, such as an autonomous vehicle running off the road. Therefore, researching safe reinforcement learning algorithms and ensuring the safety of intelligent decision-making is of great significance to promoting the application of reinforcement learning in actual environments, especially in safety-critical fields.

[0003] Safe reinforcement learning is to maximize the expected return while meeting safety constraints, reducing dangerous risks or guaranteeing system performance during learning or deploying a policy. Existing safe reinforcement learning methods can be mainly divided into two categories. The first category is safe reinforcement learning methods that modify the optimality criterion, which achieve safety by modifying the objective function. In this case, the goal of the optimal policy is not only to maximize the expected cumulative reward, but also to consider the worst-case scenario or environmental risks. The main optimality criteria include the worst-case criterion, risk-sensitive criterion and constrained optimization criterion. The second category is safe reinforcement learning methods that modify the exploration process, which use prior knowledge, artificial interference or imitation learning to modify possible dangerous actions during exploration. The above methods usually belong to model-free policy optimization safe reinforcement learning, but model-free algorithms lack prior knowledge, so the intelligent agent will experience a large number of unsafe interactions before truly learning to avoid danger.

[0004] Model-based safe reinforcement learning methods approximate the environment dynamics using collected data, and then use the model for safe control planning and policy optimization. Compared with model-free methods, model-based safe reinforcement learning has many advantages. First, it is more efficient in data sampling. The agent can use virtual experiences generated by the model to improve the policy, reducing interactions with the real environment and thus reducing the risk of dangerous situations during training. Second, the agent can use the model to "imagine" the future trajectory before making a decision, thus avoiding possible dangerous actions. In addition, the learned model can be reused for different tasks in the same environment because it does not depend on the specific task. From the above discussion, the key work of model-based safe reinforcement learning is to learn an environment model with good fitting and prediction ability in the model learning stage, to use the model to optimize the policy towards a safer direction in the policy optimization stage, and to use the model to predict the future trajectory to avoid dangerous actions as much as possible in the interaction stage.

[0005] Existing model-free safe reinforcement learning methods have low sampling efficiency and still need to go through a large number of unsafe interactions before learning the optimal policy. Existing model-based safe reinforcement learning methods either need to manually construct the environment model and are only suitable for simple low-dimensional continuous control tasks, or can only be applied to control tasks in deterministic environments and cannot adapt to stochastic environments. SUMMARY

[0006] The present application provides a safe reinforcement learning method for continuous control problems, which can solve complex high-dimensional continuous control problems in stochastic environments.

[0007] Technical solution: A safe reinforcement learning method for continuous control problems, comprising the following steps:

[0008] Step 1: Approximate the dynamic environment using a probabilistic ensemble model to construct a virtual environment.

[0009] Step 2: Combine the model-based reinforcement learning algorithm with a danger penalty mechanism, and learn the optimal policy function by interacting with the actual environment and the virtual environment.

[0010] Step 3: When the agent interacts with the actual environment, use the optimal policy function obtained in step 2 to obtain a number of pre-selected actions, and use the probabilistic ensemble model to generate N predicted trajectories of length M for each pre-selected action, and count the frequency of violating safety regulations and the average discounted cumulative reward of N predicted trajectories. Only when the frequency of violating safety regulations corresponding to the pre-selected action is not greater than the safety threshold, the pre-selected action is selected as a safe action.

[0011] Step 4: select a pre-selected action with the maximum average discounted cumulative reward from the optional safety actions to perform, achieving continuous control.

[0012] Further, the probability ensemble model in step 1 is composed of a plurality of ensemble probability neural networks, the output of each ensemble probability neural network is a Gaussian distribution, denoted as:

[0013]

[0014] In the formula, s represents the state, a represents the action, θ i represents the parameters of the i-th neural network, represents the Gaussian distribution, represents the mean of the Gaussian distribution represents the variance of the Gaussian distribution.

[0015] Randomly select the output of an ensemble probability neural network as the output of the probability ensemble model; the output of the ensemble probability neural network is the mean of the Gaussian distribution.

[0016] Further, the probability ensemble model is a model obtained according to the following training steps:

[0017] Obtain training samples from the experience pool

[0018] Update the parameters of each ensemble probability neural network using maximum likelihood estimation:

[0019]

[0020] In the formula, represents the loss function, represents the probability expectation, (s′,r|s,a) represents the training sample obtained from the experience pool D, s′ represents the next state reached by the agent in state s performing action a, and r represents the reward obtained by the agent in state s performing action a.

[0021] Further, the model-based reinforcement learning algorithm is combined with the dangerous punishment mechanism, specifically including the following steps:

[0022] Determine the dangerous punishment mechanism, and the dangerous punishment mechanism specifically includes:

[0023] The framework of the interaction between the agent and the environment is a Markov decision process, denoted as where S represents a state set, A represents an action set, represents a state transition function, represents a reward function, and γ represents a discount factor.

[0024] State transition function and reward function​ satisfies:

[0025]

[0026] wherein, represents the value of the generation of the agent entering the unsafe state;

[0027] The value of the generation of the agent entering the unsafe state C is updated according to the following formula:

[0028]

[0029] wherein, R max represents the maximum value of the reward obtained by the agent in the environment, R min represents the minimum value of the reward obtained by the agent in the environment, p represents the probability of the agent entering the unsafe state after performing any safe action, H * represents the longest step that the agent can remain safe after performing any action, and H represents the longest step that the agent can observe before performing any action.

[0030] The Bellman operator is defined;

[0031] The model-based reinforcement learning algorithm is adopted, in the interaction process of the agent and the environment, the value of the generation of the agent entering the unsafe state C is updated using the dangerous punishment mechanism, and the state value of the unsafe state is given;

[0032] The optimal state value function is learned using the Bellman operator, and the optimal policy function is obtained.

[0033] Further, the model-based reinforcement learning algorithm is SAC algorithm;

[0034] The SAC algorithm specifically includes:

[0035] Four functions are defined, which are represented using neural networks respectively: policy function π θ (a|s), state value function V ψ (s), and two state-action value functions and

[0036] The objective function of the policy function π θ (a|s) is represented as:

[0037]

[0038] The objective function of the state-action value function is represented as:

[0039]

[0040] The objective function of the state value function is:

[0041]

[0042]

[0043] Further, the Bellman operator is represented as:

[0044]

[0045] In the formula, B * The Bellman operator is represented as, and Q(s, a) represents a state-action value function The output of the probabilistic neural network is represented as.

[0046] Further, the state value of the unsafe state is specifically:

[0047] When the intelligent agent enters the unsafe state S unsafe , the state value of the unsafe state S unsafe is directly given as:

[0048] V ψ (s unsafe ) = -C / (1-γ).

[0049] Beneficial effects: the present application has the following advantages:

[0050] (1) the present application uses a model-based safe reinforcement learning method to solve the continuous random control problem, the model can effectively represent the randomness of the environment, and the learned model can be repeatedly used for different tasks in the same environment due to its independence from specific tasks; compared with the model-free safe reinforcement learning method, the present application has higher data sampling efficiency, the intelligent agent can use the virtual experience generated by the model to improve the policy, reduce the interaction with the real environment, and thus reduce the dangerous situations during training;

[0051] (2) the present application does not need to manually construct an environment model, and can automatically learn an environment model to approximate the dynamic environment;

[0052] (3) the present application can solve the continuous control problem in a random environment, has strong anti-interference ability, and is suitable for complex high-dimensional control tasks;

[0053] (4) the present application ensures high safety in solving the continuous control problem in a random environment. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 ​A schematic diagram of the overall framework of a safety reinforcement learning method for continuous control problems according to the present application;

[0055] Figure 2 A schematic diagram of four types of bionic robot control tasks for implementing the method of the present application;

[0056] Figure 3 A schematic diagram of experimental results for implementing the method of the present application. DETAILED DESCRIPTION

[0057] The method of the present application is further described below in conjunction with the accompanying drawings and examples.

[0058] This example takes a continuous control task on a MuJuCo simulator as an example, and further describes the method of this example in conjunction with Figure 2 and Figure 3 The method of this example is further described.

[0059] Figure 2 Four types of bionic robot control tasks: Ant, Cheetah-no-flip, Hopper and Humanoid are shown. The optimal policy is learned using the method of this example to control the continuous motion of the bionic robot and maximize the reward obtained by the robot in the environment.

[0060] The maximum length of a single segment of robot motion is set to 1000 time steps, and the safety constraint is that the robot cannot "fall down" to terminate the current segment. If the robot violates the safety constraint within 1000 time steps, the segment is invalid, otherwise it is successful. In order to consider the randomness of the environment, a certain disturbance is added to each executed action, and the disturbed action value follows a normal distribution with a mean of the original value and a standard deviation of 1.0.

[0061] As shown in Figure 1 , the specific implementation steps are as follows:

[0062] Step 1: Use a probabilistic ensemble model to approximate the dynamic environment, and use the probabilistic ensemble model to represent the cognitive uncertainty and accidental uncertainty of the environment; in this example, the probabilistic ensemble model consists of 5 fully connected neural networks, denoted as where the output of each fully connected neural network is a Gaussian distribution, denoted as where s represents the state, a represents the action, and i represents the parameters of the i-th neural network, represents the Gaussian distribution, represents the mean of the Gaussian distribution represents the variance of the Gaussian distribution.

[0063] Assume that the training samples are sampled from the experience pool Then each fully connected neural network updates parameters using maximum likelihood estimation:

[0064]

[0065] where, denotes the loss function, denotes the probability expectation, (s', r | s, a) denotes a training sample obtained from the experience pool D, s' denotes the next state that the agent reaches after performing action a in state s, and r denotes the reward that the agent obtains after performing action a in state s.

[0066] The fully connected neural network used in this embodiment includes 3 layers of hidden layers with a width of 200, and the nonlinear activation function is: ReLU(x) = max{0, x}. The Adam optimizer is used for optimization, the number of batch samples is 256, and the learning rate is 1E-3.

[0067] In the model using stage, each fully connected neural network outputs the mean and variance of a Gaussian distribution, in this embodiment, the mean is used as the output result, and the output of a fully connected neural network is randomly selected as the output of the probabilistic ensemble model.

[0068] Step 2: design a safety policy optimization method with a dangerous penalty mechanism as the core, increase the cost of executing unsafe actions so that the value of safe actions is higher than that of unsafe actions, and combine the safety policy optimization method with model-based reinforcement learning to obtain the optimal policy function of the interaction between the agent and the environment. The specific operation includes:

[0069] The safety policy optimization method with a dangerous penalty mechanism as the core of this embodiment is:

[0070] Suppose that the original framework of the interaction between the agent and the environment is a Markov decision process M = <S, A, T, R, γ>, and a new Markov decision process is constructed, where S represents the state set, A represents the action set, and γ represents the discount factor; the state transition function and the reward function satisfy:

[0071]

[0072] where, denotes the cost of the agent entering an unsafe state;

[0073] Suppose that the probability of the agent entering an unsafe state after performing any safe action is p, and the maximum reward that the agent obtains in the environment is R max , and the minimum reward is R min. Assuming that the hyperparameters H = 10 and p = 0.5, the maximum reward R max , the minimum reward R min , C is continuously updated according to the following formula:

[0074]

[0075] where H * represents the maximum number of steps that the agent can remain safe after performing any action, and H represents the maximum number of steps that the agent can observe before performing any action.

[0076] This embodiment uses the SAC algorithm to learn the strategy as model-based reinforcement learning, and defines four functions, each represented by a fully connected neural network: the policy function π θ (a|s), the state value function V ψ (s), and the two state-action value functions Q and Q

[0077] The target function of the fully connected neural network corresponding to the policy function is extracted from the experience pool D:

[0078]

[0079] The target function of the fully connected neural network corresponding to the two state-action value functions is:

[0080]

[0081] The target function of the fully connected neural network corresponding to the state value function is:

[0082]

[0083] where

[0084]

[0085] The fully connected neural network used here contains 2 layers of hidden layers with a width of 256, and the nonlinear activation function is ReLU; the Adam optimizer is used for optimization, the initial entropy weight is 1.0, the number of batch samples is 256, and the learning rate of the network is 3E-4.

[0086] The above-mentioned step of combining the safety strategy optimization method with model-based reinforcement learning is:

[0087] Let the true state transition function be T, and the probability ensemble model be Define a new Bellman operator:

[0088]

[0089] Based on SAC algorithm, in the process of agent sampling, the C value is constantly updated using the dangerous punishment mechanism. If the sampling reaches the dangerous state S unsafe , the state value of the dangerous state S unsafe is directly given as:

[0090] V ψ (s unsafe )=-C / (1-γ)

[0091] And the new Bellman operator B * is used to learn the optimal state value function.

[0092] Now the convergence and safety of the algorithm combining the safety policy optimization method with model-based reinforcement learning are theoretically proved.

[0093] The convergence proof is as follows:

[0094] For any Q, Q':

[0095]

[0096] Therefore, the Bellman operator B * is a γ-compression mapping in the metric space (Q(s,a),∞). According to Banach fixed point theorem, the Bellman operator B * iteratively converges to the unique optimal state value function.

[0097] The safety proof is as follows:

[0098] Let the optimal state value function of the Bellman operator B * be Q * , and the optimal policy π * (s)=argmax a Q * (s,a), for any Q, Q': If Q≤Q', then:

[0099]

[0100] Therefore, B * Q<B * Q'. Let Q0 be the initial Q function, and for any after k iterations, Q k =(B * )k Q0, Q k B * k Q0, Q k ≤Q k . Taking the limit for k, i.e. after a sufficient number of iterations, Q * = lim k→∞ Q k , Q * = lim k→∞ Q k , we obtain Q * ≤Q * . For any unsafe action a', we have

[0101]

[0102] If there exists an action a such that Q * (s,a)≥(-pC+(1-p)R min ) / (1-γ), then:

[0103]

[0104] Thus, Q * (s,a′)≤ Q * (s,a), action a is a safe action. Therefore, the optimal policy π * will perform safe actions.

[0105] Step 3: Obtain a number of pre-selected actions using the optimal policy function every time the agent takes an action in the environment, generate N predicted trajectories of length M for each pre-selected action using the probabilistic ensemble model, and count the frequency of violation of the safety regulation and the average discounted cumulative reward of the N trajectories; in this embodiment, 5 predicted trajectories of length 10 are generated for each pre-selected action, and the frequency of violation of the safety regulation and the average discounted cumulative reward of the 10 trajectories are counted;

[0106] If the frequency of violation of the safety regulation of the pre-selected action is greater than the safety threshold ∈ = 0.6, the action is considered to be an unsafe action, otherwise the action is a selectable safe action;

[0107] Finally, the action with the maximum average discounted cumulative reward is selected from the selectable safe actions to be executed.

[0108] ​​In order to balance accuracy and efficiency, the number of pre-selected actions is set to be large, so as to generate more action sequences, and the number of predicted trajectories and the prediction level are set to be small.

[0109] The embodiment uses the model prediction control as a potential shielding layer to filter high-risk actions and select high-value actions in the process of interaction between the agent and the environment by designing a model prediction shielding algorithm.

[0110] Figure 3 The simulation experiment results are shown, and the curves describe the change of the reward obtained by the learned strategy with the cumulative failure number during the training process. The results show that the safety reinforcement learning method for continuous control problems proposed in the application realizes better performance while maintaining safety.

[0111] In order to verify the effectiveness of the model prediction shielding algorithm, the performance of the method in the presence and absence of shielding is compared. As shown in the following table, the model prediction shielding algorithm can effectively reduce the dangerous situations occurring in the interaction between the agent and the environment during the training process.

[0112]

[0113]

Claims

1. A safe reinforcement learning method for continuous control problems, characterized in that: The method comprises the following steps: Step 1: approximating a dynamic environment by using a probabilistic ensemble model to construct a virtual environment; Step 2: combining a model-based reinforcement learning algorithm with a danger penalty mechanism, and learning an optimal policy function by interaction between an agent and an actual environment and the virtual environment; Step 3: when the agent interacts with the actual environment, obtaining a plurality of preselected actions by using the optimal policy function obtained in step 2, generating N predicted trajectories with a length of M for each preselected action by using the probabilistic ensemble model, and counting the frequency of violating safety regulations and the average discounted cumulative reward of the N predicted trajectories; only when the frequency of violating safety regulations corresponding to the preselected action is not greater than a safety threshold, the preselected action is selected as a safe action; Step 4: selecting a preselected action with the maximum average discounted cumulative reward from the safe actions to execute, thereby realizing continuous control; The combination of the model-based reinforcement learning algorithm and the danger penalty mechanism comprises the following steps: determining a danger penalty mechanism, wherein the danger penalty mechanism comprises: The framework for the agent to interact with the environment is a Markov Decision Process, denoted as where S denotes the set of states, A denotes the set of actions, denotes the state transition function, denotes the reward function, and γ denotes the discount factor. State transition function And reward function Satisfies: wherein represents the cost of giving the agent access to an unsafe state; updating the generation value C of the agent entering an unsafe state according to the following formula: where R max represents the maximum value of the reward obtained by the agent in the environment, R min represents the minimum value of the reward obtained by the agent in the environment, p represents the probability of the agent entering an unsafe state after performing any safe action, H * represents the longest step that the agent can remain safe after performing any action, and H represents the longest step that the agent can observe before performing any action. defining a Bellman operator; updating the generation value C of the agent entering an unsafe state by using the danger penalty mechanism in the interaction process between the agent and the environment, and assigning a state value to the unsafe state by using a model-based reinforcement learning algorithm; learning an optimal state value function by using the Bellman operator to obtain an optimal policy function.

2. The safety-augmented learning method for continuous control problems according to claim 1, wherein: The probabilistic ensemble model in step 1 is composed of a plurality of ensemble probabilistic neural networks, and the output of each ensemble probabilistic neural network is a Gaussian distribution, which is represented as: In the formula, s represents a state, a represents an action, θ i denote parameters of the i-th neural network, denote a Gaussian distribution, denote a mean of the Gaussian distribution denote a variance of the Gaussian distribution; randomly selecting an output of an ensemble probabilistic neural network as the output of the probabilistic ensemble model; the output of the ensemble probabilistic neural network is the mean value of the Gaussian distribution.

3. The method of claim 2, wherein: The probabilistic ensemble model is a model obtained according to the following training steps: From the pool of experiences obtaining training samples; updating the parameters of each ensemble probabilistic neural network by using maximum likelihood estimation: wherein denotes the loss function, denotes the probability expectation, (s', r | s, a) denotes a training sample obtained from the experience pool D, s' denotes the next state reached by the agent in state s performing action a, and r denotes the reward obtained by the agent in state s performing action a.

4. The method of claim 1, wherein: the model-based reinforcement learning algorithm is a SAC algorithm; the SAC algorithm specifically comprises: Define four functions, each using a neural network representation: the policy function π θ (a|s), the state value function V ψ (s), and the two state-action value functions and Policy function π θ The objective function for (a|s) is expressed as: the target function of the state-action value function is represented as: the target function of the state value function is:

5. The method of claim 4, wherein: the Bellman operator is represented as: wherein B * denotes the Bellman operator, Q(s, a) denotes the state-action value function denotes the output of the probabilistic neural network .

6. The safety-augmented learning method for continuous control problems according to claim 5, wherein: the state value of the unsafe state specifically comprises: When the agent enters the unsafe state S unsafe , the state value of the unsafe state S unsafe is directly given: V ψ (s unsafe )=-C / (1-γ).

Images