Safe deep reinforcement learning method based on optimal counter example and obstacle function

Abstract

The invention discloses a safe deep reinforcement learning method based on an optimal counter example and a barrier function, and the method comprises the following steps: constructing a non-negative and non-standardized target probability distribution function based on counter example sample distribution through employing a standardized flow technology, and training a standardized flow model based on the target distribution; sampling counter-example samples by using the trained standardized flow model; and the generated counter-example samples are used for guiding the obstacle function network to carry out more refined correction training, deep reinforcement learning training is carried out by continuously generating and supplementing the counter-example samples, and finally a correct obstacle function is learned. Compared with the prior art, the method has the advantages that the inductive iteration training efficiency of the obstacle function and reinforcement learning is optimized through the optimal counter-example sample generation technology, and the formalized verification efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of safety controller generation for power systems, and in particular to a safety deep reinforcement learning method based on optimal counterexamples and obstacle functions. Background Art

[0002] With the development of deep learning, deep reinforcement learning (DRL) has shown great application prospects in the field of control. According to the expectations encoded by the user-defined reward function, deep reinforcement learning has been successfully applied to synthesize controllers that meet system requirements. However, synthetic safety control is still one of the most important and challenging problems.

[0003] Traditional deep reinforcement learning methods mainly set up penalty mechanisms and limit action spaces, which can avoid some uneasy behaviors, but still do not solve the safety problem well. Most studies focus on the optimization of reinforcement learning algorithms and propose safe / constrained reinforcement learning algorithms, aiming to maximize reward returns while satisfying safety constraints. However, this type of work lacks formal proof of the safety of deep neural network controllers. One type of research draws on ideas from the control field, such as barrier functions. Based on the physical model knowledge of known dynamic systems, barrier certificates (BC) are constructed to ensure various properties such as controller safety or stability. Barrier certificates divide the state space of the system model into two parts, which can ensure that all trajectories of the system start from a given initial set on one side of the BC and cannot reach the unsafe set on the other side. Therefore, the successful synthesis of a BC represents a formal proof of the safety of the dynamic system model. This method provides a formal reasoning proof scheme for the safety and other properties of the controller. At present, more and more works adopt barrier function learning methods based on counterexample guidance. This method verifies the correctness of the BC by designing a verifier through an SMT solver. If the verification fails, a counterexample is returned to update the data set for the next training.

[0004] The traditional training method based on counterexample guidance is to generate counterexample samples through the SMT solver. This method of generating counterexamples has two main problems:

[0005] 1. Slow generation of counterexample samples: For large-scale nonlinear problems or complex dynamic systems, the SMT solver has high computational and time costs.

[0006] 2. Low quality of counterexample samples: Since the solution process of the SMT solver is usually based on satisfiability rather than optimality, the goal is to find a solution that satisfies the constraints rather than the most "extreme" counterexample. Therefore, the counterexamples generated by the SMT solver are not guaranteed to be the most violating samples. Compared with the method of the present invention, the counterexample samples generated by the SMT solver are of lower quality.

[0007] Because the SMT solver has a high time cost and the generated counterexample samples are of low quality, counterexamples need to be frequently generated by the SMT solver during training, which leads to a slow training process. Especially for the complexity brought by high-dimensional systems, the SMT solving process is too complicated and subsequent training cannot be carried out. Summary of the invention

[0008] The purpose of the present invention is to provide a safe deep reinforcement learning method based on optimal counterexamples and obstacle functions that solves the above-mentioned problems.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is: a safe deep reinforcement learning method based on optimal counterexamples and obstacle functions, the method comprising the following steps:

[0010] Step A, using the standardized flow technology, constructing a non-negative unstandardized target probability distribution function based on the counterexample sample distribution, and training the standardized flow model based on the target distribution;

[0011] Step B, using the trained standardized flow model to sample counterexample samples, wherein the counterexample samples include counterexample sample data that most violates conditional constraints;

[0012] In step C, the generated counterexample samples are used to guide the obstacle function network to perform more refined correction training. By continuously generating and supplementing counterexample samples, deep reinforcement learning training is performed to eventually learn the correct obstacle function.

[0013] Preferably, in step A, the training method of the standardized flow model is as follows:

[0014] a. According to the specific controlled constrained continuous dynamic system, set the global value range of the system's state space, the range of the initial region and the unsafe region, and set the system's dynamic equations;

[0015] b. Construct three sets of non-negative, unstandardized target probability distribution functions that satisfy the distribution of counterexample samples, corresponding to the distribution of counterexample samples under the initial domain constraint, the non-safe domain constraint, and the lie condition constraint respectively;

[0016] c. Perform the first phase of cold start training to train relatively correct candidate controllers and candidate obstacle functions, as well as the corresponding standardized flow model.

[0017] Preferably, in step b, the three groups of target probability distribution functions are respectively and The corresponding log probability is as follows,

[0018]

[0019]

[0020]

[0021] Where: α~Uniform(0,1), constantc1,c2>0,λ(x) is the auxiliary function, G(x) is the range function, so that the probability distribution function only focuses on the probability distribution within the target range. B(x) is the barrier function.

[0022] The successful synthesis of the barrier function is a sufficient condition for the safety of the controller. For a dynamic system Γ=(f(x,u),X D ,X I ,X U ), the system is safe if there exists a continuously differentiable function B(x) such that the following three conditions are satisfied:

[0023]

[0024] where f is the system of dynamic equations that satisfies the local Lipschitz condition, u is the control input, X D is the system-wide domain, X I is the initial domain, X U It is an unsafe domain. is the lie derivative of B(x) with respect to f. Therefore, the three sets of target probability distribution functions and The probability distribution of counterexamples under the three conditions of the barrier function are shown respectively.

[0025] Preferably, in step B, if the loss function Loss of the candidate obstacle function is 0, that is, the obstacle function under the current training data is successfully synthesized, the standardized flow model is used to generate the optimal counterexample sample, and the optimal counterexample verification is performed; if the obstacle function condition constraints are met, formal verification is performed and the real obstacle function and safety controller are output, otherwise the counterexample sample is used to guide the obstacle function and reinforcement learning to perform further training.

[0026] Preferably, in step C, the second stage of the inductive iterative training process is carried out, and the training process of reinforcement learning adopts the feedback information of the obstacle function, and provides a LIE feedback reward function calculation method to enhance the training of reinforcement learning in the state space that violates the LIE condition.

[0027] As a preferred feature, it is characterized in that: the reward function calculation method of providing lie feedback is as follows:

[0028]

[0029] Where: s(t) represents the system state at the current moment, r min >0 is the threshold of punishment to avoid excessive changes in the reward value. If the current state s(t) violates the lie derivative condition constraint, additional punishment will be given to strengthen the training of reinforcement learning in this state.

[0030] Preferably, the barrier function network is trained more refined based on the training data supplemented with counter-example samples. Based on the learning results after training, if the distribution of counter-example samples changes accordingly, it is necessary to train and update the standardized flow model to obtain a model that conforms to the changed distribution of counter-example samples.

[0031] Preferably, in step C, by continuously generating and supplementing counterexample samples until the counterexample verification and formal verification are passed, a true barrier function network model and a safety controller network model are obtained.

[0032] Preferably, in step C, the barrier function finally obtained is formally verified to ensure the correctness of the barrier function.

[0033] Preferably, in step C, during the deep reinforcement learning training of the obstacle function, the behavior strategy of the controller around the counterexample point is improved by introducing the optimal counterexample sample into the interaction process between the agent and the environment.

[0034] Compared with the prior art, the advantages of the present invention are:

[0035] (1) The present invention optimizes the inductive iterative training efficiency of barrier functions and reinforcement learning through the optimal counterexample sample generation technology, and improves the efficiency of formal verification.

[0036] (2) The present invention utilizes the standardized flow technology to generate an optimal counterexample. Here, the optimal counterexample has two meanings. The first meaning refers to the sample set that makes the barrier function violate the barrier function constraints to the maximum extent. The second meaning refers to the sample set that makes the barrier function closest to the conditional boundary under the premise of satisfying the barrier function constraints.

[0037] (3) The traditional method of using SMT verifiers to generate counterexamples has inherent limitations and complex nonlinear problems, which leads to high computational complexity and high time cost in the solution process. However, the present application generates counterexamples through a standardized flow model, which is faster in terms of time.

[0038] (4) In addition, the counterexample points generated by the SMT solver cannot be guaranteed to be the samples that most violate the constraint conditions of the barrier function. The present invention, based on the standardized flow technology, can not only generate the optimal counterexample sample set, but also generate a counterexample sample set under multimodality. In probability distribution, multimodality refers to a probability distribution with multiple peaks or modes, and each mode usually represents a high probability area in the distribution. Based on the standardized flow technology, the generated counterexample samples are of higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 :Framework diagram of secure deep reinforcement learning method based on optimal counterexamples and obstacle functions;

[0040] Figure 2 : Schematic diagram of inverted pendulum system;

[0041] Figure 3 : Schematic diagram of the sampling process of the standardized flow technology.

[0042] Figure 4 : Candidate neural network obstacle function diagram for inverted pendulum system;

[0043] Figure 5 : Heat map of target probability density function of counterexample samples constrained by initial domain;

[0044] Figure 6 : Counterexample sample graph obtained by sampling the standardized flow model; DETAILED DESCRIPTION

[0045] The invention is based on the standardized flow technology, and the generation of counterexample samples has the advantages of fast speed and high quality. By using the optimal counterexample samples, the overall training efficiency of the algorithm can be improved. In addition, the conditional verification of the optimal counterexample proves that the barrier function has a larger safety threshold, which is a strong guarantee for the success of formal verification and can shorten the duration of the formal verification process.

[0046] The present invention will be further described below, and specific embodiments are as follows:

[0047] A secure deep reinforcement learning method based on optimal counterexamples and barrier functions, see Figure 1 This method takes the inverted pendulum system as an example. The inverted pendulum system is relatively simple and is a classic dynamic system control model. The system consists of a pendulum, one end of which is fixed to a fixed point and can rotate freely around the fixed point, and the other end is a free end. The pendulum starts from an initial state. The goal is to apply a torque to the free end to make it swing to a vertical position. The state of the system is represented by x = [θ, ω] T , θ is the angle, ω is the angular velocity. The schematic diagram of the inverted pendulum system is as follows Figure 2 . The following steps are included:

[0048] Step 1: Set the global value range X of the state space of the inverted pendulum system D , initial area X I and unsafe area X U Range:

[0049]

[0050]

[0051]

[0052] Set the system's dynamic equations f(x, u), where u is the control variable. The dynamic system is recorded as Γ=(f(x, u), X D ,X I ,X U ), the dynamic equations are as follows:

[0053]

[0054] Where m is the mass of the inverted pendulum, g is the acceleration due to gravity, and l is the length of the pendulum.

[0055] Step 2: Construct three sets of non-negative, unstandardized target probability distribution functions that satisfy the counterexample sample distribution: and The distribution of counterexample samples corresponding to the initial domain constraint, non-safe domain constraint and lie derivative constraint respectively, the corresponding logarithmic probability is as follows:

[0056]

[0057]

[0058]

[0059] Among them, α~Uniform(0.1), constant c1, c2>0, λ(x) is an auxiliary function, G(x) is a range function, so that the probability distribution function only focuses on the probability distribution within the target range. B(x) is the obstacle function. The successful synthesis of the obstacle function is a sufficient condition for the safety of the controller. For a dynamic system Γ=(f(x,u),X D ,X I ,X U ), if there exists a continuously differentiable function B(x) such that the following three conditions are satisfied, then the system is safe.

[0060]

[0061] where f is the system of dynamic equations that satisfies the local Lipschitz condition, u is the control input, X D is the system-wide domain, X I is the initial domain, X U It is a non-secure domain. is the lie derivative of B(x) with respect to f. Therefore, the three sets of target probability distribution functions and The probability distribution of the counterexamples under the three conditional constraints of the barrier function is shown respectively. The sampling process of the standardized flow technology can be found in Figure 3 .

[0062] Step 3: Perform the first phase of cold start training, that is, train deep reinforcement learning, deep obstacle function and normalized flow network respectively, and train relatively correct candidate controllers and candidate obstacle functions, as well as the corresponding normalized flow model. See the candidate neural network obstacle function diagram for details. Figure 4 , the target probability density function heat map of the counterexample sample with initial domain constraints is as follows Figure 5 shown.

[0063] By using the standardized flow technology, the optimal counterexample can be generated. The optimal counterexample here has two meanings. The first meaning refers to the sample set that makes the barrier function violate the barrier function constraints to the maximum extent. The second meaning refers to the sample set that makes the barrier function closest to the conditional boundary under the premise of satisfying the barrier function constraints.

[0064] Step 4: If the loss function Loss of the candidate barrier function is 0, that is, the barrier function under the current training data is successfully synthesized, then the standardized flow model is used to generate the optimal counterexample sample. The standardized flow model can not only generate the optimal counterexample sample set, but also generate the counterexample sample set under multimodality. In probability distribution, multimodality refers to a probability distribution with multiple peaks or modes, and each mode usually represents a high probability area in the distribution. Therefore, based on the standardized flow technology, the generated counterexample samples are of higher quality; perform the optimal counterexample verification. If the barrier function condition constraints are met, perform formal verification and output the real barrier function and safety controller. Otherwise, use the counterexample sample to guide the barrier function and reinforcement learning to perform further training. For counterexample samples sampled by the standardized flow model, see Figure 6 .

[0065] Conclusion, through Figure 6 It can be seen that the standardized flow-based method of the present invention can not only sample in the counterexample distribution area that most violates the constraint conditions, but also sample in the multimodal probability distribution area. Compared with the counterexamples generated by the traditional SMT solver, the counterexample samples of the present invention are of higher quality.

[0066] Step 5: Perform the second phase of the inductive iterative training process. The reinforcement learning training process uses the feedback information of the obstacle function to enhance the training of reinforcement learning in the state space that violates the lie condition. Specifically, the lie feedback reward function calculation method is provided:

[0067]

[0068] Where: s(t) represents the system state at the current moment, r min >0 is the threshold of punishment to avoid excessive changes in the reward value. If the current state s(t) violates the lie derivative condition constraint, additional punishment will be given to strengthen the training of reinforcement learning in this state.

[0069] Step 6: Based on the training data supplemented with counterexample samples, the barrier function network performs more refined training.

[0070] Step 7: Based on the learning results of steps 5 and 6, the distribution of counter-example samples changes accordingly. It is necessary to train and update the standardized flow model to obtain a model that conforms to the changed distribution of counter-example samples.

[0071] Step 8: Repeat steps 4 to 7 until the counterexample verification and formal verification pass, and obtain the real barrier function network model and safety controller network model.

[0072] Experimental conclusion: The counterexample sampling method proposed in this paper can efficiently generate high-quality counterexample samples, and guide the obstacle function and reinforcement learning to train a controller that meets the system safety requirements.

[0073] The above is a detailed introduction to a safe deep reinforcement learning method based on optimal counterexamples and obstacle functions provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes, and changes and improvements to the present invention will be possible without exceeding the concept and scope specified in the appended claims. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A secure deep reinforcement learning method based on optimal counterexamples and barrier functions, characterized by: The method comprises the following steps: Step A, using the standardized flow technology, constructing a non-negative unstandardized target probability distribution function based on the counterexample sample distribution, and training the standardized flow model based on the target distribution; Step B, using the trained standardized flow model to sample counterexample samples, wherein the counterexample samples include counterexample sample data that most violates conditional constraints; In step C, the generated counterexample samples are used to guide the obstacle function network to perform more refined correction training. By continuously generating and supplementing counterexample samples, deep reinforcement learning training is performed to eventually learn the correct obstacle function.

2. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 1, characterized in that: In step A, the training method of the standardized flow model is as follows: a. According to the specific controlled constrained continuous dynamic system, set the global value range of the system's state space, the range of the initial region and the unsafe region, and set the system's dynamic equations; b. Construct three sets of non-negative, unstandardized target probability distribution functions that satisfy the distribution of counterexample samples, corresponding to the distribution of counterexample samples under the initial domain constraint, the non-safe domain constraint, and the lie derivative constraint respectively; c. Perform the first phase of cold start training to train relatively correct candidate controllers and candidate obstacle functions, as well as the corresponding standardized flow model.

3. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 2, characterized in that: In step b, the three groups of target probability distribution functions are and The corresponding log probability is as follows, Where: α~Uniform(0,1), constant c1, c2>0, λ(x) is the auxiliary function, G(t) is the range function, which makes the probability distribution function only focus on the probability distribution within the target range, and B(x) is the barrier function.

4. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 1, characterized in that: In step B, if the loss function Loss of the candidate obstacle function is 0, that is, the obstacle function under the current training data is successfully synthesized, the standardized flow model is used to generate the optimal counterexample sample, and the optimal counterexample verification is performed; if the obstacle function condition constraints are met, formal verification is performed and the real obstacle function and safety controller are output, otherwise the counterexample sample is used to guide the obstacle function and reinforcement learning to perform further training.

5. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 1, characterized in that: In step C, the second stage of the inductive iterative training process is carried out. The reinforcement learning training process uses the feedback information of the obstacle function to provide a lie feedback reward function calculation method to enhance the training of reinforcement learning in the state space that violates the lie condition.

6. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 5, characterized in that: The calculation method of the reward function providing lie feedback is as follows: Where: s(t) represents the system state at the current moment, r min >0 is the threshold of punishment to avoid excessive changes in the reward value. If the current state s(t) violates the lie derivative condition constraint, additional punishment will be given to strengthen the training of reinforcement learning in this state.

7. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 5, characterized in that: Based on the training data supplemented with counter-example samples, the barrier function network is trained more refined. Based on the learning results after training, if the distribution of counter-example samples changes accordingly, it is necessary to train and update the standardized flow model to obtain a model that conforms to the changed distribution of counter-example samples.

8. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 1, characterized in that: In step C, by continuously generating and supplementing counterexample samples until the counterexample verification and formal verification are passed, the real barrier function network model and safety controller network model are obtained.

9. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 1, characterized in that: In step C, the final barrier function is formally verified.

10. The method for secure deep reinforcement learning based on optimal counterexamples and barrier functions according to claim 1, characterized in that: In step C, during the deep reinforcement learning training of the obstacle function, the optimal counterexample samples are introduced into the interaction between the agent and the environment to improve the behavior strategy of the controller around the counterexample points.

Images