A deep reinforcement learning training and decision-making method for solving physical constraints

Abstract

The present invention discloses a deep reinforcement learning training and decision-making method for solving physical constraints, and discloses a general reinforcement learning method capable of handling physical constraints, which ensures the satisfaction of hard constraints in the process of reinforcement learning decision-making. The present invention proposes a general reinforcement learning method for solving physical constraints, which can solve any form of decision-making problem with any number of differentiable constraints, and is not limited to a decision-making problem with a certain specific form of constraints; the physical perception layer proposed by the present invention is fully differentiable. Therefore, the policy network can obtain complete gradient information, so its training process is end-to-end, which is easy to implement and operate. Its training and update process is also more accurate; the present invention is committed to solving physical hard constraints in decision-making problems, which includes equality and inequality constraints, ensuring the feasibility of decision-making. This is of great significance to the implementation of reinforcement learning technology in practical applications.

Description

Technical Field

[0001] The present invention relates to a deep reinforcement learning training and decision-making method for solving physical constraints, and belongs to the field of artificial intelligence technology. Background Art

[0002] Reinforcement Learning is an artificial intelligence method for solving sequential decision problems. In recent years, it has been widely used in fields such as robot control, game competition, and autonomous driving. Usually, sequential decision problems are modeled as Markov decision processes, which are a five-tuple.<S,,,,> , where S represents the state space, A represents the action space, R represents the reward function, P represents the probability transfer function, and γ represents the discount factor. The optimization goal of traditional reinforcement learning is to maximize the expectation of the cumulative discounted reward, that is, However, the classic Markov decision process does not take into account the inherent constraints in real-world decision-making problems, which makes it difficult to apply traditional reinforcement learning to some real-world problems with physical constraints.

[0003] In this context, relevant experts and scholars have proposed a safety reinforcement learning technology (SafeReinforcement Learning) based on the modeling of Constrained Markov Decision Processes. Constrained Markov Decision Processes introduces a cost function C on the basis of Markov decision process. The goal of safety reinforcement learning is to maximize the reward target while satisfying the expected constraint of the cumulative discounted cost, that is, Among them, d i represents the i-th constraint. However, most secure reinforcement learning techniques only consider implicit inequality constraints and can only handle scenarios with a small number of constraints and relatively simple constraints. Real-world applications, however, involve many complex physical constraints, including not only inequality constraints but also equality constraints. This makes classic secure reinforcement learning techniques difficult to solve such problems.

[0004] On the other hand, existing machine learning methods for addressing physical constraints, particularly in the field of reinforcement learning, are mostly limited to specific applications or can only address specialized forms of constraints. Furthermore, there is currently no unified approach to modeling sequential decision-making problems with physical constraints, which has led to a lack of systematic research and methodologies in this area. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that existing reinforcement learning technologies cannot well meet the physical hard constraints in real-world applications (such as power grid operation tasks).

[0006] In order to better define and standardize decision problems with physical constraints, this paper first proposes a physical-constrained Markov decision process (Physics-Constrained Markov Decision Processes). On the basis of the original Markov decision process, two elements are introduced: the equality constraint strategy set and the and the set of strategies that satisfy the inequality constraints Our goal is to find the optimal strategy among the equality constraint strategy set and the inequality constraint strategy set, that is,

[0007] The specific technical solution disclosed in the present invention is to provide a deep reinforcement learning decision-making method for solving physical constraints, which is characterized by comprising the following steps:

[0008] Step 1: Add a differentiable physical perception layer after the output of the reinforcement learning policy network to process the actions output by the policy network that do not satisfy the physical constraints so that they satisfy the physical hard constraints. The physical perception layer includes an equation construction layer for solving the satisfaction of equality constraints and an inequality projection layer for solving the satisfaction of inequality constraints.

[0009] The present invention proposes a physical-aware reinforcement learning method. This method adds a differentiable physical-aware layer to the output of the original reinforcement learning policy network. The physical-aware layer has a drawback: when the initial action output by the policy network is far from the feasible domain specified by the inequality constraints, its runtime is greatly increased. Therefore, the physical-aware reinforcement learning method proposed in the present invention also includes a monotone primal-dual policy update algorithm, which ensures that the initial action output by the policy network is already closer to the feasible domain specified by the inequality constraints, reducing the runtime of the physical-aware layer.

[0010] Step 2: Given the current state s, use the policy network to decide the initial partial action a p ;

[0011] Step 3: Change part of action a p Input the equation construction layer, which solves the remaining action a by solving the equations defined by the equality constraints under the current state s. r , and finally get the complete action

[0012] Step 4: Complete the action Input inequality projection layer, inequality projection layer takes the exact penalty function corresponding to the inequality constraint under the current state s as the optimization target, and performs the optimization on the complete action. Perform multiple projection updates until all inequality constraints are satisfied and output the final feasible action a;

[0013] Step 5: The reinforcement learning agent performs the final action a.

[0014] Preferably, in step 2, assuming that the current actual problem has m equality constraints, the original action space R n Select nm actions as the action dimension that the policy network actually needs to decide, and call it partial action a p .

[0015] Preferably, in step 3, in the equation construction layer, when the partial action a p After determination, according to the m equality constraints, the remaining m actions a to be decided r Construct an m-dimensional equation system with m equations; then, solve the remaining m-dimensional actions by solving the m-dimensional equation system.

[0016] Preferably, in step 3, in the equation construction layer, when solving the m-variable equation group, the equation construction layer uses the implicit function theorem to obtain the remaining action a r For a p The Jacobian matrix of p To the remaining action a r Regardless of whether the calculation process is differentiable, the policy network can obtain the complete gradient backpropagated from the loss function.

[0017] Preferably, in step 3, the remaining action a r For some actions a p The Jacobian matrix of is:

[0018]

[0019] Among them, the remaining action a r = r ( p ),φ r Represents the process from part action a p To the remaining action a r The calculation process; For the equality constraint on state s, The Jacobian matrix of Represents the matrix J F The first to the mth column of Represents the matrix J F The m+1th column to the nth column.

[0020] The method for solving the equation in the present invention can be any existing method, such as Newton's method; or if the equation has a certain specific form, such as a linear equation, the remaining action a can be directly obtained. r However, if we directly use numerical methods such as Newton's method to solve it, the remaining action a r For some actions a p The Jacobian matrix of will not be obtained. This will result in the policy network being unable to obtain the complete gradient flow for training. Therefore, the equation construction layer uses the implicit function theorem to obtain the remaining action a r For a p The Jacobian matrix of .

[0021] Preferably, in step 4, the inequality projection layer takes the currently unsatisfied constraints as optimization targets and performs optimization on the low-dimensional manifold defined by the equality constraints until all inequality constraints are satisfied. The optimization process of the inequality projection layer is written as:

[0022]

[0023]

[0024]

[0025]

[0026] in: is the exact penalty function used by the inequality projection layer, represents the jth inequality constraint on action a under state s; a(k) represents the action after k projection updates, where a p () represents the partial action after k projection updates, a r () represents the remaining actions after k projection updates; is the step size of each projection update. It should be noted that since the final action a output by the inequality projection layer also needs to satisfy the equality constraint, the projection update must be performed on the low-dimensional manifold defined by the equality constraint. For nonlinear equality constraints, the Jacobian matrix Therefore, for scenarios with nonlinear equality constraints, It needs to be set small to prevent the updated action from deviating too much from the low-dimensional manifold defined by the equality constraints.

[0027] Another technical solution of the present invention is to provide an application of the above-mentioned deep reinforcement learning decision-making method for solving physical constraints in power grid operation, which is characterized by comprising the following steps:

[0028] Step 1: Given the current grid state s, which includes the active power demand and reactive power demand of each node in the grid and the current day-ahead electricity price, use the strategy network to decide the initial partial action a. p , the initial part action a p Including the active power generation and voltage amplitude of the power generation node and the active discharge / charge power of the battery;

[0029] Step 2: Move the initial part a p Input the equation construction layer, which solves the remaining action a by solving the equations defined by the equality constraints under the current state s r , the remaining action a r Including the reactive power generation of the power grid generation node and the voltage amplitude and angle of each node of the power grid, the complete action is finally obtained. Among them, the voltage amplitude of each node of the power grid does not include the voltage amplitude of the power generation node;

[0030] Step 3: Complete the action Input inequality projection layer, inequality projection layer takes the exact penalty function corresponding to the inequality constraint under the current state s as the optimization target, and performs the optimization on the complete action. Perform multiple projection updates until all inequality constraints are satisfied and output the final feasible action a;

[0031] Step 4: The reinforcement learning agent executes the final action a to the power grid control center.

[0032] Another technical solution of the present invention is to provide a deep reinforcement learning training method for solving physical constraints, characterized by comprising the following steps:

[0033] Step 1: The reinforcement learning agent uses the above decision-making method to interact with the environment / simulator, collects data and stores it in the experience replay pool;

[0034] Step 2: Randomly sample data from the experience replay pool;

[0035] Step 3: Use the sampled data to update the policy network with the unconstrained optimization problem as the goal. In this step, the policy set that satisfies the equality constraint is introduced based on the Markov decision process. and the set of strategies that satisfy the inequality constraints The goal is to find the optimal strategy in the equality constraint strategy set and the inequality constraint strategy set, that is, Where π represents the strategy, i represents the i-th constraint, j represents the j-th constraint, s represents the state, π(s) represents the action output by strategy π under state s, and f i(π(s)|s) represents the value of the i-th equality constraint under state s and policy π, g j (π(s)|s) represents the value of the jth inequality constraint under state s and policy π, J R (π) represents the expected cumulative discounted reward corresponding to strategy π;

[0036] Step 4: Use the sampled data to construct the target Q value, and update the value network with MSE loss as the target;

[0037] Step 5: Use the sampled data to update the penalty factor according to the monotone primal-dual update;

[0038] Step 6: Update the target network corresponding to the policy network and the value network in a soft update manner.

[0039] Preferably, in step 5, in the monotone primal-dual update, in order to make the initial partial actions output by the policy network in the reinforcement learning closer to the feasible region specified by the inequality, the original constrained decision problem is transformed into an equivalent unconstrained problem, as described in the following formula:

[0040]

[0041] Specifically, through the Lagrangian relaxation method, the penalty term is constructed in the objective function using the exact penalty function, and the converted unconstrained optimization problem is expressed as:

[0042]

[0043] in, is the penalty factor for the jth inequality constraint, when When , the unconstrained optimization problem is equivalent to the original constrained optimization problem. Therefore, in the monotone primal-dual update, the penalty factor Perform a monotonically growing update so that it will always satisfy the above inequality. The specific update formula is as follows:

[0044]

[0045] Where, represents the penalty factor corresponding to the jth inequality constraint after the kth update, η j represents the update step size of the j-th penalty factor, g j The meaning of (s,a) is the same as g(a|s).

[0046] Another technical solution of the present invention is to provide an application of the above-mentioned deep reinforcement learning training method for solving physical constraints in power grid operation, which is characterized by comprising the following steps:

[0047] Step 1: The reinforcement learning agent uses the above-mentioned decision-making method to interact with the real power grid environment or the power grid simulation program, collects data and stores it in the experience replay pool;

[0048] Step 2: Randomly sample data from the experience replay pool;

[0049] Step 3: Using the sampled data, the policy network is updated with an unconstrained optimization problem as the objective. Taking the grid operation application with a battery pack as an example, the goal is to minimize the generator's own power generation cost and maximize the battery's benefits in peak shaving and valley filling in the time dimension. At the same time, the physical equality constraints of the power flow balance equation and the physical inequality constraints of the feasible range of operation must also be considered.

[0050] Step 4: Use the sampled data to construct the target Q value, and update the value network with MSE loss as the target;

[0051] Step 5: Use the sampled data to update the penalty factor according to the monotone primal-dual update;

[0052] Step 6: Update the target network corresponding to the policy network and the value network in a soft update manner.

[0053] The present invention discloses a general reinforcement learning method capable of handling physical constraints, ensuring that hard constraints are satisfied during the reinforcement learning decision-making process. Compared with the existing technology, it has the following obvious advantages:

[0054] (1) The present invention proposes a general reinforcement learning method for solving physical constraints, which can solve any form of decision-making problem with any number of differentiable constraints, and is not limited to decision-making problems with a certain form of constraints (such as linear constraints).

[0055] (2) The physical perception layer proposed in this invention is fully differentiable. Therefore, the policy network can obtain complete gradient information, and its training process is end-to-end, which is easy to implement and operate. Its training and update process is also more accurate.

[0056] (3) This invention addresses the physical hard constraints in decision-making problems, including equality and inequality constraints, ensuring the feasibility of decision-making. This is of great significance for the practical application of reinforcement learning technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The structure of the policy network and value network used in the embodiment of the present invention;

[0058] Figure 2 Schematic diagram of the deep reinforcement learning decision-making method for solving physical constraints in the present invention;

[0059] Figure 3 Flowchart of the deep reinforcement learning training process for solving physical constraints in the present invention;

[0060] Figure 4 This is a performance comparison chart of the present invention in practical applications with other algorithms;

[0061] Figure 5 This is a performance comparison chart of the present invention using monotone primal-dual update and directly using a fixed multiplication factor;

[0062] Figure 6 The figure is a schematic block diagram showing a specific example of the present invention being applied to the operation control of a power grid with a battery. DETAILED DESCRIPTION

[0063] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0064] In practical applications of grid operation, our goal is to minimize the long-term grid generation cost, namely:

[0065]

[0066] Since the decision needs to satisfy the equality constraints of the power flow balance equation as shown below:

[0067] ( g ()-p d ()-p b ())+(q g ()-q d ())=diag(())(t)

[0068] And the inequality constraints on decision feasibility:

[0069]

[0070]

[0071]

[0072]

[0073] Here, p g Represents the active power generation of the power generation node, p g Represents p g The lower bound, Represents pg The upper bound of p d Represents the active power demand of each node, p b represents the battery discharge / charge power of a battery node, p b Represents p b The lower bound, Represents p b The upper bound of q g Represents the reactive power generation of the power generation node, q g Represents q g The lower bound, Represents q g The upper bound of q d Represents the reactive power demand of each node, v represents the voltage of each node, v represents the lower bound of v, represents the upper bound of v, Y represents the admittance matrix of the power grid, and A, b, and c represent the coefficient matrices / vectors related to power generation cost and electricity price, respectively.

[0074] In this scenario, traditional reinforcement learning methods are difficult to implement effectively because they cannot simultaneously and effectively satisfy both constraints. To address this issue, the present invention proposes a deep reinforcement learning method that addresses physical constraints, including a deep reinforcement learning decision-making method and a training method that address physical constraints.

[0075] The following will specifically illustrate a specific training method case of combining the present invention with the classic reinforcement learning method deep deterministic policy gradient method (DDPG), such as Figure 3 It specifically includes the following steps:

[0076] Step 1: Initialize the parameters θ of the policy network μ and the value network Q μ and θ Q , where the value network Q is used to evaluate the expectation of the cumulative reward that can be obtained by action a under a given state s, that is, the expectation of the subsequent cumulative reward corresponding to the input state s and the output of action a.

[0077] Step 2: To stabilize the training process of reinforcement learning, we need to initialize the parameters θ of the target policy network μ′ and the target value network Q′. μ′ ←θ μ ,θ Q′ ←θ Q .

[0078] Step 3: Initialize the experience replay pool

[0079] Step 4: Initialize the penalty factor v ∞ ←0.

[0080] Step 5: forepisode=1 to max_episode do

[0081] Step 501, for step=1 to max_step do

[0082] Step 5011: Select the initial part of the action in, Gaussian random noise explored for reinforcement learning, s t is the state at time t, μ(s t θ μ ) is in state s t Output of the policy network.

[0083] Step 5012: Partial Action Input the equation construction layer, which solves the remaining actions by solving the equations defined by the equality constraints under the current state s Finally get the complete action

[0084] Step 5013: Complete the action Input inequality projection layer, inequality projection layer takes the exact penalty function corresponding to the inequality constraint under the current state s as the optimization target, and performs the optimization on the complete action. Perform multiple projection updates until all inequality constraints are satisfied. The update formula is as follows:

[0085]

[0086] Output the final feasible action a=a(K).

[0087] Step 5014: The agent performs the final action a t And get reward r from observation in the environment t and the next state s t+1 .

[0088] Step 5015: Transfer the quad t ,a t ,r t ,s t+1 >Put it into experience replay pool D.

[0089] Step 5016: From the experience replay pool Collect batch data.

[0090] Step 5017: Update the policy network:

[0091]

[0092] Where, ​The network is in state s t and actions The output under η μ is the learning rate of the policy network.

[0093] Step 5018: Update the value network:

[0094]

[0095] In the formula, r(s t ,a t ) is state s t Take a given action a t The reward of environmental feedback, γ is the discount factor.

[0096] Step 5019: Update the penalty factor ν using the monotone dual update formula ∞ :

[0097]

[0098] Step 5020: Update the target network of the policy network and the value network in a soft update manner:

[0099] θ Q′ ←τθ Q +(1-τ)θ Q′ ,

[0100] θ μ′ ←τθ μ +(1-τ)θ μ′ .

[0101] end for

[0102] end for

[0103] In the actual application of power grid operation control, the above state s t is the current active power and reactive power demand of each node in the power grid and the day-ahead electricity price for the next 24 hours; action a t The active power and reactive power of the power generation nodes in the power grid, the voltage amplitude and angle of each node in the power grid, and the discharge / charge power of the battery; reward r t The sum of the generator power generation costs of each power generation node in the negative grid and the current discharge income or charging consumption of the battery bank is shown in Figure 2. Obviously, those skilled in the art can easily simulate the operation of the grid using a grid simulator to implement the above-mentioned interaction between the reinforcement learning agent and the grid environment.

[0104] We call the combination of the above-mentioned DDPG and a reinforcement learning method for solving physical constraints proposed in the present invention DDPG-PAL. Obviously, those skilled in the art should understand that the various operating steps of the above-mentioned examples of the present invention can be implemented by general computing devices, and the present invention can also be combined with other off-policy reinforcement learning algorithms, such as the Soft Actor Critic algorithm. In addition, in order to verify the versatility of the present invention, under the same hyperparameters, we experimentally compared it with the reinforcement learning method that previously handled soft constraints through reward reshaping and Lagrangian relaxation methods in three practical applications of robot control and power grid operation. Among them, the robot control application includes the safe car pole task (SafeCartPole) and the spring inverted pendulum task (Spring Pendulum), and the power grid operation task is the power flow optimization task (Optimal Power Flowwith Battery Energy System) with a battery system. The experimental results are shown in Figure 4 As can be seen from the figure, DDPG-PAL outperforms other algorithms in terms of maximizing cumulative rewards and satisfying physical constraints.

[0105] In addition, to verify the superiority of the monotone primal-dual algorithm over the fixed penalty factor algorithm, we compared the performance of the two algorithms in a power grid operation case. For the fairness of the experiment, we chose the value of the fixed multiplication factor after the monotone primal-dual algorithm penalty factor converged. The experimental results are as follows: Figure 5 It can be found that the monotone primal-dual update strategy is far superior to the strategy using a fixed penalty factor in maximizing the cumulative reward.

Claims

1. A power grid control method based on a deep reinforcement learning decision-making method for solving physical constraints, characterized by: The following steps are involved: Step 1: Add a differentiable physical perception layer after the output of the reinforcement learning policy network to process the actions output by the policy network that do not satisfy the physical constraints so that they satisfy the physical hard constraints. The physical perception layer includes an equation construction layer for solving the satisfaction of equality constraints and an inequality projection layer for solving the satisfaction of inequality constraints. Step 2: Given the current grid state s, which includes the active power demand and reactive power demand of each node in the grid and the current day-ahead electricity price, use the strategy network to decide the initial partial action a. p , the initial part action a p Including the active power generation and voltage amplitude of the power generation node and the active discharge / charge power of the battery; Step 3: Move the initial part a p Input the equation construction layer, which solves the remaining action a by solving the equations defined by the equality constraints under the current state s r , the remaining action a r Including the reactive power generation of the power grid generation node and the voltage amplitude and angle of each node of the power grid, the complete action is finally obtained. Among them, the voltage amplitude of each node of the power grid does not include the voltage amplitude of the power generation node; Step 4: Complete the action Input inequality projection layer, inequality projection layer takes the exact penalty function corresponding to the inequality constraint under the current state s as the optimization target, and performs the optimization on the complete action. Perform multiple projection updates until all inequality constraints are satisfied and output the final feasible action a; Step 5: The reinforcement learning agent executes the final action a to the power grid control center.

2. The power grid control method based on a deep reinforcement learning decision-making method for solving physical constraints according to claim 1, characterized in that: In step 2, assuming that the current actual problem has m equality constraints, the original action space R n Select nm actions as the action dimension that the policy network actually needs to decide, and call it partial action a p .

3. The power grid control method based on a deep reinforcement learning decision-making method for solving physical constraints according to claim 2, characterized in that: In step 3, in the equation construction layer, when the partial action a p After determination, according to the m equality constraints, the remaining m actions a to be decided r Construct an m-variable system of equations with m equations; Then, by solving the m-dimensional equation group, the remaining m-dimensional actions are solved.

4. The power grid control method based on a deep reinforcement learning decision-making method for solving physical constraints according to claim 3, characterized in that: In step 3, in the equation construction layer, when solving the m-element equation system, the equation construction layer uses the implicit function theorem to obtain the remaining action a r For a p The Jacobian matrix of p To the remaining action a r Regardless of whether the calculation process is differentiable, the policy network can obtain the complete gradient backpropagated from the loss function.

5. The power grid control method based on a deep reinforcement learning decision-making method for solving physical constraints according to claim 4, characterized in that: In step 3, the remaining action a r For some actions a p The Jacobian matrix of is: Among them, the remaining action a r =φ r (a p ),φ r Represents the process from part action a p To the remaining action a r The calculation process; For the equality constraint on state s, The Jacobian matrix of Represents the matrix J F The first to the mth column of Represents the matrix J F The m+1th column to the nth column.

6. The power grid control method based on a deep reinforcement learning decision-making method for solving physical constraints according to claim 1, characterized in that: In step 4, the inequality projection layer takes the currently unsatisfied constraints as the optimization target and optimizes on the low-dimensional manifold defined by the equality constraints until all inequality constraints are satisfied. The optimization process of the inequality projection layer is written as: in: is the exact penalty function used by the inequality projection layer, g j (a|s) represents the jth inequality constraint on action a in state s; a(k) represents the action after k projection updates, where a p (k) represents the partial action after k projection updates, a r (k) represents the remaining action after k projection updates; is the step size of each projection update.

7. A deep reinforcement learning training method for solving physical constraints, characterized by: The following steps are involved: Step 1: The reinforcement learning agent uses the power grid control method according to claim 1 to interact with the environment / simulator, collect data and store it in an experience replay pool; Step 2: Randomly sample data from the experience replay pool; Step 3: Use the sampled data to update the policy network with the unconstrained optimization problem as the goal. In this step, the policy set that satisfies the equality constraint is introduced based on the Markov decision process. and the set of strategies that satisfy the inequality constraints The goal is to find the optimal strategy in the equality constraint strategy set and the inequality constraint strategy set, that is, Where π represents the strategy, i represents the i-th constraint, j represents the j-th constraint, s represents the state, π(s) represents the action output by strategy π under state s, and f i (π(s)|s) represents the value of the i-th equality constraint under state s and policy π, g j (π(s)|s) represents the value of the jth inequality constraint under state s and strategy π, J R (π) represents the expected cumulative discounted reward corresponding to strategy π; Step 4: Use the sampled data to construct the target Q value, and update the value network with MSE loss as the target; Step 5: Use the sampled data to update the penalty factor according to the monotone primal-dual update; Step 6: Update the target network corresponding to the policy network and the value network in a soft update manner.

8. A deep reinforcement learning training method for solving physical constraints according to claim 7, characterized in that: In step 5, in the monotone primal-dual update, in order to make the initial partial actions output by the policy network in reinforcement learning closer to the feasible region specified by the inequality, the original constrained decision problem is transformed into an equivalent unconstrained problem, as described in the following formula: Specifically, through the Lagrangian relaxation method, the penalty term is constructed in the objective function using the exact penalty function, and the converted unconstrained optimization problem is expressed as: in, is the penalty factor for the jth inequality constraint, when When , the unconstrained optimization problem is equivalent to the original constrained optimization problem. Therefore, in the monotone primal-dual update, the penalty factor Perform a monotonically growing update so that it will always satisfy the above inequality. The specific update formula is as follows: represents the penalty factor corresponding to the jth inequality constraint after the kth update, η j Represents the update step size of the j-th penalty factor.

9. An application of the deep reinforcement learning training method for solving physical constraints as claimed in claim 7 in power grid operation, characterized in that: The following steps are involved: Step 1: The reinforcement learning agent uses the power grid control method according to claim 1 to interact with the real power grid environment or the power grid simulation program, collects data and stores it in the experience replay pool; Step 2: Randomly sample data from the experience replay pool; Step 3: Use the sampled data to update the policy network with the unconstrained optimization problem as the goal; Step 4: Use the sampled data to construct the target Q value, and update the value network with MSE loss as the target; Step 5: Use the sampled data to update the penalty factor according to the monotone primal-dual update; Step 6: Update the target network corresponding to the policy network and the value network in a soft update manner.

Images