A power distribution network dispatching method and system based on multi-stage security reinforcement learning

Abstract

The application discloses a power distribution network scheduling method and system based on multi-stage safety reinforcement learning, and belongs to the technical field of power system operation control, which comprises the following steps: constructing a cost reward function with the minimum power distribution network cost as the target, taking the candidate output adjustment amount of the cost scheduling subject as the cost action space, and constructing a cost agent; obtaining the actual output adjustment amount of the cost scheduling subject by using the cost agent according to the output boundary constraint of the cost scheduling subject; constructing an adaptive reward function with the maximum safety scheduling performance of the safety scheduling subject as the target, constructing a power grid safety penalty function with the maximum power distribution network safety as the target, taking the candidate output adjustment amount of the safety scheduling subject as the safety action space, and constructing a safety agent; obtaining the actual output adjustment amount of the safety scheduling subject by using the safety agent; and scheduling the power distribution network based on the actual output adjustment amount of each subject. The technical problem that the prior art is difficult to balance the safety of the power distribution network and the scheduling performance of new energy is solved.

Description

Technical Field

[0001] This invention relates to the field of power system operation and control technology, specifically to a distribution network dispatching method and system based on multi-stage security reinforcement learning. Background Technology

[0002] With the large-scale integration of distributed renewable energy sources, such as distributed photovoltaics, and new loads like electric vehicles and energy storage systems, into distribution networks, the operational characteristics of these networks are becoming increasingly complex. The significant increase in uncertainty and volatility on both the source and load sides makes traditional passive dispatching models inadequate. These models exhibit lag in response to real-time dynamic changes, easily leading to safety issues such as node voltage exceeding limits and line overloads. Simultaneously, conservative operating strategies adopted to mitigate risks limit the dispatching performance of renewable energy sources, increase network losses and electricity purchase costs, resulting in a decline in the overall economic efficiency of the system. To ensure the dispatching performance of renewable energy sources, existing technologies use reinforcement learning agents to explore the environment and gradually master control strategies under the complex uncertainties of the distribution network. However, traditional reinforcement learning methods primarily focus on maximizing long-term cumulative rewards. Their core exploration mechanism essentially allows the agent to attempt actions that may bring higher returns but with unknown risks. In distribution network dispatching scenarios, this mechanism is highly prone to generating unsafe actions such as voltage exceeding limits and equipment overloads. If these actions are executed in a real system, they will damage equipment lifespan, reduce power quality, and may even trigger grid failures, making it difficult to guarantee the safety of the distribution network. Therefore, directly applying traditional reinforcement learning to distribution network dispatch lacks an inherent security mechanism and poses unacceptable security risks. To ensure distribution network security, some research has proposed security reinforcement learning methods. The mainstream approach is to add the penalty term for violating security constraints to the reward function, allowing the reinforcement learning agent to learn and avoid penalties to achieve security. However, security constraints in distribution networks are diverse. Using a single penalty mechanism to handle all types of constraints lacks the ability to differentiate between the characteristics of different constraints. Excessive penalties limit the maximization of renewable energy dispatch performance, while insufficient penalties fail to guarantee distribution network security. Furthermore, there is a risk of violating regulations in pursuit of rewards and an overly conservative approach to avoid penalties. Therefore, balancing distribution network security and renewable energy dispatch performance remains a technical challenge that current technologies struggle to overcome. Summary of the Invention

[0003] To address the technical challenge of balancing distribution network security and renewable energy dispatch performance in existing technologies, this invention provides a distribution network dispatching method and system based on multi-stage security reinforcement learning. By decoupling phased decision-making and reward / penalty mechanisms, under the output boundary constraints of the cost dispatching entity, the cost agent determines its actual output adjustment amount. Simultaneously, a security agent is formed using an independently constructed adaptive reward function and a grid security penalty function. The security agent outputs the actual output adjustment amount of the security dispatching entity, thereby significantly improving renewable energy dispatch performance while ensuring distribution network security, thus solving the technical problem of balancing distribution network security and renewable energy dispatch performance in existing technologies.

[0004] To address the aforementioned technical problems, this invention provides a distribution network scheduling method based on multi-stage security reinforcement learning, comprising the following steps:

[0005] A cost reward function is constructed with the goal of minimizing the distribution network cost, and the candidate output adjustment amount of the cost scheduling subject is used as the cost action space. A cost intelligent agent is constructed based on the cost reward function and the cost action space.

[0006] Based on the output boundary constraints of the cost scheduling entity, the actual output adjustment amount of the cost scheduling entity is obtained using the cost intelligence agent;

[0007] An adaptive reward function is constructed with the goal of maximizing the safety scheduling performance of the safety scheduling subject, a grid safety penalty function is constructed with the goal of maximizing the safety of the distribution network, and the candidate output adjustment of the safety scheduling subject is used as the safety action space. A safety intelligent agent is constructed based on the adaptive reward function, the grid safety penalty function, and the safety action space.

[0008] Use a safety intelligent agent to obtain the actual output adjustment of the safety scheduling subject;

[0009] The power distribution network is dispatched based on the actual output adjustment of the cost dispatching entity and the actual output adjustment of the safety dispatching entity.

[0010] Preferably, the construction of the cost agent based on the cost-reward function and the cost action space includes:

[0011] The number of neurons in the first input layer is determined based on the number of state variables of the cost scheduling subject and the number of state variables of the distribution network node. The number of neurons in the first output layer is determined based on the number of action variables in the cost action space, thereby constructing a first policy neural network. The number of neurons in the second input layer is obtained by summing the number of neurons in the first input layer and the number of neurons in the first output layer, thereby constructing a first reward neural network.

[0012] A first policy loss function is constructed based on the guiding term that guides the first policy neural network to select the optimal action from the cost action space and the first predicted reward term that determines the quality of the selected action based on the representation obtained by the first reward neural network. The first policy neural network is trained using the first policy loss function. The first actual reward term that determines the quality of the selected action based on the cost reward function is obtained. The first reward loss function is obtained based on the first predicted reward term and the first actual reward term. The first reward neural network is trained based on the first reward loss function.

[0013] A cost agent is obtained based on the first policy neural network and the first reward neural network that have been trained.

[0014] Preferably, the step of obtaining the actual output adjustment amount of the cost scheduling subject using a cost intelligence agent based on the output boundary constraints of the cost scheduling subject includes:

[0015] A first penalty neural network is constructed based on a fully connected neural network, and a subject safety-performance penalty function is constructed with the goal of maximizing the safety of the cost scheduling subject and maximizing the cost scheduling performance of the cost scheduling subject.

[0016] The first actual penalty term for the merits and demerits of the selected action is obtained based on the subject safety-performance penalty function, and the first predicted penalty term for the merits and demerits of the selected action is obtained based on the first penalty neural network.

[0017] A first penalty loss function is obtained based on the first actual penalty term and the first predicted penalty term. A first penalty neural network is trained based on the first penalty loss function. The cost agent is improved based on the trained first penalty neural network, thereby obtaining the actual output adjustment amount of the cost scheduling agent.

[0018] The number of neurons in the input layer of the first penalty neural network is determined based on the number of constraint variables in the output boundary constraints, the number of state variables of the cost scheduling subject, and the number of action variables in the cost action space.

[0019] In this scheme, considering the diversity of safety between the cost scheduling entity and the distribution network, the output boundary constraints of the cost scheduling entity are used first, and the actual output adjustment amount of the cost scheduling entity is obtained by the cost agent. By decoupling the safety of the cost scheduling entity from the safety of the distribution network, the problems of low safety and low cost scheduling performance caused by multiple types of safety constraints being redundant in the same penalty term are avoided. In addition, considering that the existing methods only restrict the scheduling output after the fact through output boundary constraints, without optimizing the exploration and decision-making process of the cost agent itself, it is often necessary to repeatedly perform safety projection operations, which not only affects the efficiency of solving the output adjustment amount, but also makes it difficult to guarantee the scheduling safety from the root. To address this, this scheme constructs a first penalty neural network and a subject safety-performance penalty function to obtain the first actual penalty term and the first predicted penalty term, respectively. Based on the difference between the two, a first penalty loss function is constructed to train the first penalty neural network. The trained first penalty neural network is used to improve the cost agent, enabling the safety constraint information of the cost scheduling subject to directly affect the decision-making process of the cost agent in the form of gradient feedback. This avoids unsafe actions from the source of exploration, reduces redundant safety projection calculations, significantly improves the efficiency of output adjustment calculation, and enhances the safety of the cost scheduling subject.

[0020] Preferably, the step of constructing a first penalty neural network based on a fully connected neural network, and constructing a subject safety-performance penalty function with the goal of maximizing the safety of the cost scheduling subject and maximizing the cost scheduling performance of the cost scheduling subject, includes:

[0021] Based on a fully connected neural network, a security penalty neural network and a performance penalty neural network are constructed. The security penalty function of the subject is constructed with the goal of maximizing the security of the cost scheduling subject, and the performance penalty function of the subject is constructed with the goal of maximizing the cost scheduling performance of the cost scheduling subject.

[0022] Among them, the security penalty neural network and the performance penalty neural network constitute the first penalty neural network.

[0023] In this scheme, considering the fundamental differences between the security and cost scheduling performance of the cost scheduling entity in terms of optimization direction and constraint characteristics, and to prevent the problem of sacrificing cost scheduling performance for security or vice versa, a penalty function and a penalty neural network are constructed respectively to achieve decoupled learning and collaborative optimization of the two types of objectives. While minimizing the distribution network cost, the scheme also improves the security and cost scheduling performance of the cost scheduling entity, thereby improving the cost scheduling efficiency of the cost scheduling entity and saving scheduling resources.

[0024] Preferably, the improvement of the cost agent based on the trained first penalty neural network to obtain the actual output adjustment amount of the cost scheduling agent includes:

[0025] The policy loss function in the cost agent is improved based on the first penalty neural network that has been trained, and the policy neural network in the cost agent is trained based on the improved policy loss function to obtain the final cost agent.

[0026] The initial output adjustment amount of the cost scheduling subject is obtained through the final cost agent. If the initial output adjustment amount exceeds the output boundary constraint, the initial output adjustment amount is safely projected to obtain the actual output adjustment amount of the cost scheduling subject. If it does not exceed the boundary constraint, the initial output adjustment amount is used as the actual output adjustment amount of the cost scheduling subject.

[0027] Preferably, the construction of the safety intelligent agent based on the adaptive reward function, the power grid safety penalty function, and the safety action space includes:

[0028] The number of neurons in the third input layer is determined based on the number of state variables of the cost scheduling subject, the number of state variables of the distribution network nodes, and the number of state variables of the safety scheduling subject. The number of neurons in the second output layer is determined based on the number of action variables in the safety action space, thereby constructing the second policy neural network. The number of neurons in the fourth input layer is obtained by summing the number of neurons in the third input layer and the number of neurons in the second output layer, thereby constructing the second reward neural network.

[0029] A second penalty neural network is constructed based on a fully connected neural network. A second policy loss function is constructed based on a guiding term that selects the optimal action from the safe action space by a guiding second policy neural network, a second predicted reward term that selects the action's merits based on the representation obtained by the second reward neural network, and a second predicted penalty term that selects the action's merits based on the representation obtained by the second penalty neural network. The second policy neural network is then trained using the second policy loss function.

[0030] The second reward neural network and the second penalty neural network are trained based on the adaptive reward function and the power grid safety penalty function;

[0031] A safe agent is obtained based on the trained second policy neural network, the trained second reward neural network, and the trained second penalty neural network.

[0032] In this scheme, to avoid risky violations in pursuit of rewards and excessive conservatism to avoid punishment, the reward and punishment mechanisms are decoupled. This balances the safety of the distribution network with the safety scheduling performance of the safety scheduling agent. A second strategy loss function is constructed through a second prediction penalty term, allowing the safety constraint information of the distribution network to directly affect the decision-making process of the safety agent in the form of gradient feedback. This avoids unsafe actions from the source, achieves the synergistic optimization of the performance of the distribution network safety and the safety scheduling agent, and improves the safety scheduling efficiency of the safety scheduling agent.

[0033] Preferably, training the second reward neural network and the second penalty neural network based on the adaptive reward function and the power grid safety penalty function includes:

[0034] The second actual reward term representing the merits of the selected action is obtained based on the adaptive reward function; the second reward loss function is obtained based on the second predicted reward term and the second actual reward term; and the second reward neural network is trained based on the second reward loss function.

[0035] The second actual penalty term, which represents the merits of the selected action, is obtained based on the power grid safety penalty function. The second penalty loss function is obtained based on the second predicted penalty term and the second actual penalty term. The second penalty neural network is trained based on the second penalty loss function.

[0036] Preferably, before training the second policy neural network using the second policy loss function, the method further includes:

[0037] A stable penalty term is obtained by using the enhanced Lagrange method and the second prediction penalty term, and the loss function of the second policy is improved by the stable penalty term.

[0038] In this scheme, considering that the safety agent has difficulty in providing stable and smooth constraint gradients, which leads to problems such as slow convergence and policy oscillation in the power distribution network constraint optimization scenario, the enhanced Lagrangian method is used to obtain a stable penalty term to improve the loss function of the second policy, thereby improving the stability and convergence speed of the safety agent's policy learning.

[0039] Preferably, the improvement of the second policy loss function through a stability penalty term includes:

[0040] The adaptive loss function of the adaptive multiplier in the second policy loss function is obtained based on the second prediction penalty term and the stable penalty term. The adaptive multiplier is updated based on the adaptive loss function, and the updated adaptive multiplier, the stable penalty term and the second policy loss function are fused together.

[0041] In this scheme, by constructing an adaptive loss function to update the adaptive multiplier, the penalty intensity can be dynamically and adaptively adjusted according to the current constraint satisfaction level of the distribution network. This avoids the problems of excessively strong penalties leading to limited scheduling performance or insufficient penalties leading to the inability to guarantee safety constraints under a fixed penalty coefficient.

[0042] By adopting the above technical solution, the present invention has the following advantages:

[0043] By decoupling phased decision-making and reward / penalty mechanisms, under the output boundary constraints of the cost scheduling subject, the cost intelligence determines its actual output adjustment amount. At the same time, a safety intelligence is formed by using an independently constructed adaptive reward function and a power grid security penalty function. The safety intelligence outputs the actual output adjustment amount of the safety scheduling subject, thereby fully improving the new energy scheduling performance while ensuring the safety of the distribution network. This solves the technical problem that existing technologies cannot balance the safety of the distribution network and the performance of new energy scheduling.

[0044] Specifically, by constructing a first penalty neural network and a subject safety-performance penalty function, the first actual penalty term and the first predicted penalty term are obtained respectively. Based on the difference between the two, a first penalty loss function is constructed to train the first penalty neural network. The trained first penalty neural network is used to improve the cost agent, so that the safety constraint information of the cost scheduling subject can directly affect the decision-making process of the cost agent in the form of gradient feedback. This avoids unsafe actions from the source and reduces redundant safety projection calculations. While significantly improving the efficiency of solving the output adjustment, it also strengthens the safety of the cost scheduling subject. In addition, considering that the safety of the cost scheduling subject and the cost scheduling performance are fundamentally different in terms of optimization direction and constraint characteristics, in order to prevent the problem of sacrificing cost scheduling performance for safety or sacrificing safety for cost scheduling performance, a penalty function and a penalty neural network are constructed respectively. This achieves decoupled learning and collaborative optimization of the two types of objectives. While minimizing the distribution network cost, it also improves the safety and cost scheduling performance of the cost scheduling subject, thereby improving the cost scheduling efficiency of the cost scheduling subject and saving scheduling resources.

[0045] Specifically, to avoid risky violations in pursuit of rewards and excessive conservatism to avoid punishment, the reward and punishment mechanisms are decoupled. This balances the safety of the distribution network with the safety scheduling performance of the safety scheduling agent. A second strategy loss function is constructed through a second prediction penalty term, allowing the safety constraint information of the distribution network to directly affect the decision-making process of the safety agent in the form of gradient feedback. This avoids unsafe actions from the source, achieves the synergistic optimization of the performance of the distribution network safety and the safety scheduling agent, and improves the safety scheduling efficiency of the safety scheduling agent.

[0046] By using the enhanced Lagrange method to obtain a stable penalty term to improve the loss function of the second policy, the stability and convergence speed of the policy learning of the safe agent are improved. It also avoids the problems of excessively strong penalties leading to limited scheduling performance or insufficient penalties leading to the inability to guarantee safety constraints under a fixed penalty coefficient.

[0047] This scheme significantly improves the overall security of scheduling and the scheduling performance of each scheduling entity by combining phased decision-making, decoupling of reward and punishment mechanisms, decoupling of punishment mechanisms, and enhanced Lagrange method, and also significantly improves the overall scheduling efficiency.

[0048] This invention also provides a distribution network dispatching system based on multi-stage security reinforcement learning, applicable to the aforementioned distribution network dispatching method based on multi-stage security reinforcement learning, comprising:

[0049] The cost agent construction module is used to construct a cost reward function with the goal of minimizing the distribution network cost, and to construct a cost agent based on the candidate output adjustment of the cost scheduling subject as the cost action space, and the cost reward function and cost action space.

[0050] The first output adjustment acquisition module is used to obtain the actual output adjustment of the cost scheduling subject based on the output boundary constraints of the cost scheduling subject and using the cost intelligence agent.

[0051] The safety intelligent agent construction module is used to construct an adaptive reward function with the goal of maximizing the safety scheduling performance of the safety scheduling agent, construct a power grid safety penalty function with the goal of maximizing the safety of the distribution network, and construct a safety intelligent agent based on the candidate output adjustment of the safety scheduling agent as the safety action space, using the adaptive reward function, the power grid safety penalty function and the safety action space.

[0052] The second output adjustment acquisition module is used to acquire the actual output adjustment of the safety scheduling subject using the safety intelligent agent;

[0053] The scheduling module is used to schedule the distribution network based on the actual output adjustment of the cost scheduling entity and the actual output adjustment of the safety scheduling entity.

[0054] By adopting the above technical solution, the present invention has the following advantages:

[0055] By decoupling phased decision-making and reward / penalty mechanisms, under the output boundary constraints of the cost scheduling entity, the cost intelligence determines its actual output adjustment amount. At the same time, a safety intelligence is formed by using an independently constructed adaptive reward function and a power grid security penalty function. The safety intelligence outputs the actual output adjustment amount of the safety scheduling entity, thereby significantly improving the performance of new energy dispatching while ensuring the safety of the distribution network. This solves the technical problem that existing technologies cannot balance the safety of the distribution network and the performance of new energy dispatching. Attached Figure Description

[0056] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0057] Figure 1 This is a flowchart illustrating a distribution network scheduling method based on multi-stage security reinforcement learning according to the present invention.

[0058] Figure 2 This is a schematic diagram illustrating the process of obtaining the actual output adjustment of the cost scheduling subject in a distribution network scheduling method based on multi-stage security reinforcement learning according to the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only one preferred embodiment of this invention and are only used to explain this invention. They do not limit the scope of protection of this invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0060] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations (or steps) can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures; the process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0061] Example 1: As Figure 1 As shown, a distribution network scheduling method based on multi-stage security reinforcement learning includes the following steps:

[0062] S1: Construct a cost reward function with the goal of minimizing the distribution network cost, and use the candidate output adjustment amount of the cost scheduling subject as the cost action space. Construct a cost agent based on the cost reward function and the cost action space.

[0063] Understandably, the cost dispatching entity is a set of equipment in the distribution network that has independent output regulation capabilities and participates in grid power balance and economic dispatch. In this embodiment, the cost dispatching entity specifically refers to gas turbines and distributed photovoltaics. The candidate output regulation quantities of the cost dispatching entity include the candidate active power regulation quantities of gas turbines, the candidate reactive power regulation quantities of gas turbines, and the candidate reactive power regulation quantities of distributed photovoltaics. The cost action space consists of several sets of candidate active power regulation quantities of gas turbines, candidate reactive power regulation quantities of gas turbines, and candidate reactive power regulation quantities of distributed photovoltaics.

[0064] The cost-reward function is: ;in, This represents the cost-reward value at time t. This represents the unit output cost of a gas turbine. Let represent the set of active power regulation parameters of the gas turbine at time t. This represents the active power regulation of the gas turbine at time t. Indicates the electricity purchase price. This represents the active power that the distribution network receives from the upstream power grid at time t.

[0065] As an optional embodiment, the construction of the cost agent based on the cost-reward function and the cost action space includes:

[0066] The number of neurons in the first input layer is determined based on the number of state variables of the cost scheduling subject and the number of state variables of the distribution network node. The number of neurons in the first output layer is determined based on the number of action variables in the cost action space, thereby constructing a first policy neural network. The number of neurons in the second input layer is obtained by summing the number of neurons in the first input layer and the number of neurons in the first output layer, thereby constructing a first reward neural network.

[0067] A first policy loss function is constructed based on the guiding term that guides the first policy neural network to select the optimal action from the cost action space and the first predicted reward term that determines the quality of the selected action based on the representation obtained by the first reward neural network. The first policy neural network is trained using the first policy loss function. The first actual reward term that determines the quality of the selected action based on the cost reward function is obtained. The first reward loss function is obtained based on the first predicted reward term and the first actual reward term. The first reward neural network is trained based on the first reward loss function.

[0068] A cost agent is obtained based on the first policy neural network and the first reward neural network that have been trained.

[0069] The state variables of the cost scheduling entity specifically include the active and reactive power outputs of the gas turbine and the distributed photovoltaic system. The state variables of the distribution network nodes specifically include the active load, reactive load, and voltage amplitude of the distribution network nodes. It is understood that the state variables of the cost scheduling entity and the distribution network nodes are different at different times; therefore, neurons corresponding to each time moment need to be constructed. In this embodiment, the number of neurons in the first input layer is specifically 8. The action variables in the cost action space specifically include the active power regulation of the gas turbine, the reactive power regulation of the gas turbine, and the reactive power regulation of the distributed photovoltaic system. In this embodiment, the number of neurons in the first output layer is specifically 3. The first policy neural network is specifically a fully connected neural network, which also contains 3 hidden layers with a total of 256 hidden layer neurons. The first reward neural network is specifically a fully connected neural network, which also contains 3 hidden layers with a total of 256 hidden layer neurons.

[0070] The expression for the loss function of the first strategy is: ;in, These represent the training parameters of the first-policy neural network. This represents the first policy loss value under the training parameters. This indicates the number of the first interaction samples, specifically 128. Represents the first set of interactive samples. This represents the first current state of the distribution network at time t. , This represents the active load of a distribution network node at time t. This represents the reactive load of a distribution network node at time t. This represents the active power output of the gas turbine at time t. This represents the reactive power output of the gas turbine at time t. This represents the active power output of the distributed photovoltaic system at time t. This represents the reactive power output of the distributed photovoltaic system at time t. This represents the voltage amplitude at a node in the distribution network at time t. The coefficient representing the policy entropy is specifically 2. This represents the first-policy neural network. , This represents the action output by the first policy neural network in the first current state. This represents the probability distribution of the first-policy neural network selecting the output action in the first current state. This represents the guiding term that guides the first-policy neural network to select the optimal action from the cost action space. This represents the first reward neural network. The first predicted reward term characterizes the quality of the action output in the first current state. The specific form of the first interaction sample can be: , This represents the second current state of the distribution network at time t+1. , This represents the active power output of the gas turbine at time t+1. This represents the reactive power output of the gas turbine at time t+1. This represents the active power output of the distributed photovoltaic system at time t+1. This represents the reactive power output of distributed photovoltaic power at time t+1. This represents the active load of a distribution network node at time t+1. This represents the reactive load of the distribution network node at time t+1. This represents the voltage amplitude of the distribution network node at time t+1. The first interaction sample is used as the training sample, and the first policy neural network is trained using the first policy loss function. Specifically, the experience pool size for storing the first interaction sample is 8000.

[0071] The expression for the first reward loss function is: ;in, This represents the training parameters of the first reward neural network. This represents the first reward loss value under the training parameters. The first actual reward term, representing the quality of the action output in the first current state. , This indicates the discount rate for future rewards, specifically 0.99. This represents the first target evaluation neural network, whose network structure is exactly the same as the first reward neural network. The training parameters of the first target evaluation neural network are initialized to be identical to those of the first reward neural network. This represents the action output by the first policy neural network in the second current state. This indicates that the first objective evaluation neural network estimates the value of the action in the second current state and the output in the second current state. This represents the probability distribution of the first policy neural network selecting the output action in the second current state.

[0072] Training the first policy neural network using the first policy loss function specifically refers to updating the training parameters in the first policy neural network using the first policy loss value. The update expression is: ; The learning rate represents the training parameters in the first policy neural network, and can specifically be 0.0005. Training the first reward neural network based on the first reward loss function specifically refers to updating the training parameters in the first reward neural network using the first reward loss value. The update expression is: ; The learning rate represents the training parameters in the first reward neural network, and can specifically be 0.001. In this embodiment, the training parameters in the first target evaluation neural network are also updated using the training parameters in the first reward neural network, and the update expression is: ; The first objective is to evaluate the training parameters in the neural network. This indicates the learning rate, which represents the first objective in evaluating the training parameters of the neural network. When When the training is complete, it indicates that the first policy neural network, the first reward neural network, and the first target evaluation neural network have all been trained.

[0073] S2: Based on the output boundary constraints of the cost scheduling subject, use the cost agent to obtain the actual output adjustment amount of the cost scheduling subject.

[0074] As an alternative embodiment, such as Figure 2 As shown, the step of obtaining the actual output adjustment amount of the cost scheduling subject using a cost intelligence agent based on the output boundary constraints of the cost scheduling subject includes:

[0075] S21: Construct a first penalty neural network based on a fully connected neural network, and build a subject safety-performance penalty function with the goal of maximizing the safety of the cost scheduling subject and maximizing the cost scheduling performance of the cost scheduling subject;

[0076] S22: Obtain the first actual penalty term for the merits and demerits of the action selected by the representation based on the subject safety-performance penalty function, and obtain the first predicted penalty term for the merits and demerits of the action selected by the representation based on the first penalty neural network;

[0077] S23: Obtain a first penalty loss function based on the first actual penalty term and the first predicted penalty term, train a first penalty neural network based on the first penalty loss function, and improve the cost agent based on the trained first penalty neural network, thereby obtaining the actual output adjustment amount of the cost scheduling agent;

[0078] The number of neurons in the input layer of the first penalty neural network is determined based on the number of constraint variables in the output boundary constraints, the number of state variables of the cost scheduling subject, and the number of action variables in the cost action space.

[0079] The output boundary constraints include gas turbine ramping constraints and distributed photovoltaic capacity constraints. It can be understood that the number of constraint variables in the output boundary constraints is specifically two. The expression for the safety-performance penalty function is:

[0080] ;

[0081] in, This represents the security-performance penalty value at time t+1. This indicates the safety penalty factor for the gas turbine. This represents a collection of gas turbines. This represents the active power output of gas turbine i at time t+1. Let represent the active power output of gas turbine i at time t. This indicates the ramp limit for gas turbine i. This represents the safety penalty coefficient for distributed photovoltaic power generation. Represents a distributed photovoltaic collection. This represents the reactive power output of distributed photovoltaic system j at time t+1. This represents the total capacity of distributed photovoltaic (PV) system j. This represents the active power output of distributed photovoltaic system j at time t+1. This represents the performance penalty factor for a gas turbine. This represents the active power output of gas turbine i after safe projection at time t+1. This represents the performance penalty coefficient for distributed photovoltaic power generation. This represents the reactive power output of the distributed photovoltaic system after the secure projection of j at time t+1.

[0082] The expression for the first penalty loss function is: ; These represent the training parameters in the first penalty neural network. This represents the safety-performance loss value under the training parameters. Indicates the number of second interaction samples. This represents the set of second interaction samples. The specific form of the second interaction samples can be as follows: , This represents the safety-performance penalty value at time t. This represents the first penalty neural network. The first prediction penalty term characterizes the quality of the output action in the first current state. The first actual penalty term characterizes the quality of the output action in the first current state. , This represents the first target penalty neural network. The network structure of the first target penalty neural network is exactly the same as that of the first penalty neural network, and the training parameters of the first target penalty neural network are initialized to be the same as those of the first penalty neural network. This represents the value estimate of the first penalty neural network for the second current state and the action output in the second current state. Training the first penalty neural network based on the first penalty loss function specifically refers to updating the training parameters in the first penalty neural network using the safety-performance loss value. When the training parameters of the first penalty neural network make... This indicates that the first penalty neural network training is complete.

[0083] In this embodiment, considering the diversity of safety between the cost scheduling subject and the distribution network, the actual output adjustment amount of the cost scheduling subject is obtained directly using the output boundary constraints of the cost scheduling subject and the cost agent. By decoupling the safety of the cost scheduling subject from the safety of the distribution network, the problems of low safety and low cost scheduling performance caused by multiple types of safety constraints being redundant in the same penalty term are avoided. In addition, considering that the existing methods only restrict the scheduling output after the fact through output boundary constraints and do not optimize the exploration and decision-making process of the cost agent itself, it is often necessary to repeatedly perform safety projection operations, which not only affects the efficiency of solving the output adjustment amount, but also makes it difficult to guarantee scheduling safety from the root. To address this, this scheme constructs a first penalty neural network and a subject safety-performance penalty function to obtain the first actual penalty term and the first predicted penalty term, respectively. Based on the difference between the two, a first penalty loss function is constructed to train the first penalty neural network. The trained first penalty neural network is used to improve the cost agent, enabling the safety constraint information of the cost scheduling subject to directly affect the decision-making process of the cost agent in the form of gradient feedback. This avoids unsafe actions from the source of exploration, reduces redundant safety projection calculations, significantly improves the efficiency of output adjustment calculation, and enhances the safety of the cost scheduling subject.

[0084] As an optional embodiment, the construction of a first penalty neural network based on a fully connected neural network, and the construction of a subject safety-performance penalty function with the goal of maximizing the safety of the cost scheduling subject and maximizing the cost scheduling performance of the cost scheduling subject, includes:

[0085] Based on a fully connected neural network, a security penalty neural network and a performance penalty neural network are constructed. The security penalty function of the subject is constructed with the goal of maximizing the security of the cost scheduling subject, and the performance penalty function of the subject is constructed with the goal of maximizing the cost scheduling performance of the cost scheduling subject.

[0086] Among them, the security penalty neural network and the performance penalty neural network constitute the first penalty neural network.

[0087] The expression for the main security penalty function is: ; Let represent the main security penalty function value at time t+1. The expression for the main performance penalty function is: , Let represent the performance penalty function at time t+1. Understandably, corresponding loss functions for the security penalty neural network and the performance penalty neural network need to be designed and used to train the latter. The expression for the loss function of the security penalty neural network is: ; This represents the training parameters of the safety penalty neural network. This represents the safety loss value under the training parameters. Indicates the number of third-interaction samples. This represents the set of third interaction samples. Third interaction samples The specific forms of expression can be as follows: , Let represent the value of the subject's security penalty function at time t. The expression for the loss function of the performance penalty neural network is: ; The training parameters of the performance-penalized neural network are represented. This represents the performance loss value under the training parameters. Indicates the number of fourth interaction samples. This represents the set of fourth interaction samples. (Fourth interaction sample) The specific forms of expression can be as follows: , This represents the value of the performance penalty function at time t. In this embodiment, considering the fundamental differences between the security and cost scheduling performance of the cost scheduling entity in terms of optimization direction and constraint characteristics, and to prevent the problem of sacrificing cost scheduling performance for security or vice versa, a penalty function and a penalty neural network are constructed separately. This achieves decoupled learning and collaborative optimization of the two types of objectives, minimizing the distribution network cost while improving the security and cost scheduling performance of the cost scheduling entity. Through the decoupling of the two types of objectives, the scheduling role of the corresponding cost scheduling entity can be maximized, thereby improving the cost scheduling efficiency of the cost scheduling entity and saving scheduling resources.

[0088] Specifically, the improvement of the cost agent based on the trained first penalty neural network to obtain the actual output adjustment amount of the cost scheduling agent includes:

[0089] The policy loss function in the cost agent is improved based on the first penalty neural network that has been trained, and the policy neural network in the cost agent is trained based on the improved policy loss function to obtain the final cost agent.

[0090] The initial output adjustment amount of the cost scheduling subject is obtained through the final cost agent. If the initial output adjustment amount exceeds the output boundary constraint, the initial output adjustment amount is safely projected to obtain the actual output adjustment amount of the cost scheduling subject. If it does not exceed the boundary constraint, the initial output adjustment amount is used as the actual output adjustment amount of the cost scheduling subject.

[0091] The improved policy loss function is expressed as follows: Understandably, the actual output adjustment of the cost scheduling entity includes the actual active power output adjustment of the gas turbine, the actual reactive power output adjustment of the gas turbine, and the actual reactive power output adjustment of distributed photovoltaic power. Considering that exceeding the output boundary constraint will cause stress overload and operational risks to the gas turbine, when the initial active power output adjustment of the gas turbine obtained through the final cost intelligence exceeds the output boundary constraint, it needs to be safely projected. The actual active power output of the gas turbine at time t... , This represents the initial active power output of the gas turbine at time t. This represents the truncation function. This represents the active power output of the gas turbine at time t-1. This indicates the gradeability limit of the gas turbine. , , This represents the initial active power output adjustment of the gas turbine at time t. , This represents the actual active power output adjustment of the gas turbine at time t. The actual reactive power output of the distributed photovoltaic system at time t is also shown. ; This represents the initial reactive power output of the distributed photovoltaic system at time t. This represents the total capacity of distributed photovoltaic power. , This represents the initial reactive power output adjustment of the distributed photovoltaic system at time t. This represents the reactive power output of distributed photovoltaic power at time t-1. , , This represents the actual reactive power output regulation of distributed photovoltaic systems at time t. Let t represent the set of reactive power regulation quantities of distributed photovoltaic systems at time t.

[0092] S3: Construct an adaptive reward function with the goal of maximizing the safety scheduling performance of the safety scheduling subject, construct a power grid safety penalty function with the goal of maximizing the safety of the distribution network, and construct a safety intelligent agent based on the candidate output adjustment of the safety scheduling subject as the safety action space, using the adaptive reward function, the power grid safety penalty function and the safety action space.

[0093] The main body of safe dispatching is a collection of devices in the distribution network with safe regulation capabilities, used to ensure the voltage stability and branch power flow safety of the distribution network. In this embodiment, it specifically refers to output energy storage and reactive power compensators. The candidate output regulation quantities of the main body of safe dispatching include the candidate active power regulation quantities of output energy storage and the candidate reactive power regulation quantities of reactive power compensators. The safe action space consists of several sets of candidate active power regulation quantities of output energy storage and candidate reactive power regulation quantities of reactive power compensators.

[0094] The expression for the adaptive reward function is:

[0095] ; This represents the fit reward value at time t. Let represent the set of active power regulation quantities for energy storage output at time t. This represents the active power regulation of the energy storage output at time t. This represents the set of reactive power adjustment values ​​of the reactive power compensator at time t. This represents the reactive power adjustment of the reactive power compensator at time t. The expression for the power grid security penalty function is: ; This represents the safety penalty value of the distribution network at time t. This represents the voltage over-limit coefficient of the distribution network at time t. This represents the branch power flow limit exceedance coefficient of the distribution network at time t.

[0096] In some embodiments, the construction of a safety agent based on an adaptive reward function, a power grid safety penalty function, and a safety action space includes:

[0097] The number of neurons in the third input layer is determined based on the number of state variables of the cost scheduling subject, the number of state variables of the distribution network nodes, and the number of state variables of the safety scheduling subject. The number of neurons in the second output layer is determined based on the number of action variables in the safety action space, thereby constructing the second policy neural network. The number of neurons in the fourth input layer is obtained by summing the number of neurons in the third input layer and the number of neurons in the second output layer, thereby constructing the second reward neural network.

[0098] A second penalty neural network is constructed based on a fully connected neural network. A second policy loss function is constructed based on a guiding term that selects the optimal action from the safe action space by a guiding second policy neural network, a second predicted reward term that selects the action's merits based on the representation obtained by the second reward neural network, and a second predicted penalty term that selects the action's merits based on the representation obtained by the second penalty neural network. The second policy neural network is then trained using the second policy loss function.

[0099] The second reward neural network and the second penalty neural network are trained based on the adaptive reward function and the power grid safety penalty function;

[0100] A safe agent is obtained based on the trained second policy neural network, the trained second reward neural network, and the trained second penalty neural network.

[0101] In this embodiment, the state variables of the safety scheduling subject include the active power output of the output energy storage, the reactive power output of the reactive power compensator, and the state of charge of the output energy storage. It is understood that the number of neurons in the third input layer is 11. The action variables in the safety action space specifically include the active power adjustment of the output energy storage and the reactive power adjustment of the reactive power compensator. It is understood that the number of neurons in the second output layer is 2. In this embodiment, the second policy neural network is specifically a fully connected neural network, which also contains 3 hidden layers with a total of 256 neurons. The second reward neural network is specifically a fully connected neural network, which also contains 3 hidden layers with a total of 256 neurons.

[0102] The expression for the loss function of the second strategy is:

[0103] ; This represents the training parameters in the second-policy neural network. This represents the second policy loss value under the training parameters. This indicates the number of the fifth interaction sample, specifically 128. This represents the fifth set of interactive samples. This represents the third current state of the distribution network at time t. , This indicates the state of charge of the stored energy at time t. This represents the second policy neural network. , This represents the action output by the second policy neural network in the third current state. This represents the probability distribution of the second-policy neural network choosing the output action in the third current state. This represents the guiding term that guides the second-policy neural network to select the optimal action from the safe action space. This represents the second reward neural network. The second predicted reward term characterizes the quality of the action output in the third current state. This represents the second penalty neural network. The second prediction penalty term characterizes the quality of the action output in the third current state. The specific form of the fifth interaction sample can be as follows: , This represents the fourth current state of the distribution network at time t+1. The fifth interaction sample is used as a training sample, and the second policy neural network is trained using the second policy loss function. Specifically, the experience pool storing the fifth interaction sample is 8000. Training the second policy neural network using the second policy loss function specifically refers to updating the training parameters in the second policy neural network using the second policy loss value. The update expression is: ; The learning rate represents the training parameters in the second policy neural network, specifically 0.0005. In this embodiment, when... When the time is reached, it indicates that the second policy neural network, the second reward neural network, and the second penalty neural network have all been trained.

[0104] In this embodiment, to avoid the problems of risky violations in pursuit of rewards and overly conservative behavior to avoid punishment, the reward and punishment mechanisms are decoupled. This balances the safety of the distribution network with the safety scheduling performance of the safety scheduling agent. A second strategy loss function is constructed through a second prediction penalty term, allowing the distribution network safety constraint information to directly affect the decision-making process of the safety agent in the form of gradient feedback. This avoids unsafe actions from the source of exploration, achieves the synergistic optimization of the performance of the distribution network safety and the safety scheduling agent, and improves the safety scheduling efficiency of the safety scheduling agent.

[0105] In some embodiments, training the second reward neural network and the second penalty neural network based on the adaptive reward function and the power grid safety penalty function includes:

[0106] The second actual reward term representing the merits of the selected action is obtained based on the adaptive reward function; the second reward loss function is obtained based on the second predicted reward term and the second actual reward term; and the second reward neural network is trained based on the second reward loss function.

[0107] The second actual penalty term, which represents the merits of the selected action, is obtained based on the power grid safety penalty function. The second penalty loss function is obtained based on the second predicted penalty term and the second actual penalty term. The second penalty neural network is trained based on the second penalty loss function.

[0108] The expression for the second reward loss function is:

[0109] ;in, This represents the training parameters of the second reward neural network. This represents the second reward loss value under the training parameters. The second actual reward term characterizes the quality of the action output in the third current state. , This represents the second objective evaluation neural network, whose network structure is exactly the same as the second reward neural network. The training parameters of the second objective evaluation neural network are initialized to be identical to those of the second reward neural network. This represents the action output by the second policy neural network in the fourth current state. This indicates that the second objective evaluation neural network estimates the value of the action in the fourth current state and the output in the fourth current state. This represents the probability distribution of the second policy neural network choosing the action to output in the fourth current state.

[0110] The expression for the second penalty loss function is:

[0111] ; This represents the training parameters in the second penalty neural network. This represents the second penalty loss value under the training parameters. The second actual penalty term characterizes the quality of the action output in the third current state. , This represents the second-target penalty neural network, whose network structure is exactly the same as the second-target penalty neural network. The training parameters of the second-target penalty neural network are initialized to be identical to those of the second-target penalty neural network. This represents the value estimate of the second objective penalty neural network for the action output in the fourth current state and the fourth current state.

[0112] Training the second reward neural network based on the second reward loss function specifically refers to updating the training parameters in the second reward neural network using the second reward loss value. The update expression is: ; This represents the learning rate of the training parameters in the second reward neural network, which can be 0.001. Training the second penalty neural network based on the second penalty loss function specifically refers to updating the training parameters in the second penalty neural network using the second penalty loss value. The update expression is as follows: ; This represents the learning rate of the training parameters in the second penalty neural network, which can be 0.001.

[0113] In some embodiments, before training the second policy neural network using the second policy loss function, the method further includes:

[0114] A stable penalty term is obtained by using the enhanced Lagrange method and the second prediction penalty term, and the loss function of the second policy is improved by the stable penalty term.

[0115] Specifically, the improvement of the second policy loss function through a stability penalty term includes:

[0116] The adaptive loss function of the adaptive multiplier in the second policy loss function is obtained based on the second prediction penalty term and the stable penalty term. The adaptive multiplier is updated based on the adaptive loss function, and the updated adaptive multiplier, the stable penalty term and the second policy loss function are fused together.

[0117] The expression for the adaptive loss function is:

[0118] ; Represents the adaptive multiplier. Indicates the adaptive loss value. This represents the learning rate of the adaptive multiplier, which can specifically be 0.002. This represents the stability penalty term. The expression for the second policy loss function obtained after fusion is:

[0119] When the adaptive multiplier makes When the adaptive multiplier update is complete, it indicates that the update is finished.

[0120] S4: Use the safety intelligent agent to obtain the actual output adjustment of the safety scheduling subject.

[0121] S5: Dispatch the distribution network based on the actual output adjustment of the cost dispatching entity and the actual output adjustment of the safety dispatching entity.

[0122] The actual output adjustment of the safety dispatching entity specifically includes the actual active power output adjustment of the output energy storage and the actual reactive power output adjustment of the reactive power compensator. In this embodiment, the actual active power output adjustment of the gas turbine, the actual reactive power output adjustment of the gas turbine, and the actual reactive power output adjustment of the distributed photovoltaic system are obtained through a cost intelligent agent. The actual active power output adjustment of the output energy storage and the actual reactive power output adjustment of the reactive power compensator are obtained through a safety intelligent agent. This significantly improves the overall safety of dispatching and the dispatching performance of each dispatching entity, and also significantly improves the overall dispatching efficiency.

[0123] Example 2: This example also provides a distribution network dispatching system based on multi-stage security reinforcement learning, applicable to the aforementioned distribution network dispatching method based on multi-stage security reinforcement learning, including:

[0124] The cost agent construction module is used to construct a cost reward function with the goal of minimizing the distribution network cost, and to construct a cost agent based on the candidate output adjustment of the cost scheduling subject as the cost action space, and the cost reward function and cost action space.

[0125] The first output adjustment acquisition module is used to obtain the actual output adjustment of the cost scheduling subject based on the output boundary constraints of the cost scheduling subject and using the cost intelligence agent.

[0126] The safety intelligent agent construction module is used to construct an adaptive reward function with the goal of maximizing the safety scheduling performance of the safety scheduling agent, construct a power grid safety penalty function with the goal of maximizing the safety of the distribution network, and construct a safety intelligent agent based on the candidate output adjustment of the safety scheduling agent as the safety action space, using the adaptive reward function, the power grid safety penalty function and the safety action space.

[0127] The second output adjustment acquisition module is used to acquire the actual output adjustment of the safety scheduling subject using the safety intelligent agent;

[0128] The scheduling module is used to schedule the distribution network based on the actual output adjustment of the cost scheduling entity and the actual output adjustment of the safety scheduling entity.

[0129] The specific embodiments described above are preferred embodiments of a distribution network dispatching method and system based on multi-stage security reinforcement learning according to the present invention, and are not intended to limit the specific scope of the present invention. The scope of the present invention includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. A distribution network scheduling method based on multi-stage security reinforcement learning, characterized in that, Includes the following steps: A cost reward function is constructed with the goal of minimizing the distribution network cost, and the candidate output adjustment amount of the cost scheduling subject is used as the cost action space. A cost intelligent agent is constructed based on the cost reward function and the cost action space. Based on the output boundary constraints of the cost scheduling entity, the actual output adjustment amount of the cost scheduling entity is obtained using the cost intelligence agent; An adaptive reward function is constructed with the goal of maximizing the safety scheduling performance of the safety scheduling subject, a grid safety penalty function is constructed with the goal of maximizing the safety of the distribution network, and the candidate output adjustment of the safety scheduling subject is used as the safety action space. A safety intelligent agent is constructed based on the adaptive reward function, the grid safety penalty function, and the safety action space. Use a safety intelligent agent to obtain the actual output adjustment of the safety scheduling subject; The distribution network is dispatched based on the actual output adjustment of the cost dispatching entity and the actual output adjustment of the safety dispatching entity. The step of obtaining the actual output adjustment amount of the cost scheduling subject using a cost intelligence agent based on the output boundary constraints of the cost scheduling subject includes: A first penalty neural network is constructed based on a fully connected neural network, and a subject safety-performance penalty function is constructed with the goal of maximizing the safety of the cost scheduling subject and maximizing the cost scheduling performance of the cost scheduling subject. The first actual penalty term for the merits and demerits of the selected action is obtained based on the subject safety-performance penalty function, and the first predicted penalty term for the merits and demerits of the selected action is obtained based on the first penalty neural network. A first penalty loss function is obtained based on the first actual penalty term and the first predicted penalty term. A first penalty neural network is trained based on the first penalty loss function. The policy loss function in the cost agent is improved based on the trained first penalty neural network, thereby obtaining the actual output adjustment amount of the cost scheduling agent. The number of neurons in the input layer of the first penalty neural network is determined based on the number of constraint variables in the output boundary constraints, the number of state variables of the cost scheduling subject, and the number of action variables in the cost action space. The inputs of the first penalty neural network are the first current state of the distribution network at time t and the action output by the first policy neural network in the first current state. The output of the first penalty neural network is the first prediction penalty term that characterizes the quality of the action output in the first current state.

2. The distribution network dispatching method based on multi-stage security reinforcement learning according to claim 1, characterized in that, The construction of the cost-based intelligent agent based on the cost-reward function and cost action space includes: The number of neurons in the first input layer is determined based on the number of state variables of the cost scheduling subject and the number of state variables of the distribution network node. The number of neurons in the first output layer is determined based on the number of action variables in the cost action space, thereby constructing a first policy neural network. The number of neurons in the second input layer is obtained by summing the number of neurons in the first input layer and the number of neurons in the first output layer, thereby constructing a first reward neural network. A first policy loss function is constructed based on the guiding term that guides the first policy neural network to select the optimal action from the cost action space and the first predicted reward term that determines the quality of the selected action based on the representation obtained by the first reward neural network. The first policy neural network is trained using the first policy loss function. The first actual reward term that determines the quality of the selected action based on the cost reward function is obtained. The first reward loss function is obtained based on the first predicted reward term and the first actual reward term. The first reward neural network is trained based on the first reward loss function. A cost agent is obtained based on the first policy neural network and the first reward neural network that have been trained.

3. The distribution network dispatching method based on multi-stage security reinforcement learning according to claim 1, characterized in that, The first penalty neural network, constructed based on a fully connected neural network, aims to maximize both the safety and cost scheduling performance of the cost scheduling subject, and establishes a subject safety-performance penalty function, including: Based on a fully connected neural network, a security penalty neural network and a performance penalty neural network are constructed. The security penalty function of the subject is constructed with the goal of maximizing the security of the cost scheduling subject, and the performance penalty function of the subject is constructed with the goal of maximizing the cost scheduling performance of the cost scheduling subject. Among them, the security penalty neural network and the performance penalty neural network constitute the first penalty neural network; The input to the safety penalty neural network is the first current state of the distribution network at time t and the action output by the first strategy neural network in the first current state. The output of the safety penalty neural network is a predicted safety penalty term that characterizes the quality of the action output in the first current state. The input to the performance penalty neural network is the first current state of the distribution network at time t and the action output by the first strategy neural network in the first current state. The output of the performance penalty neural network is a predicted performance penalty term that characterizes the quality of the action output in the first current state.

4. The distribution network dispatching method based on multi-stage security reinforcement learning according to claim 1, characterized in that, The improved policy loss function of the cost agent based on the trained first penalty neural network is used to obtain the actual output adjustment of the cost scheduling agent, including: The final cost agent is obtained by training the policy neural network in the cost agent based on the improved policy loss function. The initial output adjustment amount of the cost scheduling subject is obtained through the final cost agent. If the initial output adjustment amount exceeds the output boundary constraint, the initial output adjustment amount is safely projected to obtain the actual output adjustment amount of the cost scheduling subject. If it does not exceed the boundary constraint, the initial output adjustment amount is used as the actual output adjustment amount of the cost scheduling subject.

5. The distribution network dispatching method based on multi-stage security reinforcement learning according to claim 1, characterized in that, The construction of a safety intelligent agent based on the adaptive reward function, the power grid safety penalty function, and the safety action space includes: The number of neurons in the third input layer is determined based on the number of state variables of the cost scheduling subject, the number of state variables of the distribution network nodes, and the number of state variables of the safety scheduling subject. The number of neurons in the second output layer is determined based on the number of action variables in the safety action space, thereby constructing the second policy neural network. The number of neurons in the fourth input layer is obtained by summing the number of neurons in the third input layer and the number of neurons in the second output layer, thereby constructing the second reward neural network. A second penalty neural network is constructed based on a fully connected neural network. A second policy loss function is constructed based on a guiding term that selects the optimal action from the safe action space by a guiding second policy neural network, a second predicted reward term that selects the action's merits based on the representation obtained by the second reward neural network, and a second predicted penalty term that selects the action's merits based on the representation obtained by the second penalty neural network. The second policy neural network is then trained using the second policy loss function. The second reward neural network and the second penalty neural network are trained based on the adaptive reward function and the power grid safety penalty function; A safe agent is obtained based on the trained second policy neural network, the trained second reward neural network, and the trained second penalty neural network; The input to the second penalty neural network is the third current state of the distribution network at time t and the action output by the second strategy neural network in the third current state. The output of the second penalty neural network is a second prediction penalty term that characterizes the quality of the action output in the third current state.

6. A distribution network scheduling method based on multi-stage security reinforcement learning according to claim 5, characterized in that, The training of the second reward neural network and the second penalty neural network based on the adaptive reward function and the power grid safety penalty function includes: The second actual reward term representing the merits of the selected action is obtained based on the adaptive reward function; the second reward loss function is obtained based on the second predicted reward term and the second actual reward term; and the second reward neural network is trained based on the second reward loss function. The second actual penalty term, which represents the merits of the selected action, is obtained based on the power grid safety penalty function. The second penalty loss function is obtained based on the second predicted penalty term and the second actual penalty term. The second penalty neural network is trained based on the second penalty loss function.

7. A distribution network dispatching method based on multi-stage security reinforcement learning according to claim 5, characterized in that, Before training the second-policy neural network using the second-policy loss function, the method further includes: A stable penalty term is obtained by using the enhanced Lagrange method and the second prediction penalty term, and the loss function of the second policy is improved by the stable penalty term.

8. A distribution network scheduling method based on multi-stage security reinforcement learning according to claim 7, characterized in that, The improvement of the second policy loss function through a stable penalty term includes: The adaptive loss function of the adaptive multiplier in the second policy loss function is obtained based on the second prediction penalty term and the stable penalty term. The adaptive multiplier is updated based on the adaptive loss function, and the updated adaptive multiplier, the stable penalty term and the second policy loss function are fused together.

9. A distribution network dispatching system based on multi-stage security reinforcement learning, applicable to the distribution network dispatching method based on multi-stage security reinforcement learning as described in any one of claims 1-8, characterized in that, include: The cost agent construction module is used to construct a cost reward function with the goal of minimizing the distribution network cost, and to construct a cost agent based on the candidate output adjustment of the cost scheduling subject as the cost action space, and the cost reward function and cost action space. The first output adjustment acquisition module is used to obtain the actual output adjustment of the cost scheduling subject based on the output boundary constraints of the cost scheduling subject and using the cost intelligence agent. The safety intelligent agent construction module is used to construct an adaptive reward function with the goal of maximizing the safety scheduling performance of the safety scheduling agent, construct a power grid safety penalty function with the goal of maximizing the safety of the distribution network, and construct a safety intelligent agent based on the candidate output adjustment of the safety scheduling agent as the safety action space, using the adaptive reward function, the power grid safety penalty function and the safety action space. The second output adjustment acquisition module is used to acquire the actual output adjustment of the safety scheduling subject using the safety intelligent agent; The scheduling module is used to schedule the distribution network based on the actual output adjustment of the cost scheduling entity and the actual output adjustment of the safety scheduling entity.

Images