Method, apparatus, equipment and readable storage medium for generating control strategy samples

Abstract

This application discloses a method, apparatus, device, and readable storage medium for generating control strategy samples. The method obtains system operating points, as well as a policy agent and an adversarial agent trained interactively. The adversarial agent is trained with the generation probability of controlling extreme operating points as the target. Using the adversarial agent, the system operating points are processed to generate adversarial operating states. The adversarial operating states are input to the policy agent to generate control strategies matching the adversarial operating states. The adversarial operating states and control strategies form control strategy samples. Based on this, this application can utilize the adversarial agent to generate various adversarial operating states, even extreme operating states, and guide the policy agent to provide matching control strategies for different operating states, ensuring the extreme nature and versatility of the adversarial operating states and their matching control strategies. Therefore, this application can generate various control strategy samples that guarantee both extreme nature and versatility.

Description

Technical Field

[0001] This application relates to the field of power grid technology, and more specifically, to a method, apparatus, device, and readable storage medium for generating control strategy samples. Background Technology

[0002] With the increasing penetration of renewable energy, the randomness and uncertainty of power systems have increased significantly, posing a huge challenge to the formulation of traditional emergency control strategies. In recent years, in order to address these issues, attempts have been made to apply deep reinforcement learning models to the formulation of emergency control strategies for power systems. These models generate corresponding emergency control strategies based on the operating state of the power grid system, thereby maintaining the safe and stable operation of the power grid system.

[0003] However, training deep reinforcement learning models requires manually selecting samples located at the edge of the operating space based on experience or sampling methods such as Monte Carlo. This method heavily relies on domain experience; insufficient experience can lead to samples lacking extrema and having poor generalizability, resulting in shortcomings in deep reinforcement learning models when handling safety constraints and high-risk scenarios. Therefore, how to generate samples for training deep reinforcement learning models has become a key focus for researchers in this field. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, device and readable storage medium for generating control strategy samples, for generating training samples for deep reinforcement learning models.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] A method for generating control strategy samples, comprising:

[0007] The system operating point is obtained, as well as the policy agent and adversarial agent trained interactively, wherein the adversarial agent is trained with the goal of controlling the generation probability of extreme operating points;

[0008] The adversarial agent is used to process the system's operating points to generate an adversarial operating state;

[0009] The adversarial operating state is input into the policy agent to generate a control policy that matches the adversarial operating state. The adversarial operating state and the control policy form a control policy sample.

[0010] Optionally, obtain the policy agent and adversarial agent trained interactively, including:

[0011] Generate initial adversarial agents and initial policy agents;

[0012] With the goal of adjusting the probability of generating extreme operating points of the initial adversarial agent, the parameters of the initial adversarial agent are adjusted;

[0013] By combining zero-sum game theory, the initial policy agent and the initial adversarial agent after parameter adjustment are trained alternately. The final initial policy agent is used as the policy agent, and the final initial adversarial agent is used as the adversarial agent.

[0014] Optionally, generating an initial adversarial agent includes:

[0015] Generate an intelligent agent constructed by combining the SAC algorithm and a single-step Markov decision process;

[0016] Multiple historical running points are sequentially input into the intelligent agent to obtain the predicted running point corresponding to each historical running point;

[0017] Based on KL divergence, the similarity between each historical running point and its corresponding predicted running point is calculated;

[0018] Based on each similarity score, the parameters of the latest agent are adjusted until the similarity score meets the preset similarity threshold. The resulting agent is then used as the initial adversarial agent.

[0019] Optionally, adjusting the parameters of the initial adversarial agent with the goal of controlling the probability of generating extreme operating points includes:

[0020] Based on the ε-greedy algorithm, the parameters of the initial adversarial agent are adjusted with the goal of adjusting the probability of generating extreme running points.

[0021] Optionally, the step of combining zero-sum game theory with alternating training of the initial policy agent and the parameter-adjusted initial adversarial agent includes:

[0022] Determine the reward functions for the initial adversarial agent and the initial policy agent based on a zero-sum game setting;

[0023] Stop adjusting the parameters of the initial adversarial agent and obtain two types of training running states. One type of training running state is a running state randomly selected from the power grid system; the other type of training running state is multiple running states obtained based on the latest processing of the initial adversarial agent.

[0024] Each training running state is sequentially input into the latest initial policy agent to obtain the training control policy corresponding to each training running state;

[0025] Based on each training running state and its corresponding training control strategy, combined with the reward function of the initial policy agent, the corresponding training reward value is calculated, and the parameters of the latest initial policy agent are adjusted based on each training reward value until the stopping condition is met.

[0026] Stop adjusting the parameters of the initial policy agent and obtain multiple target running points;

[0027] Each target running point is sequentially input into the latest initial adversarial agent to generate the target running state corresponding to each target running point;

[0028] Each target's running state is sequentially input into the latest initial policy agent to obtain the target control policy corresponding to each target's running state;

[0029] Based on the running state of each target and its corresponding target control strategy, combined with the reward function of the initial adversarial agent, the corresponding target reward value is calculated, and the parameters of the initial adversarial agent are adjusted based on each target reward value until the stopping condition is met.

[0030] Return to the previous step to stop adjusting the parameters of the initial adversarial agent and obtaining the two training states, until both the initial adversarial agent and the initial policy agent converge.

[0031] Optionally, determining the reward function for the initial adversarial agent and the initial policy agent based on the zero-sum game setting includes:

[0032] Set the reward function for the initial policy agent, and set the reward function for the initial adversarial agent to be the negative of the lower bound of the initial policy agent's reward in the sample space.

[0033] Optionally, the step of using the adversarial agent to process the system's operating points and generate an adversarial operating state includes:

[0034] The system operating point is input into the adversarial agent to obtain the adversarial operating point output by the adversarial agent;

[0035] The adversarial operation point is simulated to obtain the adversarial operation state.

[0036] A control strategy sample generation device, comprising:

[0037] The acquisition module is used to acquire the system running point, as well as the policy agent and adversarial agent trained interactively, wherein the adversarial agent is trained with the goal of controlling the generation probability of extreme running points;

[0038] The processing module is used to process the system's operating points using the adversarial agent to generate an adversarial operating state;

[0039] The generation module is used to input the adversarial operation state into the policy agent and generate a control policy that matches the adversarial operation state. The adversarial operation state and the control policy form a control policy sample.

[0040] A control strategy sample generation device includes a memory and a processor;

[0041] The memory is used to store programs;

[0042] The processor is used to execute the program to implement each step of the control strategy sample generation method described above.

[0043] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the control strategy sample generation method described above.

[0044] As can be seen from the above technical solution, the control strategy sample generation method provided in this application can obtain a strategy agent and an adversarial agent trained with the generation probability of controlling extreme operating points as the target. The adversarial agent and the strategy agent undergo interactive training. Based on this, a dynamic balance can be maintained between the adversarial agent and the strategy agent, ensuring their stability and matching. Subsequently, the adversarial agent can be used to process the acquired system operating points to generate adversarial operating states. These adversarial operating states are then input into the strategy agent to generate a control strategy matching the adversarial operating states. Since the adversarial agent is trained with the generation probability of controlling extreme operating points as the target, and the adversarial operating states generated by the adversarial agent can be input into the strategy agent, various adversarial operating states, even extreme operating states, can be generated using the adversarial agent. This guides the strategy agent to provide matching control strategies for different operating states, expanding the data dimensions of both the adversarial operating states and the control strategies, ensuring the extreme nature and versatility of the adversarial operating states and their matching control strategies. Based on this, the adversarial operating state and the control strategy form a control strategy sample. The adversarial operating state of the control strategy sample corresponds to the input of the deep reinforcement learning model applied to the formulation of emergency control strategies, and the control strategy of the control strategy sample corresponds to the output of the deep reinforcement learning model. Therefore, the control strategy sample generated in this application can be used to train the deep reinforcement learning model. Since the adversarial operating state and control strategy of this application can correspond to both extreme and normal scenarios, it has strong extremism and versatility. Therefore, using the control strategy sample of this application to train the deep reinforcement learning model can further improve the robustness and reliability of emergency control formulation. It can be seen that this application can generate a variety of control strategy samples that guarantee extremism and versatility to improve the robustness and reliability of emergency control. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a flowchart of a control strategy sample generation method disclosed in an embodiment of this application;

[0047] Figure 2 This is a structural block diagram of a control strategy sample generation device disclosed in an embodiment of this application;

[0048] Figure 3This is a hardware structure block diagram of a control strategy sample generation device disclosed in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] This application provides a method for generating control strategy samples. This method can be applied to various power grid systems or training systems, and can also be applied to various computer terminals or smart terminals. The executing entity can be the processor or server of the computer terminal or smart terminal. The flowchart of the control strategy sample generation method is shown below. Figure 1 As shown, it specifically includes:

[0051] Step S1: Obtain the system running point, as well as the policy agent and adversarial agent that have been interactively trained.

[0052] Specifically, given the complexity of power systems, it is impractical to comprehensively explore all possible operating points. Therefore, identifying and generating extreme scenarios, and training agents under these scenarios, can enhance the robustness of existing strategies and thus improve the performance of the agents.

[0053] Based on this, adversarial agents can be trained to control the generation probability of extreme running points.

[0054] Adversarial agents can generate adversarial execution points based on system execution points.

[0055] Since adversarial agents have a certain probability of generating extreme operating points, adversarial operating points also have a certain probability of being extreme operating points.

[0056] Extreme operating points can correspond to various extreme scenarios in the power grid system. Therefore, adversarial agents can be used to explore these extreme scenarios.

[0057] Policy agents can be used to reduce violations in control policy samples, while adversarial agents can be used to generate extreme and challenging run points to increase the data dimensionality of policy agents.

[0058] The policy agent and the adversarial agent can be trained alternately using the semantic role labeling algorithm GAN-SRL based on generative adversarial networks.

[0059] It can periodically collect the latest system operating points from the power grid system.

[0060] Step S2: Using the adversarial agent, process the system operation point to generate an adversarial operation state.

[0061] Specifically, the system operating point can be input into the adversarial agent to obtain the adversarial operating point output by the adversarial agent, and the adversarial operating state can be generated based on the adversarial operating point.

[0062] The countermeasures operation status can include node active power, node reactive power, and node voltage amplitude.

[0063] Step S3: Input the adversarial operation state into the policy agent to generate a control policy that matches the adversarial operation state. The adversarial operation state and the control policy form a control policy sample.

[0064] Specifically, the adversarial operating state can be input into the policy agent to obtain a control policy output by the policy agent that matches the adversarial operating state.

[0065] The opposing operating states and their matching control strategies can be combined to form a control strategy sample.

[0066] As can be seen from the above technical solutions, the control strategy sample generation method provided in this application can obtain a strategy agent and an adversarial agent trained with the generation probability of controlling extreme operating points as the target. The adversarial agent and the strategy agent undergo interactive training. Based on this, a dynamic balance can be maintained between the adversarial agent and the strategy agent, ensuring their stability and matching. Subsequently, the adversarial agent can be used to process the obtained system operating points to generate adversarial operating states. These adversarial operating states are then input into the strategy agent to generate a control strategy matching the adversarial operating states. Since the adversarial agent is trained with the generation probability of controlling extreme operating points as the target, and the adversarial operating states generated by the adversarial agent can be input into the strategy agent, various adversarial operating states, even extreme operating states, can be generated using the adversarial agent. This guides the strategy agent to provide matching control strategies for different operating states, expanding the data dimensions of both the adversarial operating states and the control strategies, ensuring the extreme nature and versatility of the adversarial operating states and their matching control strategies. Based on this, the adversarial operating state and the control strategy form a control strategy sample. The adversarial operating state of the control strategy sample corresponds to the input of the deep reinforcement learning model applied to the formulation of emergency control strategies, and the control strategy of the control strategy sample corresponds to the output of the deep reinforcement learning model. Therefore, the control strategy sample generated in this application can be used to train the deep reinforcement learning model. Since the adversarial operating state and control strategy of this application can correspond to both extreme and normal scenarios, it has strong extremism and versatility. Therefore, using the control strategy sample of this application to train the deep reinforcement learning model can further improve the robustness and reliability of emergency control formulation. It can be seen that this application can generate a variety of control strategy samples that guarantee extremism and versatility to improve the robustness and reliability of emergency control.

[0067] In some embodiments of this application, the process of obtaining the policy agent and adversarial agent after interactive training at the system operating point in step S1 is described in detail, and the steps are as follows:

[0068] S10. Generate the initial adversarial agent and the initial policy agent.

[0069] Specifically, an objective function can be constructed for production control strategies based on operational states, and an initial policy agent can be constructed based on the objective function.

[0070] It can generate pre-trained initial adversarial agents.

[0071] S11. With the goal of adjusting the generation probability of extreme operating points of the initial adversarial agent, the parameters of the initial adversarial agent are adjusted.

[0072] Specifically, by adjusting the parameters of the initial adversarial agent, the intensity of generating extreme operating points of the adversarial agent can be adjusted, and the balance between exploration and utilization of the initial adversarial agent can be dynamically adjusted to avoid the initial policy agent from being too difficult to generate control strategies, thus preventing the policy agent from overfitting.

[0073] S12. Combining zero-sum game theory, the initial policy agent and the initial adversarial agent after parameter adjustment are trained alternately, and the final initial policy agent is used as the policy agent, and the final initial adversarial agent is used as the adversarial agent.

[0074] Specifically, the reward functions for the initial policy agent and the initial adversarial agent can be set by combining zero-sum game theory.

[0075] Based on the reward function of the initial policy agent and the reward function of the initial adversarial agent, the initial policy agent and the parameter-adjusted initial adversarial agent can be trained alternately.

[0076] The final initial policy agent can be used as the policy agent, and the final initial adversarial agent can be used as the adversarial agent.

[0077] As can be seen from the above technical solution, this embodiment provides an optional method for obtaining adversarial agents and policy agents. Through this method, the extreme performance point generation probability of the initial adversarial agent can be adjusted as the target, and the parameters of the initial adversarial agent can be adjusted. Furthermore, by combining zero-sum game theory, the policy agent and adversarial agent are trained alternately. This mechanism not only enhances the adversarial agent's ability to generate high-quality adversarial performance points but also avoids the initial policy agent from overfitting to a fixed adversarial strategy during alternating training, and solves the distribution shift problem, ensuring stability during joint training. This enables the policy agent to learn control strategies not only under normal conditions but also under extreme conditions, thereby improving the policy agent's robustness to extreme operating scenarios.

[0078] In some embodiments of this application, the process of generating the initial adversarial agent in step S10 is described in detail, and the steps are as follows:

[0079] S100 is an intelligent agent constructed by combining the SAC algorithm and a single-step Markov decision process.

[0080] Specifically, the process of an agent generating adversarial running points can be modeled as a single-step Markov decision process. The agent can be constructed by combining the evaluation network, adversarial policy network, and replay buffer network of the soft actor-critic algorithm SAC.

[0081] S101. Input multiple historical running points into the intelligent agent in sequence to obtain the predicted running point corresponding to each historical running point.

[0082] Specifically, multiple historical operating points can be selected from the power grid system and the power system simulator, and then input into the agent in sequence to obtain the predicted operating point corresponding to each historical operating point output by the agent.

[0083] S102. Based on KL divergence, calculate the similarity between each historical running point and its corresponding predicted running point.

[0084] Specifically, a similarity calculation expression based on KL divergence can be obtained, and based on the similarity calculation expression, the similarity between predicted running points corresponding to each historical running point machine can be calculated.

[0085] The similarity calculation expression can be shown below:

[0086]

[0087] Among them, L adv Indicates the point of confrontation; L pot Let P represent the historical running point, and let P be the preset probability distribution.

[0088] S103. Based on each similarity, adjust the parameters of the latest agent until the similarity meets the preset similarity threshold, and use the final agent as the initial adversarial agent.

[0089] Specifically, the parameters of the agent can be adjusted step by step based on each similarity until the latest calculated similarity meets the preset similarity threshold. The resulting agent can then be used as the initial adversarial agent.

[0090] As can be seen from the above technical solution, this embodiment provides an optional method for pre-training an initial adversarial agent. Through this method, supervised learning can be used to guide the adversarial agent to generate adversarial operating points within feasible constraints. Since historical operating points can be obtained from simulators and real power grid systems, the boundary of the feasible operating space of the adversarial agent in this application is determined by historical operating scenarios and selected simulated feasible operating scenarios. The adversarial agent can learn to generate operating points that satisfy the "potential operating space" constraints. In terms of technical implementation, the introduction of KL divergence helps to effectively regulate the generation capability of the adversarial agent during the pre-training stage, ensuring that its output operating scenarios are both challenging and conform to the actual constraints of the power system. Furthermore, by optimizing the generation strategy of the adversarial agent during pre-training, we can better balance the exploration and utilization of the adversarial agent during the joint training stage, thereby improving the robustness of the overall system.

[0091] Furthermore, the adversarial agent includes an evaluation network, an adversarial policy network, and a replay buffer network. The exploration process of the run points can be adjusted using the adaptive temperature parameter of the SAC algorithm, ensuring that the adversarial agent can dynamically adjust its balance between exploration and exploitation at different training stages, thereby more effectively generating challenging adversarial run points. By introducing the dual network structure and target network update mechanism from the SAC algorithm, the robustness and stability of the adversarial agent in complex environments are enhanced. Moreover, the introduction of the adaptive temperature parameter allows the adversarial agent to flexibly respond to policy exploration demands of varying intensities, thereby improving the efficiency and effectiveness of generating adversarial examples.

[0092] In some embodiments of this application, the process of adjusting the parameters of the initial adversarial agent in step S11, with the goal of adjusting the probability of generating extreme operating points, is described in detail as follows:

[0093] S11, based on A greedy algorithm is used to adjust the parameters of the initial adversarial agent with the goal of adjusting the probability of generating extreme running points.

[0094] Specifically, it can be based on Greedy algorithm, set in the initial adversarial agent Parameters are used to adjust the probability of generating extreme running points for the initial adversarial agent.

[0095] As can be seen from the above technical solution, this embodiment provides an optional method for adjusting the parameters of the initial adversarial agent. This method can be combined with... The greedy algorithm adjusts the parameters of the initial adversarial agent, further controlling the probability of generating extreme running points of the initial adversarial agent, enhancing the data dimension of the adversarial agent in generating adversarial running points, and controlling the adversarial agent to generate not only normal running points but also extreme running points.

[0096] In some embodiments of this application, the process of alternately training the initial policy agent and the initial adversarial agent after parameter adjustment in step S12 by combining zero-sum game theory is described in detail as follows:

[0097] S120. Determine the reward functions for the initial adversarial agent and the initial policy agent based on the zero-sum game setting.

[0098] Specifically, the reward function of the initial policy agent can be set, and the reward function of the initial adversarial agent can be set by combining the zero-sum game and the reward function of the initial policy agent.

[0099] S121. Stop adjusting the parameters of the initial adversarial agent and obtain two types of training running states. One type of training running state is a running state randomly selected from the power grid system; the other type of training running state is multiple running states obtained based on the latest processing of the initial adversarial agent.

[0100] Specifically, the parameters of the initial adversarial agent can be fixed, and the initial policy agent can be trained.

[0101] During the training of the initial policy agent, operating states can be randomly selected from the power grid system and power system simulator as training operating states, and historical operating states can be input into the latest initial adversarial agent to obtain training operating states.

[0102] S122. Input each training running state sequentially into the latest initial policy agent to obtain the training control policy corresponding to each training running state.

[0103] Specifically, the training running state can be input into the latest initial policy agent to obtain the training control policy corresponding to the training running state output by the initial policy agent.

[0104] S123. Based on each training running state and its corresponding training control strategy, and combined with the reward function of the initial policy agent, calculate the corresponding training reward value, and adjust the parameters of the latest initial policy agent based on each training reward value until the stopping condition is met.

[0105] Specifically, each training running state and training control strategy can be analyzed, and the training reward value corresponding to each training running state can be calculated by combining the reward function of the initial policy agent.

[0106] The parameters of the latest initial policy agent can be adjusted based on each training reward value until the number of iterations exceeds the preset iteration threshold or the initial policy agent converges.

[0107] S124. Stop adjusting the parameters of the initial policy agent and obtain multiple target running points.

[0108] Specifically, the parameters of the initial policy agent can be fixed, and the initial adversarial agent can be trained.

[0109] During the training of the initial adversarial agent, multiple target operating points can be obtained from the power grid system and power system simulator.

[0110] S125. Input each target running point into the latest initial adversarial agent in sequence to generate the target running state corresponding to each target running point.

[0111] Specifically, the target running point can be input into the latest initial adversarial agent to obtain multiple running points output by the initial adversarial agent, and each running point can be simulated to obtain the target running state corresponding to each running point.

[0112] S126. Input each target's running state into the latest initial policy agent in sequence to obtain the target control policy corresponding to each target's running state.

[0113] Specifically, the target operating state can be input into the latest fixed initial policy agent to obtain the target control policy output by the initial policy agent.

[0114] S127. Based on the running state of each target and its corresponding target control strategy, and combined with the reward function of the initial adversarial agent, calculate the corresponding target reward value, and adjust the parameters of the initial adversarial agent based on each target reward value until the stopping condition is met; return to step S121 until both the initial adversarial agent and the initial policy agent converge.

[0115] Specifically, the target reward value corresponding to each target running state can be calculated based on each target running state and its corresponding target control strategy, combined with the reward function of the initial adversarial agent.

[0116] Based on each target reward value and a preset update step size, the parameters of the initial adversarial agent are adjusted until the number of iterations exceeds a preset iteration threshold or the initial adversarial agent converges.

[0117] You can return to steps S121-S127 until both the initial adversarial agent and the initial policy agent converge.

[0118] As can be seen from the above technical solution, this embodiment provides an optional method that combines zero-sum game theory to alternately train the initial policy agent and the initial adversarial agent after parameter adjustment. The above method can better train the initial policy agent and the initial adversarial agent and ensure the generation of edge samples.

[0119] In some embodiments of this application, the process of step S120, determining the reward function of the initial adversarial agent and the initial policy agent based on the zero-sum game setting, is described in detail as follows:

[0120] S1200. Set the reward function of the initial policy agent, and set the reward function of the initial adversarial agent to be the negative of the lower limit of the reward of the initial policy agent in the sample space.

[0121] Specifically, a reward function can be set for an initial policy agent, which consists of the running state and its corresponding control policy.

[0122] The reward function for the initial policy agent can be:

[0123]

[0124] Combining zero-sum game theory, the reward function of the initial adversarial agent can be set as the negative of the lower bound of the reward of the initial policy agent in the sample space;

[0125] The reward function for the initial adversarial agent can be:

[0126]

[0127] in, It can be the reward value for the initial policy agent; The running state can be input to the initial policy agent; D can be the corresponding control strategy; D can be the training set consisting of the running states of the initial policy agent. , It can be a preset coefficient; It can be the node voltage when the power grid system reaches a quasi-steady state under the current operating state s and control strategy a of the power grid system; It can be a cost function; It can be the reward value for the initial adversarial agent.

[0128] As can be seen from the above technical solution, this embodiment provides an optional method for determining the initial adversarial agent and the initial policy agent. The performance of the initial adversarial agent and the initial policy agent can be further improved through the above method.

[0129] In some embodiments of this application, the process of step S2, which involves using the adversarial agent to process the system operating point and generate an adversarial operating state, is described in detail below:

[0130] S20. Input the system running point into the adversarial agent to obtain the adversarial running point output by the adversarial agent.

[0131] Specifically, the system operating point can be the injected power vector in the injected power space.

[0132] Adversarial agents can be used to make extreme adjustments to the injected power vector to form adversarial operating points.

[0133] S21. Simulate the adversarial operation point to obtain the adversarial operation state.

[0134] Specifically, a power system simulator can be used to simulate countermeasures operation points and collect countermeasures operation status data.

[0135] As can be seen from the above technical solution, this embodiment provides an optional method for obtaining the adversarial operating state. Through the above method, the injected power vector can be adjusted to form an adversarial operating point and control the probability of the extreme operating point of this application.

[0136] Next, we will combine Figure 2 The control strategy sample generation apparatus provided in this application will be described in detail. The control strategy sample generation apparatus described below can be compared with the control strategy sample generation method described above.

[0137] See Figure 2 It can be observed that the control strategy sample generation device may include:

[0138] The acquisition module 10 is used to acquire the system running point, as well as the policy agent and adversarial agent trained interactively, wherein the adversarial agent is trained with the goal of controlling the generation probability of extreme running points;

[0139] Processing module 20 is used to process the system running point using the adversarial agent to generate an adversarial running state;

[0140] The generation module 30 is used to input the adversarial operation state into the policy agent and generate a control policy that matches the adversarial operation state. The adversarial operation state and the control policy form a control policy sample.

[0141] Furthermore, the acquisition module may include:

[0142] The agent generation unit is used to generate the initial adversarial agent and the initial policy agent;

[0143] The parameter adjustment unit is used to adjust the parameters of the initial adversarial agent with the goal of adjusting the probability of generating extreme operating points of the initial adversarial agent;

[0144] The agent training unit is used to combine zero-sum game theory to alternately train the initial policy agent and the initial adversarial agent after parameter adjustment, and to use the final initial policy agent as the policy agent and the final initial adversarial agent as the adversarial agent.

[0145] Furthermore, the agent generation unit may include:

[0146] The first intelligent agent generation component is used to generate intelligent agents that combine the SAC algorithm and the single-step Markov decision process.

[0147] The second agent generation component is used to sequentially input multiple historical running points into the agent to obtain the predicted running point corresponding to each historical running point;

[0148] The third intelligent agent generation component is used to calculate the similarity between each historical running point and its corresponding predicted running point based on KL divergence.

[0149] The fourth agent generation component is used to adjust the parameters of the latest agent based on each similarity until the similarity meets the preset similarity threshold, and the final agent is used as the initial adversarial agent.

[0150] Furthermore, the parameter adjustment unit may include:

[0151] The greedy algorithm utilizes a component to adjust the parameters of the initial adversarial agent based on the ε-greedy algorithm, with the goal of adjusting the probability of generating extreme running points.

[0152] Furthermore, the agent training unit may include:

[0153] The reward function determination component is used to determine the reward functions of the initial adversarial agent and the initial policy agent based on the zero-sum game setting.

[0154] The training runtime state acquisition component is used to stop parameter adjustments to the initial adversarial agent and acquire two types of training runtime states. One type of training runtime state is a runtime state randomly selected from the power grid system; the other type of training runtime state is multiple runtime states obtained based on the latest processing of the initial adversarial agent.

[0155] The training control policy acquisition component is used to sequentially input each training running state into the latest initial policy agent to obtain the training control policy corresponding to each training running state.

[0156] The training reward value calculation component is used to calculate the corresponding training reward value based on each training running state and its corresponding training control policy, combined with the reward function of the initial policy agent, and to adjust the parameters of the latest initial policy agent based on each training reward value until the stopping condition is met.

[0157] The target run point acquisition component is used to stop parameter adjustments to the initial policy agent and acquire multiple target run points;

[0158] The target running state generation component is used to sequentially input each target running point into the latest initial adversarial agent and generate the target running state corresponding to each target running point.

[0159] The target control strategy acquisition component is used to sequentially input the running state of each target into the latest initial strategy agent to obtain the target control strategy corresponding to each target running state.

[0160] The target reward value calculation component is used to calculate the corresponding target reward value based on the running state of each target and its corresponding target control policy, combined with the reward function of the initial adversarial agent, and to adjust the parameters of the initial adversarial agent based on each target reward value until the stopping condition is met; it then returns to call the training running state acquisition component until both the initial adversarial agent and the initial policy agent converge.

[0161] Furthermore, the reward function determination component may include:

[0162] The first reward function determination sub-component is used to set the reward function of the initial policy agent and to set the reward function of the initial adversarial agent to be the negative of the lower limit of the reward of the initial policy agent in the sample space.

[0163] Furthermore, the processing module may include:

[0164] The first processing unit is used to input the system operating point to the adversarial agent and obtain the adversarial operating point output by the adversarial agent;

[0165] The second processing unit is used to simulate the adversarial operation point to obtain the adversarial operation state.

[0166] The control strategy sample generation device provided in this application embodiment can be applied to control strategy sample generation devices, such as PC terminals, cloud platforms, servers, and server clusters. Optionally, Figure 3 The hardware structure block diagram of the control strategy sample generation device is shown below. Figure 3 The hardware structure of the control strategy sample generation device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0167] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0168] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0169] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0170] The memory stores a program, which the processor can call. The program is used for:

[0171] The system operating point is obtained, as well as the policy agent and adversarial agent trained interactively, wherein the adversarial agent is trained with the goal of controlling the generation probability of extreme operating points;

[0172] The adversarial agent is used to process the system's operating points to generate an adversarial operating state;

[0173] The adversarial operating state is input into the policy agent to generate a control policy that matches the adversarial operating state. The adversarial operating state and the control policy form a control policy sample.

[0174] Optionally, the refined and extended functions of the program can be referred to the above description.

[0175] This application embodiment also provides a readable storage medium that can store a program suitable for execution by a processor, the program being used for:

[0176] The system operating point is obtained, as well as the policy agent and adversarial agent trained interactively, wherein the adversarial agent is trained with the goal of controlling the generation probability of extreme operating points;

[0177] The adversarial agent is used to process the system's operating points to generate an adversarial operating state;

[0178] The adversarial operating state is input into the policy agent to generate a control policy that matches the adversarial operating state. The adversarial operating state and the control policy form a control policy sample.

[0179] Optionally, the refined and extended functions of the program can be referred to the above description.

[0180] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0181] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0182] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. The various embodiments of this application can be combined with each other. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating control strategy samples, characterized in that, include: Obtain the system run point; Generate initial adversarial agents and initial policy agents; With the goal of adjusting the generation probability of extreme operating points of the initial adversarial agent, the parameters of the initial adversarial agent are adjusted; By combining zero-sum game theory, the initial policy agent and the initial adversarial agent after parameter adjustment are trained alternately. The final initial policy agent is used as the policy agent, and the final initial adversarial agent is used as the adversarial agent. The adversarial agent is used to process the system's operating points to generate an adversarial operating state; The adversarial operating state is input into the policy agent to generate a control policy that matches the adversarial operating state. The adversarial operating state and the control policy form a control policy sample. During alternating training, the reward function of the initial policy agent is: ； The reward function for the initial adversarial agent is: ； in, The reward value for the initial policy agent; The running state is input to the initial policy agent; D represents the corresponding control strategy; D is the training set consisting of the running states of the initial policy agent. , These are preset coefficients; Let be the node voltage when the power grid system reaches quasi-steady state under the current operating state s and control strategy a; It is the cost function; This represents the initial reward value for the adversary agent.

2. The control strategy sample generation method according to claim 1, characterized in that, Generate the initial adversarial agent, including: Generate an intelligent agent constructed by combining the SAC algorithm and a single-step Markov decision process; Multiple historical running points are sequentially input into the intelligent agent to obtain the predicted running point corresponding to each historical running point; Based on KL divergence, the similarity between each historical running point and its corresponding predicted running point is calculated; Based on each similarity score, the parameters of the latest agent are adjusted until the similarity score meets the preset similarity threshold. The resulting agent is then used as the initial adversarial agent.

3. The control strategy sample generation method according to claim 1, characterized in that, The step of adjusting the parameters of the initial adversarial agent with the goal of adjusting the probability of generating extreme operating points includes: Based on the ε-greedy algorithm, the parameters of the initial adversarial agent are adjusted with the goal of adjusting the probability of generating extreme running points.

4. The control strategy sample generation method according to claim 1, characterized in that, The method of combining zero-sum game theory with alternating training of the initial policy agent and the initial adversarial agent after parameter adjustment includes: Determine the reward functions for the initial adversarial agent and the initial policy agent based on a zero-sum game setting; Stop adjusting the parameters of the initial adversarial agent and obtain two types of training running states. One type of training running state is a running state randomly selected from the power grid system; the other type of training running state is multiple running states obtained based on the latest processing of the initial adversarial agent. Each training running state is sequentially input into the latest initial policy agent to obtain the training control policy corresponding to each training running state; Based on each training running state and its corresponding training control strategy, combined with the reward function of the initial policy agent, the corresponding training reward value is calculated, and the parameters of the latest initial policy agent are adjusted based on each training reward value until the stopping condition is met. Stop adjusting the parameters of the initial policy agent and obtain multiple target running points; Each target running point is sequentially input into the latest initial adversarial agent to generate the target running state corresponding to each target running point; Each target's running state is sequentially input into the latest initial policy agent to obtain the target control policy corresponding to each target's running state; Based on the running state of each target and its corresponding target control strategy, combined with the reward function of the initial adversarial agent, the corresponding target reward value is calculated, and the parameters of the initial adversarial agent are adjusted based on each target reward value until the stopping condition is met. Return to the previous step to stop adjusting the parameters of the initial adversarial agent and obtaining the two training states, until both the initial adversarial agent and the initial policy agent converge.

5. The control strategy sample generation method according to claim 4, characterized in that, The determination of the reward functions for the initial adversarial agent and the initial policy agent based on the zero-sum game setting includes: Set the reward function for the initial policy agent, and set the reward function for the initial adversarial agent to be the negative of the lower bound of the initial policy agent's reward in the sample space.

6. The control strategy sample generation method according to claim 1, characterized in that, The step of using the adversarial agent to process the system's operating points and generate an adversarial operating state includes: The system operating point is input into the adversarial agent to obtain the adversarial operating point output by the adversarial agent; The adversarial operation point is simulated to obtain the adversarial operation state.

7. A control strategy sample generation device, characterized in that, include: The acquisition module is used to acquire the system runtime point; Generate initial adversarial agents and initial policy agents; With the goal of adjusting the probability of generating extreme running points of the initial adversarial agent, the parameters of the initial adversarial agent are adjusted; combined with zero-sum game, the initial policy agent and the initial adversarial agent after parameter adjustment are trained alternately, and the final initial policy agent is used as the policy agent, and the final initial adversarial agent is used as the adversarial agent. During alternating training, the reward function of the initial policy agent is: ； The reward function for the initial adversarial agent is: ； in, The reward value for the initial policy agent; The running state is input to the initial policy agent; D represents the corresponding control strategy; D is the training set consisting of the running states of the initial policy agent. , These are preset coefficients; Let be the node voltage when the power grid system reaches quasi-steady state under the current operating state s and control strategy a; It is the cost function; The initial reward value for the adversarial agent; The processing module is used to process the system's operating points using the adversarial agent to generate an adversarial operating state; The generation module is used to input the adversarial operation state into the policy agent and generate a control policy that matches the adversarial operation state. The adversarial operation state and the control policy form a control policy sample.

8. A control strategy sample generation device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the control strategy sample generation method as described in any one of claims 1-6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the control strategy sample generation method as described in any one of claims 1-6.

Images