Control strategy sample generation method, device and equipment and readable storage medium

Abstract

The invention discloses a control strategy sample generation method, device and equipment and a readable storage medium, and the method can obtain a system operation point, and a strategy agent and an adversarial agent which are subjected to interactive training, and the adversarial agent is obtained through training by taking the generation probability of a control extreme operation point as a target; processing the system operation point by using a confrontation agent to generate a confrontation operation state; and inputting the confrontation operation state into a strategy agent, generating a control strategy matched with the confrontation operation state, and forming a control strategy sample by the confrontation operation state and the control strategy. On the basis, various confrontation operation states and even extreme operation states can be generated by utilizing the confrontation intelligent agent, and the strategy intelligent agent is guided to provide matched control strategies for different operation states, so that the extreme property and the universality of the confrontation operation states and the matched control strategies are ensured. Therefore, the method can generate various control strategy samples which guarantee the extreme performance and the universality.

Description

Technical Field

[0001] The present application relates to the field of power grid technology, and more specifically, to a control strategy sample generation method, device, equipment and readable storage medium. Background Art

[0002] As the penetration rate of renewable energy continues to increase, the randomness and uncertainty of the power system have increased significantly, bringing huge challenges to the formulation of traditional emergency control strategies. In recent years, in order to solve the above problems, attempts have been made to apply deep reinforcement learning models to the formulation of emergency control strategies for power systems. Deep reinforcement learning models are used to generate corresponding emergency control strategies based on the operating status of the power grid system to maintain the safe and stable operation of the power grid system.

[0003] However, when training deep reinforcement learning models, it is necessary to manually select samples at the edge of the operating space based on artificial experience or sampling methods such as Monte Carlo. However, this method relies heavily on domain experience. Once experience is insufficient, the selected samples are prone to lack of extremism and poor versatility, resulting in the deep reinforcement learning model still having deficiencies when dealing with safety constraints and high-risk scenarios. On this basis, how to generate samples for training deep reinforcement learning models has become the focus of attention of people in this field. Summary of the invention

[0004] In view of this, the present application provides a control strategy sample generation method, apparatus, device and readable storage medium for generating training samples of a deep reinforcement learning model.

[0005] In order to achieve the above objectives, the proposed solution is as follows:

[0006] A control strategy sample generation method, comprising:

[0007] Obtaining a system operating point, as well as a strategy agent and an adversarial agent that have undergone interactive training, wherein the adversarial agent is trained to control the generation probability of an extreme operating point;

[0008] Using the confrontation agent, processing the system operation point to generate a confrontation operation state;

[0009] The confrontation running state is input into the strategy agent, and a control strategy matching the confrontation running state is generated. The confrontation running state and the control strategy form a control strategy sample.

[0010] Optionally, obtain interactively trained policy agents and adversarial agents, including:

[0011] Generate initial adversarial agents and initial strategic agents;

[0012] Adjusting parameters of the initial adversarial agent with the goal of adjusting the probability of generating an extreme operating point of the initial adversarial agent;

[0013] In combination with zero-sum game, the initial strategy agent and the initial confrontation agent with adjusted parameters are trained alternately, and the final initial strategy agent is used as the strategy agent, and the final initial confrontation agent is used as the confrontation agent.

[0014] Optionally, generate an initial adversarial agent, including:

[0015] Generate an agent that combines the SAC algorithm and the single-step Markov decision process;

[0016] Inputting a plurality of historical operating points into the intelligent agent in sequence to obtain a predicted operating point corresponding to each historical operating point;

[0017] Based on KL divergence, the similarity between each historical operating point and its corresponding predicted operating point is calculated;

[0018] Based on each similarity, the parameters of the latest agent are adjusted until the similarity meets the preset similarity threshold, and the final agent is used as the initial adversarial agent.

[0019] Optionally, the step of adjusting parameters of the initial adversarial agent with the goal of controlling the probability of generating an extreme operating point of the initial adversarial agent comprises:

[0020] Based on the ε-greedy algorithm, the parameters of the initial adversarial agent are adjusted with the goal of adjusting the probability of generating the extreme operating point of the initial adversarial agent.

[0021] Optionally, combining the zero-sum game to alternately train the initial strategy agent and the initial adversarial agent with adjusted parameters includes:

[0022] Determine the reward function of the initial adversarial agent and the initial strategy agent based on the zero-sum game setting;

[0023] Stop adjusting the parameters of the initial adversarial agent and obtain two types of training operating states, wherein one type of training operating state is an operating state randomly selected from the power grid system; the other type of training operating state is a plurality of operating states obtained based on the latest initial adversarial agent processing;

[0024] Input each training running state into the latest initial strategy agent in turn to obtain the training control strategy corresponding to each training running state;

[0025] Based on each training running state and its corresponding training control strategy, combined with the reward function of the initial strategy agent, the corresponding training reward value is calculated, and the parameters of the latest initial strategy agent are adjusted based on each training reward value until the stopping condition is met;

[0026] Stop adjusting parameters of the initial strategy agent and obtain multiple target operating points;

[0027] Input each target operating point into the latest initial adversarial agent in turn to generate the target operating state corresponding to each target operating point;

[0028] Input each target operating state into the latest initial strategy agent in turn to obtain the target control strategy corresponding to each target operating state;

[0029] Based on each target operating state and its corresponding target control strategy, combined with the reward function of the initial adversarial agent, the corresponding target reward value is calculated, and the parameters of the initial adversarial agent are adjusted based on each target reward value until the stopping condition is met;

[0030] Return to the step of stopping parameter adjustment of the initial adversarial agent and obtaining two types of training running status until both the initial adversarial agent and the initial policy agent converge.

[0031] Optionally, determining the reward function of the initial confrontation agent and the initial strategy agent based on the zero-sum game setting includes:

[0032] Set the reward function of the initial strategy agent, and set the reward function of the initial adversarial agent to the opposite of the lower limit of the reward of the initial strategy agent in the sample space.

[0033] Optionally, the using the confrontation agent to process the system operation point to generate a confrontation operation state includes:

[0034] Inputting the system operating point into the antagonistic agent to obtain the antagonistic operating point output by the antagonistic agent;

[0035] The antagonistic operation point is simulated to obtain an antagonistic operation state.

[0036] A control strategy sample generating device, comprising:

[0037] An acquisition module, used to acquire the system operating point, as well as the interactively trained strategy agent and the adversarial agent, wherein the adversarial agent is trained to control the generation probability of extreme operating points;

[0038] A processing module, used to process the system operation point using the confrontation agent to generate a confrontation operation state;

[0039] A generation module is used to input the confrontation running state into the strategy agent, generate a control strategy matching the confrontation running state, and the confrontation running state and the control strategy form a control strategy sample.

[0040] A control strategy sample generating device comprises a memory and a processor;

[0041] The memory is used to store programs;

[0042] The processor is used to execute the program to implement the various steps of the above-mentioned control strategy sample generation method.

[0043] A readable storage medium stores a computer program, and when the computer program is executed by a processor, each step of the control strategy sample generation method is implemented.

[0044] It can be seen from the above technical solutions that the control strategy sample generation method provided by the present application can obtain a strategy agent and an adversarial agent trained with the goal of controlling the generation probability of extreme operating points, and the adversarial agent and the strategy agent are interactively trained; based on this, the dynamic balance between the adversarial agent and the strategy agent can be maintained, and the stability and matching of the adversarial agent and the strategy agent can be guaranteed; then, the adversarial agent can be used to process the acquired system operating points to generate an adversarial operating state; and the adversarial operating state is input into the strategy agent to generate a control strategy matching the adversarial operating state. Based on this, since the adversarial agent is trained with the goal of controlling the generation probability of extreme operating points, and the adversarial operating state generated by the adversarial agent can be input into the strategy agent, the adversarial agent can be used to generate a variety of adversarial operating states or even extreme operating states, and guide the strategy agent to provide matching control strategies for different operating states, expand the data dimension of the adversarial operating state, expand the data dimension of the control strategy, and ensure the extremeness and versatility of the adversarial operating state and its matching control strategy. On this basis, the adversarial operating state and the control strategy form a control strategy sample, the adversarial operating state of the control strategy sample corresponds to the input of the deep reinforcement learning model used for emergency control strategy formulation, and the control strategy of the control strategy sample corresponds to the output of the deep reinforcement learning model. Therefore, the control strategy samples generated by this application can be used to train the deep reinforcement learning model; since the adversarial operating state and control strategy of this application can correspond to extreme scenarios and normal scenarios, the extremeness and versatility are relatively strong; therefore, the deep reinforcement learning model is trained using the control strategy samples of this application, which can further improve the robustness and reliability of emergency control formulation. It can be seen that this application can generate a variety of control strategy samples that guarantee extremeness and versatility to improve the robustness and reliability of emergency control. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 A flow chart of a control strategy sample generation method disclosed in an embodiment of the present application;

[0047] Figure 2 A structural block diagram of a control strategy sample generating device disclosed in an embodiment of the present application;

[0048] Figure 3This is a hardware structure block diagram of a control strategy sample generation device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The present application embodiment provides a control strategy sample generation method, which can be applied to various power grid systems or training systems, and can also be applied to various computer terminals or smart terminals. The execution subject can be a processor or server of the computer terminal or smart terminal. The method flow chart of the control strategy sample generation method is as follows: Figure 1 As shown, specifically including:

[0051] Step S1, obtain the system operating point, as well as the interactively trained strategy agent and adversarial agent.

[0052] Specifically, considering the complexity of the power system, it is impractical to fully explore all possible operating points. Therefore, identifying and generating extreme scenarios and training the agent under these extreme scenarios can enhance the robustness of the existing strategy and thus improve the performance of the agent.

[0053] Based on this, the adversarial agent can adjust the generation probability of extreme operating points for target training.

[0054] The adversarial agent can generate adversarial operating points based on the system operating points.

[0055] Since there is a certain probability that the adversarial agent will generate an extreme operating point, there is a certain probability that the adversarial operating point will be an extreme operating point.

[0056] The extreme operating points can correspond to various extreme scenarios of the power grid system. Therefore, adversarial agents can be used to achieve the purpose of exploring extreme scenarios.

[0057] The policy agent can be used to reduce violations in the control policy samples, while the adversarial agent can be used to generate extreme and challenging operating points to increase the data dimensionality of the policy agent.

[0058] The strategic agent and the adversarial agent can be trained alternately by the semantic role labeling algorithm GAN-SRL based on the generative adversarial network.

[0059] The latest system operating points can be collected from the power grid system at regular intervals.

[0060] Step S2: using the adversarial agent to process the system operation point and generate an adversarial operation state.

[0061] Specifically, the system operating point can be input into the adversarial agent to obtain the adversarial operating point output by the adversarial agent, and based on the adversarial operating point, the adversarial operating state is generated.

[0062] The confrontation operation status may include node active power, node reactive power, and node voltage amplitude.

[0063] Step S3: input the confrontation running state into the strategy agent, generate a control strategy matching the confrontation running state, and the confrontation running state and the control strategy form a control strategy sample.

[0064] Specifically, the confrontation running state can be input into the strategy agent to obtain a control strategy output by the strategy agent that matches the confrontation running state.

[0065] The adversarial operating states and their matching control strategies can be combined to form a control strategy sample.

[0066] It can be seen from the above technical solutions that the control strategy sample generation method provided by the embodiment of the present application can obtain a strategy agent and an adversarial agent trained with the goal of controlling the generation probability of extreme operating points, and the adversarial agent and the strategy agent are interactively trained; based on this, the dynamic balance between the adversarial agent and the strategy agent can be maintained, and the stability and matching of the adversarial agent and the strategy agent can be guaranteed; then, the adversarial agent can be used to process the acquired system operating points to generate an adversarial operating state; and the adversarial operating state is input into the strategy agent to generate a control strategy matching the adversarial operating state. Based on this, since the adversarial agent is trained with the goal of controlling the generation probability of extreme operating points, and the adversarial operating state generated by the adversarial agent can be input into the strategy agent, the adversarial agent can be used to generate a variety of adversarial operating states or even extreme operating states, and guide the strategy agent to provide matching control strategies for different operating states, expand the data dimension of the adversarial operating state, expand the data dimension of the control strategy, and ensure the extremeness and versatility of the adversarial operating state and its matching control strategy. On this basis, the adversarial operating state and the control strategy form a control strategy sample, the adversarial operating state of the control strategy sample corresponds to the input of the deep reinforcement learning model used for emergency control strategy formulation, and the control strategy of the control strategy sample corresponds to the output of the deep reinforcement learning model. Therefore, the control strategy samples generated by this application can be used to train the deep reinforcement learning model; since the adversarial operating state and control strategy of this application can correspond to extreme scenarios and normal scenarios, the extremeness and versatility are relatively strong; therefore, the deep reinforcement learning model is trained using the control strategy samples of this application, which can further improve the robustness and reliability of emergency control formulation. It can be seen that this application can generate a variety of control strategy samples that guarantee extremeness and versatility to improve the robustness and reliability of emergency control.

[0067] In some embodiments of the present application, the process of obtaining the interactively trained strategy agent and the confrontation agent at the system operation point in step S1 is described in detail, and the steps are as follows:

[0068] S10. Generate an initial adversarial agent and an initial strategy agent.

[0069] Specifically, an objective function for producing a control strategy based on the operating status may be constructed, and an initial strategy agent may be constructed based on the objective function.

[0070] A pre-trained initial adversarial agent can be generated.

[0071] S11, adjusting parameters of the initial adversarial agent with the goal of adjusting the probability of generating extreme operating points of the initial adversarial agent.

[0072] Specifically, by adjusting the parameters of the initial adversarial agent, adjusting the intensity of the generation of extreme operating points of the adversarial agent, and dynamically adjusting the balance between the exploration and utilization of the initial adversarial agent, it is possible to avoid the initial strategy agent's control strategy being too difficult to generate, resulting in overfitting of the strategy agent.

[0073] S12. In combination with the zero-sum game, the initial strategy agent and the initial confrontation agent with adjusted parameters are alternately trained, and the final initial strategy agent is used as the strategy agent, and the final initial confrontation agent is used as the confrontation agent.

[0074] Specifically, we can combine the zero-sum game to set the reward function of the initial strategy agent and the initial adversarial agent;

[0075] Based on the reward function of the initial strategy agent and the reward function of the initial adversarial agent, the initial strategy agent and the initial adversarial agent with adjusted parameters may be alternately trained;

[0076] The final obtained initial strategy agent can be used as the strategy agent, and the final obtained initial confrontation agent can be used as the confrontation agent.

[0077] It can be seen from the above technical solution that this embodiment provides an optional way to obtain adversarial agents and strategic agents. Through the above method, the extreme operating point generation probability of the initial adversarial agent can be adjusted as the goal, the parameters of the initial adversarial agent can be adjusted, and the strategic agent and the adversarial agent can be alternately trained in combination with zero-sum game; this mechanism not only enhances the ability of the adversarial agent to generate high-quality adversarial operating points, but also avoids the initial strategic agent from overfitting to the fixed adversarial strategy during alternating training, and solves the distribution transfer problem to ensure stability during joint training. This allows the strategic agent to learn not only the control strategy under normal conditions, but also the control strategy under extreme conditions, thereby improving the robustness of the strategic agent to extreme operating scenarios.

[0078] In some embodiments of the present application, the process of generating an initial adversarial agent in step S10 is described in detail, and the steps are as follows:

[0079] S100. Generate an intelligent agent constructed by combining the SAC algorithm and the single-step Markov decision process.

[0080] Specifically, the process of the agent generating adversarial running points can be modeled as a single-step Markov decision process, and the evaluation network, adversarial strategy network and replay cache network of the soft actor-critic algorithm SAC can be combined to construct the agent.

[0081] S101. Input multiple historical operating points into the intelligent agent in sequence to obtain a predicted operating point corresponding to each historical operating point.

[0082] Specifically, multiple historical operating points can be selected from the power grid system and the power system simulator, and input into the intelligent agent in sequence to obtain the predicted operating point corresponding to each historical operating point output by the intelligent agent.

[0083] S102: Calculate the similarity between each historical operating point and its corresponding predicted operating point based on KL divergence.

[0084] Specifically, a similarity calculation expression constructed based on KL divergence may be obtained, and based on the similarity calculation expression, the similarity between the predicted operating points corresponding to each historical operating point machine may be calculated.

[0085] The similarity calculation expression can be as follows:

[0086]

[0087] Among them, L adv Indicates the confrontation operation point; L pot represents the historical operating point, and P is the preset probability distribution.

[0088] S103, based on each similarity, adjusting the parameters of the latest agent until the similarity meets the preset similarity threshold, and using the final agent as the initial adversarial agent.

[0089] Specifically, the parameters of the agent can be gradually adjusted based on each similarity in turn until the latest calculated similarity meets the preset similarity threshold, and the final agent can be used as the initial adversarial agent.

[0090] It can be seen from the above technical solution that this embodiment provides an optional way to pre-train the initial adversarial agent. Through the above method, supervised learning can be used to guide the adversarial agent to generate adversarial operating points within the feasible constraints. Since the historical operating points can be obtained from the simulator and the real power grid system, the boundary part of the feasible operating space of the adversarial agent of this application is determined by the historical operating scenarios and the selected simulated feasible operating scenarios. The adversarial agent can learn to generate operating points that meet the constraints of the "potential operating space". In terms of technical implementation, the introduction of KL divergence helps to effectively regulate the generation ability of the adversarial agent in the pre-training stage, ensuring that the operating scenarios it outputs are both challenging and meet the actual constraints of the power system. In addition, by optimizing the generation strategy of the adversarial agent in pre-training, we can better balance the exploration and utilization of the adversarial agent in the joint training stage, thereby improving the robustness of the overall system.

[0091] In addition, the adversarial agent includes an evaluation network, an adversarial strategy network, and a replay cache network. The adaptive temperature parameters of the SAC algorithm can be used to adjust the exploration process of the operating point, ensuring that the adversarial agent can dynamically adjust its balance between exploration and utilization at different training stages, thereby more effectively generating challenging adversarial operating points. By introducing the dual network structure and target network update mechanism in the SAC algorithm, the robustness and stability of the adversarial agent in the face of complex environments are enhanced. In addition, the introduction of the adaptive temperature parameter enables the adversarial agent to flexibly respond to different intensities of strategy exploration needs, thereby improving the efficiency and effectiveness of generating adversarial samples.

[0092] In some embodiments of the present application, step S11, a process of adjusting parameters of the initial adversarial agent with the goal of adjusting the extreme operating point generation probability of the initial adversarial agent, is described in detail, and the steps are as follows:

[0093] S11, based on The greedy algorithm adjusts the parameters of the initial adversarial agent with the goal of adjusting the probability of generating the extreme operating point of the initial adversarial agent.

[0094] Specifically, it can be based on Greedy algorithm, set in the initial adversarial agent Parameters are used to adjust the probability of generating extreme operating points of the initial adversarial agent.

[0095] It can be seen from the above technical solution that this embodiment provides an optional method for adjusting the parameters of the initial confrontation agent. The greedy algorithm adjusts the parameters of the initial adversarial agent, further regulates the probability of generating extreme operating points of the initial adversarial agent, enhances the data dimension of the adversarial agent to generate adversarial operating points, and controls the adversarial agent to generate not only normal operating points but also extreme operating points.

[0096] In some embodiments of the present application, the process of alternately training the initial strategy agent and the initial adversarial agent with adjusted parameters in step S12 in combination with the zero-sum game is described in detail, and the steps are as follows:

[0097] S120, determining the reward function of the initial adversarial agent and the initial strategy agent based on the zero-sum game setting.

[0098] Specifically, the reward function of the initial strategy agent can be set, and the reward function of the initial adversarial agent can be set by combining the zero-sum game and the reward function of the initial strategy agent.

[0099] S121. Stop adjusting parameters of the initial adversarial agent, and obtain two types of training operating states, wherein one type of training operating state is an operating state randomly selected from the power grid system; the other type of training operating state is a plurality of operating states obtained based on the latest initial adversarial agent processing.

[0100] Specifically, the parameters of the initial adversarial agent can be fixed, and the initial policy agent can be trained.

[0101] In the process of training the initial strategy agent, the operating state can be randomly selected from the power grid system and the power system simulator as the training operating state, and the historical operating state can be input into the latest initial adversarial agent to obtain the training operating state.

[0102] S122. Input each training operation state into the latest initial strategy agent in turn to obtain the training control strategy corresponding to each training operation state.

[0103] Specifically, the training running state can be input into the latest initial strategy agent to obtain the training control strategy corresponding to the training running state output by the initial strategy agent.

[0104] S123. Based on each training operation state and its corresponding training control strategy, combined with the reward function of the initial strategy agent, calculate the corresponding training reward value, and adjust the parameters of the latest initial strategy agent based on each training reward value until the stopping condition is met.

[0105] Specifically, each training running state and training control strategy can be analyzed, and the training reward value corresponding to each training running state can be calculated in combination with the reward function of the initial strategy agent;

[0106] The parameters of the latest initial strategy agent can be adjusted based on each training reward value until the number of iterations exceeds a preset iteration threshold or the initial strategy agent converges.

[0107] S124. Stop adjusting parameters of the initial strategy agent and obtain multiple target operating points.

[0108] Specifically, the parameters of the initial strategy agent can be fixed, and the initial adversarial agent can be trained.

[0109] During the training of the initial adversarial agent, multiple target operating points can be obtained from the power grid system and the power system simulator.

[0110] S125. Input each target operating point into the latest initial adversarial agent in turn to generate a target operating state corresponding to each target operating point.

[0111] Specifically, the target operating point can be input into the latest initial adversarial agent to obtain multiple operating points output by the initial adversarial agent, and each operating point is simulated to obtain the target operating state corresponding to each operating point.

[0112] S126. Input each target operating state into the latest initial strategy agent in turn to obtain the target control strategy corresponding to each target operating state.

[0113] Specifically, the target operating state can be input into the latest fixed initial strategy agent to obtain the target control strategy output by the initial strategy agent.

[0114] S127. Based on each target operating state and its corresponding target control strategy, combined with the reward function of the initial adversarial agent, calculate the corresponding target reward value, and adjust the parameters of the initial adversarial agent based on each target reward value until the stopping condition is met; return to execute step S121 until the initial adversarial agent and the initial strategy agent converge.

[0115] Specifically, based on each target operating state and its corresponding target control strategy, combined with the reward function of the initial adversarial agent, the target reward value corresponding to each target operating state can be calculated;

[0116] Based on each target reward value and the preset update step size, the parameters of the initial adversarial agent are adjusted until the number of iterations exceeds the preset iteration threshold or the initial adversarial agent converges.

[0117] The process may return to executing steps S121-S127 until both the initial adversarial agent and the initial strategic agent converge.

[0118] It can be seen from the above technical solution that this embodiment provides an optional method for alternately training the initial strategy agent and the initial adversarial agent with adjusted parameters in combination with zero-sum game. The above method can better train the initial strategy agent and the initial adversarial agent to ensure the generation of marginal samples.

[0119] In some embodiments of the present application, the process of determining the reward function of the initial confrontation agent and the initial strategy agent based on the zero-sum game setting in step S120 is described in detail, and the steps are as follows:

[0120] S1200, setting the reward function of the initial strategy agent, and setting the reward function of the initial adversarial agent to the opposite of the lower limit of the reward of the initial strategy agent in the sample space.

[0121] Specifically, a reward function of an initial strategy agent consisting of an operating state and its corresponding control strategy may be set.

[0122] The reward function of the initial policy agent can be:

[0123]

[0124] Combined with the zero-sum game, the reward function of the initial adversarial agent can be set to the opposite of the lower limit of the reward of the initial strategy agent in the sample space;

[0125] Among them, the reward function of the initial adversarial agent can be:

[0126]

[0127] in, It can be the reward value of the initial strategy agent; It can be the running state of the input to the initial strategy agent; can be the corresponding control strategy; D can be the training set composed of the operating states input to the initial strategy agent; , Can be a preset coefficient; It can be the node voltage when the power grid system reaches a quasi-steady state under the current operation state s and control strategy a of the power grid system; It can be a cost function; It can be the reward value of the initial adversarial agent.

[0128] It can be seen from the above technical solution that this embodiment provides an optional method for determining the initial adversarial agent and the initial strategy agent, and the performance of the initial adversarial agent and the initial strategy agent can be further improved through the above method.

[0129] In some embodiments of the present application, the process of step S2, using the confrontation agent to process the system operation point and generate the confrontation operation state is described in detail, and the steps are as follows:

[0130] S20, inputting the system operating point into the adversarial agent to obtain the adversarial operating point output by the adversarial agent.

[0131] Specifically, the system operating point may be an injected power vector in the injected power space.

[0132] The adversarial agent can be used to make extreme adjustments to the injected power vector to form an adversarial operating point.

[0133] S21, simulating the antagonistic operation point to obtain an antagonistic operation state.

[0134] Specifically, the power system simulator can be used to simulate the confrontation operation point and collect the confrontation operation status.

[0135] It can be seen from the above technical solution that this embodiment provides an optional method for obtaining an antagonistic operating state, through which the injected power vector can be adjusted to form an antagonistic operating point and control the probability of the extreme operating point of the present application.

[0136] Next, we will combine Figure 2 The control strategy sample generating device provided in the present application is introduced in detail. The control strategy sample generating device provided below can be compared with the control strategy sample generating method provided above.

[0137] See also Figure 2 It can be found that the control strategy sample generating device may include:

[0138] An acquisition module 10 is used to acquire the system operating point, as well as the interactively trained strategy agent and the adversarial agent, wherein the adversarial agent is trained to control the generation probability of the extreme operating point;

[0139] A processing module 20, for processing the system operation point using the confrontation agent to generate a confrontation operation state;

[0140] The generation module 30 is used to input the confrontation operation state into the strategy agent, and generate a control strategy matching the confrontation operation state. The confrontation operation state and the control strategy form a control strategy sample.

[0141] Furthermore, the acquisition module may include:

[0142] An agent generation unit, used to generate an initial adversarial agent and an initial strategy agent;

[0143] A parameter adjustment unit, configured to adjust the parameters of the initial adversarial agent with the goal of adjusting the extreme operating point generation probability of the initial adversarial agent;

[0144] The agent training unit is used to alternately train the initial strategy agent and the initial confrontation agent with adjusted parameters in combination with the zero-sum game, and use the final initial strategy agent as the strategy agent and the final initial confrontation agent as the confrontation agent.

[0145] Furthermore, the agent generation unit may include:

[0146] The first agent generation component is used to generate an agent constructed by combining the SAC algorithm and the single-step Markov decision process;

[0147] A second agent generation component is used to sequentially input a plurality of historical operating points into the agent to obtain a predicted operating point corresponding to each historical operating point;

[0148] A third agent generation component is used to calculate the similarity between each historical operating point and its corresponding predicted operating point based on KL divergence;

[0149] The fourth agent generation component is used to adjust the parameters of the latest agent based on each similarity until the similarity meets the preset similarity threshold, and the final agent is used as the initial adversarial agent.

[0150] Furthermore, the parameter adjustment unit may include:

[0151] The greedy algorithm utilizes a component for adjusting parameters of the initial adversarial agent based on an ε-greedy algorithm with the goal of adjusting the probability of generating an extreme operating point of the initial adversarial agent.

[0152] Furthermore, the agent training unit may include:

[0153] A reward function determination component is used to determine the reward functions of the initial adversarial agent and the initial strategy agent based on a zero-sum game setting;

[0154] A training operation state acquisition component is used to stop adjusting parameters of the initial adversarial agent and obtain two types of training operation states, wherein one type of training operation state is an operation state randomly selected from the power grid system; and the other type of training operation state is a plurality of operation states obtained based on the latest initial adversarial agent processing;

[0155] A training control strategy acquisition component is used to input each training operation state into the latest initial strategy agent in turn to obtain the training control strategy corresponding to each training operation state;

[0156] A training reward value calculation component is used to calculate the corresponding training reward value based on each training running state and its corresponding training control strategy, combined with the reward function of the initial strategy agent, and adjust the parameters of the latest initial strategy agent based on each training reward value until the stopping condition is met;

[0157] A target operating point acquisition component is used to stop adjusting parameters of the initial strategy agent and obtain multiple target operating points;

[0158] A target running state generation component is used to input each target running point into the latest initial adversarial agent in turn to generate a target running state corresponding to each target running point;

[0159] The target control strategy acquisition component is used to input each target operating state into the latest initial strategy agent in turn to obtain the target control strategy corresponding to each target operating state;

[0160] The target reward value calculation component is used to calculate the corresponding target reward value based on each target operating state and its corresponding target control strategy, combined with the reward function of the initial adversarial agent, and adjust the parameters of the initial adversarial agent based on each target reward value until the stopping condition is met; return to call the training operating state acquisition component until both the initial adversarial agent and the initial strategy agent converge.

[0161] Further, the reward function determination component may include:

[0162] The first reward function determination subcomponent is used to set the reward function of the initial strategy agent, and set the reward function of the initial adversarial agent to the opposite of the lower limit of the reward of the initial strategy agent in the sample space.

[0163] Furthermore, the processing module may include:

[0164] A first processing unit, configured to input the system operating point into the antagonistic agent to obtain an antagonistic operating point output by the antagonistic agent;

[0165] The second processing unit is used to simulate the confrontation operation point to obtain the confrontation operation state.

[0166] The control strategy sample generation device provided in the embodiment of the present application can be applied to control strategy sample generation equipment, such as PC terminals, cloud platforms, servers and server clusters. Figure 3 The hardware structure diagram of the control strategy sample generation device is shown in FIG. Figure 3 ,The hardware structure of the control strategy sample generation device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0167] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0168] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0169] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0170] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0171] Obtaining a system operating point, as well as a strategy agent and an adversarial agent that have undergone interactive training, wherein the adversarial agent is trained to control the generation probability of an extreme operating point;

[0172] Using the confrontation agent, processing the system operation point to generate a confrontation operation state;

[0173] The confrontation running state is input into the strategy agent, and a control strategy matching the confrontation running state is generated. The confrontation running state and the control strategy form a control strategy sample.

[0174] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0175] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0176] Obtaining a system operating point, as well as a strategy agent and an adversarial agent that have undergone interactive training, wherein the adversarial agent is trained to control the generation probability of an extreme operating point;

[0177] Using the confrontation agent, processing the system operation point to generate a confrontation operation state;

[0178] The confrontation running state is input into the strategy agent, and a control strategy matching the confrontation running state is generated. The confrontation running state and the control strategy form a control strategy sample.

[0179] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0180] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0181] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0182] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. The various embodiments of the present application can be combined with each other. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control strategy sample generation method, characterized in that: include: Obtaining a system operating point, as well as a strategy agent and an adversarial agent that have undergone interactive training, wherein the adversarial agent is trained to control the generation probability of an extreme operating point; Using the confrontation agent, processing the system operation point to generate a confrontation operation state; The confrontation running state is input into the strategy agent, and a control strategy matching the confrontation running state is generated. The confrontation running state and the control strategy form a control strategy sample.

2. The control strategy sample generation method according to claim 1, characterized in that: Obtain interactively trained policy agents and adversarial agents, including: Generate initial adversarial agents and initial strategic agents; Adjusting parameters of the initial adversarial agent with the goal of adjusting the probability of generating an extreme operating point of the initial adversarial agent; In combination with zero-sum game, the initial strategy agent and the initial confrontation agent with adjusted parameters are trained alternately, and the final initial strategy agent is used as the strategy agent, and the final initial confrontation agent is used as the confrontation agent.

3. The control strategy sample generation method according to claim 2, characterized in that: Generate an initial adversarial agent, including: Generate an agent that combines the SAC algorithm and the single-step Markov decision process; Inputting a plurality of historical operating points into the intelligent agent in sequence to obtain a predicted operating point corresponding to each historical operating point; Based on KL divergence, the similarity between each historical operating point and its corresponding predicted operating point is calculated; Based on each similarity, the parameters of the latest agent are adjusted until the similarity meets the preset similarity threshold, and the final agent is used as the initial adversarial agent.

4. The control strategy sample generation method according to claim 2, characterized in that: The step of adjusting the parameters of the initial adversarial agent with the goal of adjusting the probability of generating the extreme operating point of the initial adversarial agent comprises: Based on the ε-greedy algorithm, the parameters of the initial adversarial agent are adjusted with the goal of adjusting the probability of generating the extreme operating point of the initial adversarial agent.

5. The control strategy sample generation method according to claim 2, characterized in that: The method of combining the zero-sum game and alternately training the initial strategy agent and the initial confrontation agent after parameter adjustment includes: Determine the reward function of the initial adversarial agent and the initial strategy agent based on the zero-sum game setting; Stop adjusting the parameters of the initial adversarial agent and obtain two types of training operating states, wherein one type of training operating state is an operating state randomly selected from the power grid system; the other type of training operating state is a plurality of operating states obtained based on the latest initial adversarial agent processing; Input each training running state into the latest initial strategy agent in turn to obtain the training control strategy corresponding to each training running state; Based on each training running state and its corresponding training control strategy, combined with the reward function of the initial strategy agent, the corresponding training reward value is calculated, and the parameters of the latest initial strategy agent are adjusted based on each training reward value until the stopping condition is met; Stop adjusting parameters of the initial strategy agent and obtain multiple target operating points; Input each target operating point into the latest initial adversarial agent in turn to generate the target operating state corresponding to each target operating point; Input each target operating state into the latest initial strategy agent in turn to obtain the target control strategy corresponding to each target operating state; Based on each target operating state and its corresponding target control strategy, combined with the reward function of the initial adversarial agent, the corresponding target reward value is calculated, and the parameters of the initial adversarial agent are adjusted based on each target reward value until the stopping condition is met; Return to the step of stopping parameter adjustment of the initial adversarial agent and obtaining two types of training running status until both the initial adversarial agent and the initial policy agent converge.

6. The control strategy sample generation method according to claim 5, characterized in that: The step of determining a reward function of an initial adversarial agent and an initial strategy agent based on a zero-sum game setting includes: Set the reward function of the initial strategy agent, and set the reward function of the initial adversarial agent to the opposite of the lower limit of the reward of the initial strategy agent in the sample space.

7. The control strategy sample generation method according to claim 1, characterized in that: The method of using the confrontation agent to process the system operation point and generate a confrontation operation state includes: Inputting the system operating point into the antagonistic agent to obtain the antagonistic operating point output by the antagonistic agent; The antagonistic operation point is simulated to obtain an antagonistic operation state.

8. A control strategy sample generating device, characterized in that: include: An acquisition module, used to acquire the system operating point, as well as the interactively trained strategy agent and the adversarial agent, wherein the adversarial agent is trained to control the generation probability of extreme operating points; A processing module, used to process the system operation point using the confrontation agent to generate a confrontation operation state; A generation module is used to input the confrontation running state into the strategy agent, and generate a control strategy matching the confrontation running state. The confrontation running state and the control strategy form a control strategy sample.

9. A control strategy sample generating device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the control strategy sample generation method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the control strategy sample generation method according to any one of claims 1 to 7 is implemented.

Images