Virtual power plant system scheduling method and device based on deep reinforcement learning and terminal

Abstract

The virtual power plant system scheduling method, device and terminal based on deep reinforcement learning provided by the application model the simulation environment of the virtual power plant energy system, convert the problem into a Markov decision process, and optimize the PPO algorithm to obtain a PPO-mixclip algorithm; the PPO-mixclip algorithm with a mixed clipping mechanism is used to generate actions, and then the actions are corrected through a backup control strategy, so that the mechanism and the data model are fused. The application provides an effective solution to the actual problem of virtual power plant energy system scheduling, which is of great significance in the field of energy management.

Description

Technical Field

[0001] This invention relates to the field of virtual power plant system scheduling technology, and in particular to a virtual power plant system scheduling method, device and terminal based on deep reinforcement learning. Background Technology

[0002] As the types of electricity loads and the penetration rate of renewable energy continue to increase, distributed renewable energy sources such as photovoltaics are easily affected by natural conditions, exhibiting significant intermittency and volatility. Furthermore, the aggregation of controllable and uncontrollable units (distributed energy sources such as wind and solar), energy storage devices, and loads in virtual power plants exacerbates the overall uncertainty and complexity of the system, making existing dispatching strategies insufficient to meet the system's operational requirements.

[0003] To address the high uncertainty of virtual power plant systems, deep reinforcement learning algorithms offer a promising solution. These algorithms leverage extensive historical data to capture system uncertainty, representing a data-driven approach. Well-trained models exhibit strong generalization capabilities across various uncertain scenarios and can formulate scheduling strategies in real-time. However, the pruning mechanisms of existing proximal policy optimization (PPO) algorithms still have room for improvement. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a virtual power plant system scheduling method, device and terminal based on deep reinforcement learning, so as to solve the above technical problems in the prior art.

[0005] To achieve the above and other related objectives, this invention provides a virtual power plant system scheduling method based on deep reinforcement learning. The method includes: modeling the simulation environment of the virtual power plant energy system and transforming the problem of the constructed virtual power plant energy scheduling optimization model into a Markov decision process; optimizing the PPO algorithm based on the transformed Markov decision process to obtain the PPO-mixclip algorithm; outputting the corresponding action according to the current state of the simulation environment based on the PPO-mixclip algorithm; and correcting the action output by the PPO-mixclip algorithm to obtain a corrected action.

[0006] In one embodiment of the present invention, the step of modeling the simulation environment of the virtual power plant energy system and transforming the constructed virtual power plant energy dispatch optimization model into a problem includes: modeling the simulation environment based on the environmental data of the virtual power plant energy system using a mixed integer linear programming model to obtain a virtual power plant energy dispatch optimization model; and transforming the virtual power plant energy dispatch optimization problem corresponding to the virtual power plant energy dispatch optimization model into a Markov decision process; wherein the Markov decision process includes: a defined state space of the simulation environment, an action space of the agent, and a reward function of the agent.

[0007] In one embodiment of the present invention, optimizing the PPO algorithm based on the Markov decision process obtained by transformation includes: optimizing the policy network and evaluation network of the PPO algorithm based on the state space of the determined simulation environment, the action space of the agent, and the reward function of the agent.

[0008] In one embodiment of the present invention, the optimization of the policy network and evaluation network of the PPO algorithm based on the determined state space of the simulation environment, the action space of the agent, and the reward function of the agent includes: setting initial parameter information; wherein, the initial parameter information includes: initial policy parameters, initial value function parameters, and time steps; based on the determined state space of the simulation environment, the action space of the agent, and the reward function of the agent, the policy network of the PPO algorithm interacts with the simulation environment using the selected current policy to collect samples corresponding to each time step; wherein, the samples include: quadruple information corresponding to the time step, including: The algorithm identifies the state, action, transition state, and reward for each time step. Based on the information from each quadruple, it calculates the reward for each sample and uses the evaluation network of the PPO algorithm to calculate the estimated advantage function for each sample from the calculated state value function value for the corresponding time step. Based on the maximization objective function of the PPO algorithm, it updates the policy network of the PPO algorithm according to the information from each quadruple to obtain the policy network of the PPO-mixclip algorithm. Finally, based on the calculated advantage function estimate for each sample, it updates the evaluation network of the PPO algorithm using mean squared error to obtain the evaluation network of the PPO-mixclip algorithm.

[0009] In one embodiment of the present invention, based on the determined state space of the simulation environment, the action space of the agent, and the reward function of the agent, the policy network of the PPO algorithm interacts with the simulation environment using the selected current policy to collect samples corresponding to each time step. This includes: the policy network using the current policy to select the action to be executed in the current time step from the action space of the agent and interacting with the simulation environment; at the same time, the state of the simulation environment in the current time step is transitioned to another state in the state space of the simulation environment; and a reward corresponding to the current time step is provided based on the reward function of the agent to obtain a quadruple corresponding to the current time step.

[0010] In one embodiment of the present invention, the policy network of the PPO algorithm, which is updated according to the information of each quadruple, is based on the maximization objective function of the PPO algorithm. This includes: the objective function of the PPO algorithm and the objective function of the reinforcement learning algorithm using the sigmoid function, and the policy network is updated by training each sample.

[0011] In one embodiment of the present invention, the objective function of the policy network of the PPO-mixclip algorithm is:

[0012]

[0013] Among them, Γ k The sample set, t is the time step, k is the current policy, and L PPO Let L be the objective function of the PPO algorithm. P3O For the objective function of a reinforcement learning algorithm employing the sigmoid function, L PPO and L P3O The specific formula is as follows:

[0014]

[0015]

[0016] Furthermore, Clip is the clipping mechanism, ρ t S represents the importance sampling ratio at time step t, ε is the clipping factor, and S t Let a be the state at time step t. t For the action at time step t, σ is the estimated value of the dominance function, τ is the hyperparameter, and σ is the sigmoid function.

[0017] In one embodiment of the present invention, the action corresponding to the current state is corrected to obtain the corrected action, which includes: correcting the output battery action based on the battery action correction rule; and correcting the output virtual power plant public generator set action based on the generator set action correction rule. The battery action correction rule includes: if the battery charge after executing the battery action is less than a lower battery charge threshold, then the battery action is corrected to the ratio of a first calculated battery charge value to the time interval of a unit time step; wherein the first calculated battery charge value is the difference between the battery charge before executing the battery action and the lower battery charge threshold; if the battery charge after executing the battery action is greater than a higher battery charge threshold, then the battery action is corrected to the ratio of a second calculated battery charge value to the time interval of a unit time step; wherein the second calculated battery charge value is the difference between the upper battery charge threshold and the battery charge after executing the battery action; if the battery charge after executing the battery action is greater than a higher battery charge threshold, then the battery action is corrected to the ratio of a second calculated battery charge value to the time interval of a unit time step; wherein the second calculated battery charge value is corrected to ... If the power output is less than the lower threshold and not greater than the upper threshold, the battery action will not be corrected. The generator set action correction rules include: if the power output after executing the virtual power plant public generator set action is not greater than 0, the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power output of 0; if the power output after executing the virtual power plant public generator set action is greater than 0 and less than the lower threshold, the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power output equal to the lower threshold; if the power output after executing the virtual power plant public generator set action is not less than the lower threshold and not greater than the upper threshold, the virtual power plant public generator set action will not be corrected; if the power output after executing the virtual power plant public generator set action is greater than the upper threshold, the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power output equal to the upper threshold.

[0018] To achieve the above and other related objectives, this invention provides a virtual power plant system scheduling device based on deep reinforcement learning. The device includes: a simulation environment modeling module for modeling the simulation environment of the virtual power plant energy system and transforming the problem of the constructed virtual power plant energy scheduling optimization model into a Markov decision process; an algorithm optimization module connected to the simulation environment modeling module for optimizing the PPO algorithm based on the transformed Markov decision process to obtain the PPO-mixclip algorithm; an action scheduling module connected to the algorithm optimization module for outputting corresponding actions based on the current state of the simulation environment according to the PPO-mixclip algorithm; and an action correction module connected to the action scheduling module for correcting the actions output by the PPO-mixclip algorithm to obtain corrected actions.

[0019] To achieve the above and other related objectives, the present invention provides a virtual power plant system scheduling terminal based on deep reinforcement learning, comprising: one or more memory and one or more processors; the one or more memory is used to store a computer program; the one or more processors are connected to the memory and are used to run the computer program to execute the virtual power plant system scheduling method based on deep reinforcement learning.

[0020] As described above, this invention provides a virtual power plant system scheduling method, device, and terminal based on deep reinforcement learning, offering the following advantages: This invention models the simulation environment of the virtual power plant energy system and transforms the problem into a Markov decision process. It then optimizes the PPO algorithm to obtain the PPO-mixclip algorithm. Actions are generated using the PPO-mixclip algorithm with a hybrid pruning mechanism, and then the actions are corrected through a backup control strategy, achieving the fusion of mechanism and data model. This invention provides an effective solution to the practical problem of virtual power plant energy system scheduling, which is of great significance in the field of energy management. Attached Figure Description

[0021] Figure 1 The diagram shown is a flowchart illustrating a virtual power plant system scheduling method based on deep reinforcement learning according to an embodiment of the present invention.

[0022] Figure 2 The diagram shown is a schematic representation of a virtual power plant system scheduling device based on deep reinforcement learning, according to an embodiment of the present invention.

[0023] Figure 3 The diagram shown is a schematic representation of the structure of a virtual power plant system scheduling terminal based on deep reinforcement learning in one embodiment of the present invention. Detailed Implementation

[0024] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0025] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0026] Throughout this specification, when it is said that a part is "connected" to another part, this includes not only "direct connection" but also "indirect connection" by placing other elements in between. Furthermore, when it is said that a part "includes" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather means that other constituent elements may also be included.

[0027] The terms "first," "second," and "third," etc., used herein are for the purpose of describing various parts, components, regions, layers, and / or segments, but are not limiting. These terms are used only to distinguish one part, component, region, layer, or segment from others. Therefore, the "first part," "component," "region," "layer," or "segment" described below may refer to a "second part," "component," "region," "layer," or "segment" without departing from the scope of this invention.

[0028] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0029] This invention provides a method, device, and terminal for scheduling virtual power plant systems based on deep reinforcement learning. It models the simulation environment of a virtual power plant energy system, transforms the problem into a Markov decision process, and then optimizes the PPO algorithm to obtain the PPO-mixclip algorithm. Actions are generated using the PPO-mixclip algorithm with a hybrid pruning mechanism, and then corrected using a backup control strategy, achieving a fusion of mechanism and data model. This invention provides an effective solution to the practical problem of scheduling virtual power plant energy systems, which is of great significance in the field of energy management.

[0030] The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement it. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0031] like Figure 1 This is a flowchart illustrating a virtual power plant system scheduling method based on deep reinforcement learning in an embodiment of the present invention.

[0032] The method includes:

[0033] Step S1: Model the simulation environment of the virtual power plant energy system and transform the problem of the constructed virtual power plant energy dispatch optimization model into a Markov decision process.

[0034] In one embodiment, step S1 includes:

[0035] Based on the mixed-integer linear programming (MILP) model, the simulation environment is modeled using environmental data from the virtual power plant energy system to obtain the virtual power plant energy dispatch optimization model; wherein, the environmental data of the virtual power plant energy system includes: photovoltaic power generation data and load data, etc.

[0036] The virtual power plant energy dispatch optimization problem corresponding to the aforementioned virtual power plant energy dispatch optimization model is transformed into a Markov decision process. The Markov decision process includes: a defined state space of the simulation environment, an action space of the agent, and a reward function for the agent. The state space of the simulation environment contains multiple states, the action space of the agent contains multiple actions, and the reward function of the agent is used to determine the reward.

[0037] Specifically, the state space of the simulation environment includes states such as photovoltaic power generation, load information, and battery power consumption; the action space of the intelligent agent includes actions such as battery actions and actions of the public generator sets of the virtual power plant.

[0038] Step S2: Based on the Markov decision process obtained by transformation, optimize the PPO algorithm to obtain the PPO-mixclip algorithm.

[0039] In detail, the PPO algorithm is a deep reinforcement learning algorithm with a pruning mechanism, employing a policy network and an evaluation network.

[0040] In one embodiment, step S2 includes:

[0041] Based on the defined state space of the simulation environment, the action space of the agent, and the reward function of the agent, the policy network and evaluation network of the PPO algorithm are optimized to obtain the PPO-mixclip algorithm.

[0042] In one embodiment, optimizing the policy network and evaluation network of the PPO algorithm based on the state space of the determined simulation environment, the action space of the agent, and the reward function of the agent includes:

[0043] Set the initial parameter information for the algorithm; wherein, the initial parameter information includes: policy initial parameters θ0, initial value function parameters. And the number of time steps t;

[0044] Based on the state space of the determined simulation environment, the action space of the agent, and the agent's reward function, the policy network of the PPO algorithm utilizes the initial policy parameters and the selected current policy π. k =π(θ) k It interacts with the simulation environment to collect samples corresponding to each time step under the current strategy, in order to collect a sample set Γ containing samples of a set number of time steps. k ={τ i}; wherein, the sample includes: quadruple information corresponding to the time step; the quadruple information includes: the state S of the current time step. t Action A at the current time step t The state of transition S t+1 And the reward R for the current time step t ;

[0045] Calculate the return for each sample based on the information from each quadruple. The PPO algorithm's evaluation network calculates the advantage function estimate for each sample based on the current state value function. Specifically, it calculates the reward for each sample by applying a discount factor to the reward value Rt at each time step from the end to the beginning. The network is evaluated by using the initial value function parameters to calculate the state value function value at the current time step. Calculate the advantage function estimate for each sample.

[0046]

[0047] Based on the maximization objective function of the PPO algorithm, the policy network of the PPO algorithm is updated according to the information of each quadruple to obtain the policy network of the PPO-mixclip algorithm.

[0048] Based on the calculated advantage function estimate for each sample, the evaluation network of the PPO algorithm is updated using mean squared error to obtain the evaluation network of the PPO-mixclip algorithm.

[0049] In one embodiment, based on the state space of the determined simulation environment, the action space of the agent, and the reward function of the agent, the policy network of the PPO algorithm interacts with the simulation environment using the selected current policy, collecting samples corresponding to each time step, including:

[0050] The policy network uses the current policy to select the action A to be executed at the current time step t from the agent's action space. t ;

[0051] Interact with the simulation environment, and simultaneously monitor the state S of the simulation environment at the current time step. t The state S is transferred to another state S in the state space of the simulation environment. t+1 And based on the agent's reward function, it provides the reward R for the current time step. t This is done to obtain the quadruple corresponding to the current time step. Based on the above method, samples corresponding to each time step are collected into the sample set.

[0052] In one embodiment, updating the policy network of the PPO algorithm based on the maximization objective function of the PPO algorithm and the information of each quadruple includes:

[0053] The objective function is based on the PPO algorithm and the objective function of the reinforcement learning algorithm using the sigmoid function. The policy network is then trained and updated based on each sample.

[0054] In one specific embodiment, the maximization objective function of the original PPO algorithm is:

[0055]

[0056] Clip is the clipping mechanism, ρ t S represents the importance sampling ratio at time step t, ε is the clipping factor, and S t Let a be the state at time step t. t For the action at time step t, This is the estimated value of the advantage function.

[0057] And ρ t The calculation formula is:

[0058]

[0059] The maximization objective functions of reinforcement learning algorithms using the sigmoid function include:

[0060]

[0061] Where σ is the sigmoid function, τ is a hyperparameter that can be taken as 4, and ρ t S represents the importance sampling ratio at time step t. t Let a be the state at time step t. t For the action at time step t, This is the estimated value of the advantage function.

[0062] The objective function of the policy network of the PPO-mixclip algorithm of this invention is updated using the maximization objective function of the original PPO algorithm and the maximization objective function of the reinforcement learning algorithm using the sigmoid function, as follows:

[0063]

[0064] Among them, Γ k The sample set is given, t is the time step, k is the current policy, LPPO is the objective function of the PPO algorithm, and LP3O is the objective function of the reinforcement learning algorithm using the sigmoid function.

[0065] Step S3: Based on the PPO-mixclip algorithm, output the corresponding action according to the current state of the simulation environment.

[0066] In detail, the current state of the simulation environment is input into the PPO-mixclip algorithm, which then outputs the action corresponding to the current state of the simulation environment.

[0067] Step S4: Modify the action output by the PPO-mixclip algorithm to obtain the modified action.

[0068] In one embodiment, step S4 includes:

[0069] Based on the battery action correction rules, the output battery action is corrected;

[0070] Based on the generator set action correction rules, the output virtual power plant public generator set actions are corrected;

[0071] The battery action correction rules include:

[0072] If the battery level after the battery action is less than the lower battery level threshold, then the battery action corrects the battery level to the ratio of the first calculated battery level value to the time interval of the unit time step; wherein, the first calculated battery level value is the difference between the battery level before the battery action is executed and the lower battery level threshold.

[0073] If the battery level after performing the battery action is greater than the upper threshold of the battery level, then the battery action is corrected to the battery action that is the ratio of the second calculated battery level value to the time interval of the unit time step; wherein, the second calculated battery level value is corrected to the difference between the upper threshold of the battery level and the battery level after performing the battery action.

[0074] If the battery level after the action is less than the lower threshold and not greater than the upper threshold, the battery action will not be corrected.

[0075] Specifically, if the battery's action at time step t is... The battery level after execution without modification is... State transition to Based on this value and the upper threshold of battery capacity and the lower threshold of battery power e b Compare and make corresponding corrections.

[0076]

[0077] The generator set operation correction rules include:

[0078] If the power generation after executing the virtual power plant public generator set action is not greater than 0, then the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power generation of 0.

[0079] If the power generation after executing the virtual power plant public generator set action is greater than 0 and less than the lower threshold of power generation, then the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with power generation equal to the lower threshold of power generation.

[0080] If the power generation after the virtual power plant public generator set operation is not less than the lower threshold of power generation and not greater than the upper threshold of power generation, then the virtual power plant public generator set operation will not be corrected.

[0081] If the power generation after the virtual power plant public generator set action is greater than the upper threshold of power generation, then the virtual power plant public generator set action will correct the power generation to the upper threshold of the virtual power plant public generator set action.

[0082] Specifically, if the generator set's action at time step t is... The power generation at this time meets the upper threshold of power generation. and the battery power threshold p dg Constraints, based on this value and the upper threshold of power generation. and the battery power threshold p dg Compare and make corresponding corrections.

[0083]

[0084] Preferably, if there is still a power difference value after the virtual power plant's public generator set operation correction, then the generator set is used to meet the generated power difference value, that is, the operation of the generator set is corrected a second time to ensure that the virtual power plant's load demand is met in real time.

[0085] Experimental results using the PPO-mixclip-based virtual power plant system scheduling algorithm framework of this invention show that, in partially domain adaptive tasks, the proposed algorithm reduces the average daily electricity cost by 4.24%, outperforming other algorithms. This invention provides an effective solution to the practical problem of virtual power plant system scheduling strategies, which is of great significance in the field of energy management.

[0086] Similar to the principles of the above embodiments, the present invention provides a virtual power plant system scheduling device based on deep reinforcement learning.

[0087] The following specific embodiments are provided in conjunction with the accompanying drawings:

[0088] like Figure 2 This invention presents a schematic diagram of a virtual power plant system scheduling device based on deep reinforcement learning, as described in an embodiment of the present invention.

[0089] The device includes:

[0090] Simulation environment modeling module 1 is used to model the simulation environment of the virtual power plant energy system and transform the problem of the constructed virtual power plant energy dispatch optimization model into a Markov decision process.

[0091] Algorithm optimization module 2, connected to the simulation environment modeling module 1, is used to optimize the PPO algorithm based on the Markov decision process obtained by transformation to obtain the PPO-mixclip algorithm;

[0092] Action scheduling module 3, connected to the algorithm optimization module 2, is used to output the corresponding action based on the current state of the simulation environment according to the PPO-mixclip algorithm;

[0093] Action correction module 4, connected to action scheduling module 3, is used to correct the actions output by the PPO-mixclip algorithm to obtain corrected actions.

[0094] It should be noted that, as should be understood Figure 2 The division of modules in the system embodiment is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these units can be implemented entirely in software through processing element calls; they can be implemented entirely in hardware; or some units can be implemented by processing element calls to software, while others are implemented in hardware.

[0095] Since the implementation principle of the virtual power plant system scheduling device based on deep reinforcement learning has been described in the previous embodiments, it will not be repeated here.

[0096] In one embodiment, modeling the simulation environment of the virtual power plant energy system and transforming the constructed virtual power plant energy dispatch optimization model into a problem includes: modeling the simulation environment based on the environmental data of the virtual power plant energy system using a mixed-integer linear programming model to obtain a virtual power plant energy dispatch optimization model; and transforming the virtual power plant energy dispatch optimization problem corresponding to the virtual power plant energy dispatch optimization model into a Markov decision process; wherein the Markov decision process includes: a defined state space of the simulation environment, an action space of the agent, and a reward function of the agent.

[0097] In one embodiment, optimizing the PPO algorithm based on the Markov decision process obtained by transformation includes optimizing the policy network and evaluation network of the PPO algorithm based on the state space of the determined simulation environment, the action space of the agent, and the reward function of the agent.

[0098] In one embodiment, optimizing the policy network and evaluation network of the PPO algorithm based on the determined state space of the simulation environment, the action space of the agent, and the reward function of the agent includes: setting initial parameter information; wherein the initial parameter information includes: initial policy parameters, initial value function parameters, and the number of time steps; based on the determined state space of the simulation environment, the action space of the agent, and the reward function of the agent, the policy network of the PPO algorithm interacts with the simulation environment using the selected current policy to collect samples corresponding to each time step; wherein the samples include: quadruple information corresponding to the time step, including: corresponding... The algorithm considers the state at each time step, the action at the corresponding time step, the state during transitions, and the reward at the corresponding time step. Based on the information from each quadruple, it calculates the reward for each sample and uses the evaluation network of the PPO algorithm to calculate the estimated advantage function for each sample from the calculated state value function value at the corresponding time step. Based on the maximization objective function of the PPO algorithm, it updates the policy network of the PPO algorithm according to the information from each quadruple to obtain the policy network of the PPO-mixclip algorithm. Finally, based on the calculated advantage function estimate for each sample, it uses mean squared error to update the evaluation network of the PPO algorithm to obtain the evaluation network of the PPO-mixclip algorithm.

[0099] In one embodiment, based on the defined state space of the simulation environment, the action space of the agent, and the reward function of the agent, the policy network of the PPO algorithm interacts with the simulation environment using the selected current policy to collect samples corresponding to each time step. This includes: the policy network using the current policy to select the action to be executed in the current time step from the action space of the agent and interacting with the simulation environment; at the same time, the state of the simulation environment in the current time step is transitioned to another state in the state space of the simulation environment; and a reward corresponding to the current time step is provided based on the reward function of the agent to obtain a quadruple corresponding to the current time step.

[0100] In one embodiment, the policy network of the PPO algorithm, which is updated based on the maximization objective function of the PPO algorithm and the information of each quadruple, includes: the objective function of the PPO algorithm and the objective function of the reinforcement learning algorithm using the sigmoid function, and the policy network is updated by training each sample.

[0101] In one embodiment, the objective function of the policy network of the PPO-mixclip algorithm is:

[0102]

[0103] Among them, Γ k The sample set, t is the time step, k is the current policy, and L PPO Let L be the objective function of the PPO algorithm. P3OFor the objective function of a reinforcement learning algorithm employing the sigmoid function, L PPO and L P3O The specific formula is as follows:

[0104]

[0105]

[0106] Furthermore, Clip is the clipping mechanism, ρ t S represents the importance sampling ratio at time step t, ε is the clipping factor, and S t Let a be the state at time step t. t For the action at time step t, σ is the estimated value of the dominance function, τ is the hyperparameter, and σ is the sigmoid function.

[0107] In one embodiment, correcting the action corresponding to the current state to obtain the corrected action includes: correcting the output battery action based on battery action correction rules; correcting the output virtual power plant public generator set action based on generator set action correction rules; wherein, the battery action correction rules include: if the battery level after executing the battery action is less than the lower threshold of battery level, then the battery action is corrected to the ratio of a first calculated battery level value to the time interval of a unit time step; wherein, the first calculated battery level value is the difference between the battery level before executing the battery action and the lower threshold of battery level; if the battery level after executing the battery action is greater than the upper threshold of battery level, then the battery action is corrected to the ratio of a second calculated battery level value to the time interval of a unit time step; wherein, the second calculated battery level value is corrected to the difference between the upper threshold of battery level and the battery level after executing the battery action; if the battery level after the battery action is less than the lower ... If the battery power level is below the lower threshold and not greater than the upper threshold, the battery action will not be corrected. The generator set action correction rules include: if the power generation after executing the virtual power plant public generator set action is not greater than 0, the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power generation of 0; if the power generation after executing the virtual power plant public generator set action is greater than 0 and less than the lower threshold, the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power generation of the lower threshold; if the power generation after executing the virtual power plant public generator set action is not less than the lower threshold and not greater than the upper threshold, the virtual power plant public generator set action will not be corrected; if the power generation after executing the virtual power plant public generator set action is greater than the upper threshold, the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power generation of the upper threshold.

[0108] like Figure 3 This is a schematic diagram illustrating the structure of a virtual power plant system scheduling terminal based on deep reinforcement learning in an embodiment of the present invention.

[0109] The virtual power plant system scheduling terminal 30 based on deep reinforcement learning includes: a memory 31 and a processor 32. The memory 31 is used to store computer programs; the processor 32 runs the computer programs to implement, for example,... Figure 1 The aforementioned virtual power plant system scheduling method based on deep reinforcement learning.

[0110] Optionally, the number of memories 31 can be one or more, and the number of processors 32 can be one or more. Figure 3 Each example is taken as an instance.

[0111] Optionally, the processor 32 in the deep reinforcement learning-based virtual power plant system scheduling terminal 30 will follow the instructions as follows: Figure 1 The steps described involve loading one or more instructions corresponding to the process of an application into memory 31, and then having the processor 32 run the application stored in the first memory 31, thereby achieving the following: Figure 1 The various functions in the virtual power plant system scheduling method based on deep reinforcement learning.

[0112] Optionally, the memory 31 may include, but is not limited to, high-speed random access memory and non-volatile memory. For example, one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices; the processor 32 may include, but is not limited to, a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0113] Optionally, the processor 32 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0114] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed, implements as follows: Figure 1 The illustrated method is a virtual power plant system scheduling method based on deep reinforcement learning. The computer-readable storage medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs (Read-Only Optical Disk Memory), magneto-optical disks, ROMs (Read-Only Memory), RAMs (Random Access Memory), EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable Programmable Read-Only Memory), magnetic cards or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product not connected to a computer device or a component used in a computer device.

[0115] In summary, the virtual power plant system scheduling device based on deep reinforcement learning of this invention models the simulation environment of the virtual power plant energy system, transforms the problem into a Markov decision process, and then optimizes the PPO algorithm to obtain the PPO-mixclip algorithm. Actions are generated through the PPO-mixclip algorithm with a hybrid pruning mechanism, and then the actions are corrected through a backup control strategy, achieving the fusion of mechanism and data model. This invention provides an effective solution to the practical problem of virtual power plant energy system scheduling, which is of great significance in the field of energy management. Therefore, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0116] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A virtual power plant system scheduling method based on deep reinforcement learning, characterized in that, The method includes: The simulation environment of the virtual power plant energy system is modeled, and the problem of the constructed virtual power plant energy dispatch optimization model is transformed into a Markov decision process. Based on the Markov decision process obtained through transformation, the PPO algorithm is optimized to obtain the PPO-mixclip algorithm; Based on the PPO-mixclip algorithm, the corresponding action is output according to the current state of the simulation environment; The action output by the PPO-mixclip algorithm is modified to obtain the modified action; The process of modeling the simulation environment of the virtual power plant energy system and transforming the constructed virtual power plant energy dispatch optimization model into a problem includes: Based on a mixed-integer linear programming model, the simulation environment is modeled using environmental data from the virtual power plant energy system to obtain a virtual power plant energy dispatch optimization model. The virtual power plant energy dispatch optimization problem corresponding to the virtual power plant energy dispatch optimization model is transformed into a Markov decision process; wherein, the Markov decision process includes: a defined state space of the simulation environment, an action space of the agent, and a reward function of the agent; the state space of the simulation environment includes states such as photovoltaic power generation, load information, and battery power consumption; the action space of the agent includes actions such as battery actions and public generator set actions of the virtual power plant; Based on the Markov decision process obtained through transformation, the PPO algorithm is optimized as follows: Based on the state space of the defined simulation environment, the action space of the agent, and the reward function of the agent, the policy network and evaluation network of the PPO algorithm are optimized. The action corresponding to the current state is modified to obtain the modified action, which includes: Based on the battery action correction rules, the output battery action is corrected; Based on the generator set action correction rules, the output virtual power plant public generator set actions are corrected; The battery action correction rules include: If the battery level after the battery action is less than the lower battery level threshold, then the battery action corrects the battery level to the ratio of the first calculated battery level value to the time interval of the unit time step; wherein, the first calculated battery level value is the difference between the battery level before the battery action is executed and the lower battery level threshold. If the battery level after performing the battery action is greater than the upper threshold of the battery level, then the battery action is corrected to the battery action that is the ratio of the second calculated battery level value to the time interval of the unit time step; wherein, the second calculated battery level value is corrected to the difference between the upper threshold of the battery level and the battery level after performing the battery action. If the battery level after the action is less than the lower threshold and not greater than the upper threshold, the battery action will not be corrected. The generator set operation correction rules include: If the power generation after executing the virtual power plant public generator set action is not greater than 0, then the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with a power generation of 0. If the power generation after executing the virtual power plant public generator set action is greater than 0 and less than the lower threshold of power generation, then the virtual power plant public generator set action will be corrected to a virtual power plant public generator set action with power generation equal to the lower threshold of power generation. If the power generation after the virtual power plant public generator set operation is not less than the lower threshold of power generation and not greater than the upper threshold of power generation, then the virtual power plant public generator set operation will not be corrected. If the power generation after the virtual power plant public generator set action is greater than the upper threshold of power generation, then the virtual power plant public generator set action will correct the power generation to the upper threshold of the virtual power plant public generator set action.

2. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, The optimization of the policy network and evaluation network of the PPO algorithm based on the state space of the determined simulation environment, the action space of the agent, and the reward function of the agent includes: Set initial parameter information; wherein, the initial parameter information includes: initial policy parameters, initial value function parameters, and time steps; Based on the determined state space of the simulation environment, the action space of the agent, and the reward function of the agent, the policy network of the PPO algorithm interacts with the simulation environment using the selected current policy to collect samples corresponding to each time step; wherein, the samples include: quadruple information for the corresponding time step, including: the state of the corresponding time step, the action of the corresponding time step, the transition state, and the reward of the corresponding time step. The reward for each sample is calculated based on the information of each quadruple, and the advantage function estimate for each sample is calculated from the state value function value of the corresponding time step through the evaluation network of the PPO algorithm. Based on the maximization objective function of the PPO algorithm, the policy network of the PPO algorithm is updated according to the information of each quadruple to obtain the policy network of the PPO-mixclip algorithm. Based on the calculated advantage function estimate for each sample, the evaluation network of the PPO algorithm is updated using mean squared error to obtain the evaluation network of the PPO-mixclip algorithm.

3. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 2, characterized in that, Based on the defined state space of the simulation environment, the action space of the agent, and the agent's reward function, the policy network of the PPO algorithm interacts with the simulation environment using the selected current policy, collecting samples corresponding to each time step, including: The policy network uses the current policy to select the action to be executed at the current time step from the agent's action space and interact with the simulation environment. At the same time, the state of the simulation environment at the current time step is transitioned to another state in the state space of the simulation environment, and a reward corresponding to the current time step is provided based on the agent's reward function to obtain the quadruple corresponding to the current time step.

4. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 2, characterized in that, The maximization objective function based on the PPO algorithm, and the policy network of the PPO algorithm updated according to the information of each quadruple, include: The objective function is based on the PPO algorithm and the objective function of the reinforcement learning algorithm using the sigmoid function. The policy network is then trained and updated based on each sample.

5. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 4, characterized in that, The objective function of the policy network of the PPO-mixclip algorithm is: ； in, The sample set, t is the time step, k is the current policy, and L PPO Let L be the objective function of the PPO algorithm. P3O For the objective function of a reinforcement learning algorithm employing the sigmoid function, L PPO and L P3O The specific formula is as follows: ； ； Furthermore, Clip is the clipping mechanism. This indicates the importance sampling ratio at time step t. For the clipping factor, The state at time step t. For the action at time step t, This is the estimated value of the dominance function. For hyperparameters, This is the sigmoid function.

6. A virtual power plant system scheduling device based on deep reinforcement learning, characterized in that, The apparatus, used in any one of claims 1 to 5, comprises: The simulation environment modeling module is used to model the simulation environment of the virtual power plant energy system and transform the problem of the constructed virtual power plant energy dispatch optimization model into a Markov decision process. The algorithm optimization module, connected to the simulation environment modeling module, is used to optimize the PPO algorithm based on the Markov decision process obtained by transformation to obtain the PPO-mixclip algorithm; The action scheduling module, connected to the algorithm optimization module, is used to output the corresponding action based on the current state of the simulation environment according to the PPO-mixclip algorithm. An action correction module, connected to the action scheduling module, is used to correct the actions output by the PPO-mixclip algorithm to obtain corrected actions.

7. A virtual power plant system dispatching terminal based on deep reinforcement learning, characterized in that, include: One or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are used to run the computer program to perform the method as described in any one of claims 1 to 5.

Images