New energy consumption capability assessment method and device, and medium

Abstract

The invention discloses a new energy consumption capability assessment method and device and a medium. The method comprises the steps of building a deep reinforcement learning model for new energy consumption capability evaluation; constructing a deep reinforcement learning framework based on a Lagrange flexible action evaluation algorithm; training the deep reinforcement learning model by adopting a deep reinforcement learning framework based on historical new energy output data and load data, and generating a new energy consumption capability evaluation agent; and performing consumption evaluation on the new energy output data and the load data of the to-be-evaluated scene by using the new energy consumption capability evaluation agent, and determining a new energy consumption capability evaluation result of the to-be-evaluated scene.

Description

Technical Field

[0001] The present invention relates to the technical field of new energy consumption capacity assessment, and more specifically, to a new energy consumption capacity assessment method, device and medium. Background Art

[0002] As new energy gradually becomes the main installed capacity and turns to the main power, its output characteristics such as randomness, volatility and anti-peak characteristics have an increasingly prominent impact on the power balance of the system. Scientifically and quantitatively evaluating the new energy consumption level of the power grid under different scenarios will help to formulate reasonable new energy planning and development goals and supporting measures to promote consumption, and promote the clean transformation of the power system.

[0003] Traditional quantitative evaluation methods for renewable energy absorption capacity are usually based on production simulation and other methods. By considering the power system source, grid, load and storage operation constraints, a mixed integer mathematical optimization model is established and solved using mathematical programming techniques. Existing methods mainly have two problems: high computational complexity and insufficient dynamic adaptability. On the one hand, the mixed integer mathematical optimization model needs to process a large number of combinations of integers and continuous variables. With the increase in the number of renewable energy sites and grid nodes, the scale of the optimization problem grows exponentially, resulting in a rapid increase in computational complexity and difficulty in obtaining an effective solution within a reasonable time. On the other hand, the randomness and volatility of renewable energy output make the traditional mixed integer programming model lack flexibility when facing the changing renewable energy output scenarios, making it difficult to dynamically adapt to the real-time scheduling needs of the system, limiting its effectiveness in practical applications.

[0004] In recent years, the rapid development of deep reinforcement learning (DRL) technology has provided a new idea for solving the problem of optimal dispatching of complex power systems. DRL can effectively cope with the volatility of renewable energy output and significantly improve the efficiency of solution by learning the operation strategy of the system under different states. Existing studies have shown that the DRL method has a significant effect on the dispatch optimization of power systems on a short time scale. However, for the problems of power system production simulation and renewable energy consumption evaluation on a long time scale, the existing research is relatively limited, especially for how to evaluate the renewable energy consumption capacity under the regulation of energy storage, there is still a lack of effective solutions.

[0005] Existing deep reinforcement learning algorithms, such as soft actor-critic (SAC) and deep deterministic policy gradient (DDPG), have certain limitations when dealing with complex power system operation constraints. They are unable to effectively meet the power balance equation constraints, which in turn causes the evaluation results of the system's absorption capacity to deviate from actual needs. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a method, device and medium for evaluating the new energy consumption capacity.

[0007] According to one aspect of the present invention, a method for evaluating new energy consumption capacity is provided, comprising:

[0008] Build a deep reinforcement learning model for evaluating new energy consumption capacity;

[0009] Construct a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm;

[0010] Based on historical renewable energy output data and load data, a deep reinforcement learning framework is used to train the deep reinforcement learning model to generate an intelligent agent for evaluating renewable energy consumption capacity.

[0011] The new energy consumption capacity assessment intelligent body is used to conduct consumption assessment on the new energy output data and load data of the assessment scenario to determine the assessment result of the new energy consumption capacity of the assessment scenario.

[0012] Optionally, build a deep reinforcement learning model for evaluating new energy consumption capacity, including:

[0013] Construct a simulation environment for evaluating the absorption capacity of new energy in the power system. The absorption capacity evaluation model includes: system power balance constraints, system reserve constraints, wind and solar output constraints, thermal power unit operation constraints, and energy storage power station operation constraints.

[0014] Construct an agent action space of all the action sets taken by the agent in each operating state, where the action set includes: thermal power unit output, number of thermal power units in operation, and energy storage charging and discharging power;

[0015] Construct the agent state space of all the operating states of the agent at each moment, where the operating state set includes: daily load demand, available wind and solar power generation, wind and solar power generation grid-connected power, thermal power unit output, number of thermal power units in operation, energy storage power station charge state and the action at the previous moment;

[0016] Construct an agent reward and punishment mechanism to determine the feedback obtained after the agent takes a certain action in a specific state, where the feedback includes: the operating cost of thermal power units, the cost of power abandonment and load shedding, the penalty cost of system imbalance, the penalty cost of thermal power ramp-up exceeding the limit, and the penalty cost of energy storage power station charging and discharging power exceeding the limit;

[0017] A deep reinforcement learning model is constructed based on the absorption capacity evaluation model, agent action space, agent state space and agent reward and punishment mechanism.

[0018] Optionally, a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm is constructed, including:

[0019] Construct the basic theoretical model of Lagrangian flexible motion evaluation algorithm;

[0020] Construct a network model of Lagrangian flexible action evaluation algorithm;

[0021] Based on the basic theoretical model and network model, a deep reinforcement learning framework is constructed.

[0022] Optionally, the objective function of the basic theoretical model is:

[0023]

[0024] In the formula, π* is the optimal strategy; r(s t ,a t ) is the immediate reward function; H(π(·|s t )) is state s t Next, take action a t The entropy regularization term; α is the temperature coefficient, λ t is the Lagrange multiplier, c(s t ,a t ) is an auxiliary cost function representing the degree of constraint violation.

[0025] Optionally, the network module includes: a policy network, a value network, a Lagrangian network and a target network, wherein

[0026] The loss function of the policy network guiding the network parameter update is:

[0027]

[0028] In the formula, J π (θ π ) indicates that the parameter θ π The loss function under Represents the state s sampled in the experience pool D t And according to the strategy The sampled action a t The expectation under the sampling (s t ,a t ) is the current state action pair, α is the temperature coefficient, λ t is the Lagrange multiplier, Q π (s t ,a t ) is the state-action value function of the agent at time t; is the state-value function of the agent at time t; Cπ (s t ,a t ) is the auxiliary cost state-action value function of the agent at time t; is the state-auxiliary cost value function of the agent at time t+1; γ is the discount factor;

[0029] The loss function of the value network guiding the network parameter update is:

[0030]

[0031] In the formula, q t is the target Q value;

[0032] The loss function of the Lagrangian network to guide the update of network parameters is:

[0033]

[0034] In the formula, c t is the target C value;

[0035] The loss function of the target network to guide the network parameter update is:

[0036]

[0037] θ'=τθ+(1-τ)θ'

[0038] Where, d t is the termination mark. When the training is finished, d t is 1, otherwise it is 0; (s t+1 ,a t+1 ) is the state-action pair for the next moment; is the target network’s state-action pair for the next moment (s t+1 ,a t+1 )’s estimated Q value; is the instantaneous cost value at time t; is the target network’s state-action pair for the next moment (s t+1 ,a t+1 )’s estimated C value; π θ (a t+1 |s t+1 ) in a given state s t+1 When , the action a is generated by the strategy π (with parameter θ) t+1 The probability of; θ is the current network parameter; θ' is the target network parameter; τ is the soft update learning rate.

[0039] Optionally, a deep reinforcement learning model is trained using a deep reinforcement learning framework based on historical renewable energy output data and load data to generate a renewable energy consumption capacity assessment agent, including:

[0040] Step 1: State initialization: extract one day from the historical wind, solar and load sequences as the system operation state P0 at each moment of the current operation day D0, and use the unit operation state at moment t0 of the day as the initial state value s0;

[0041] Step 2: Collect experience pairs. During the interaction between the policy network and the system simulation environment, collect a new set of experience pairs and determine whether the experience pairs in the experience replay pool have reached the specified number. The experience data pairs include running status, action, reward, cost, and next step status.

[0042] If the specified number is not reached, the new experience number pair is stored in the experience replay pool, and the time t=t+1 is set, and then further judge whether t≤24 is established. If so, step 2 is re-executed; if not, the number of days d=d+1 is set, and step 1 is returned to execute; if the specified number is reached, the experience number pair stored in the front row of the experience replay pool is deleted, and the new experience number pair is stored at the end of the experience replay pool, and step 3 is continued;

[0043] Step 3: Neural grid parameter update: randomly sample n sets of experience pairs from the experience replay pool, and extract the running state set {s1, s2, …, s n} as the input of the policy network, output the corresponding action probability distribution parameters, and further sample and select the current specific action; the state-action pair (s k ,a k ), k=1,2,…,n are input into the value network Q1, value network Q2, Lagrangian network C1 and Lagrangian network C2 in turn, and the corresponding Q values ​​are calculated respectively. and C value The target Q value q is calculated by using the target value network TQ1 and the target value network TQ2 and the target Lagrangian network TC1 and the target Lagrangian network TC2 respectively. k With the target C value c k ; Calculate the loss function J respectively π (θ π ), J Q (θ Q ), J C (θ C ), realize the parameter update of the policy network, value network and Lagrangian network, and use the soft update strategy to realize the parameter update of the target network;

[0044] Step 4: Iterate the loop to determine whether the agent training rounds have reached the specified number of days. The iteration cycle ends and the new energy consumption capacity evaluation intelligent entity is generated; if the number of days Then set the time t=t+1, and further determine whether t≤24 holds; if so, return to step 2; if not, set the number of days d=d+1, and return to step 1.

[0045] Optionally, the new energy consumption capacity assessment intelligent agent is used to conduct consumption assessment on the new energy output data and load data of the scenario to be assessed, and the new energy consumption capacity assessment result of the scenario to be assessed is determined, including:

[0046] Input the predicted output data of new energy and load forecast data of the scenario to be evaluated into the simulation environment;

[0047] Using the new energy consumption capacity to evaluate the intelligent agent to accept the state s from the simulation environment t ;

[0048] According to the acceptance status s t And the optimal strategy π* obtained in advance selects action a t ;

[0049] The selected action a t Transmitted back to the simulation environment to get the state s at the next moment t+1 , repeat the selection action until the consumption assessment is completed, and determine the evaluation result of the new energy consumption capacity of the scenario to be evaluated.

[0050] According to another aspect of the present invention, there is provided a new energy consumption capacity evaluation device, comprising:

[0051] Build a module for building a deep reinforcement learning model for evaluating new energy absorption capacity;

[0052] Building blocks for constructing a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm;

[0053] A generation module is used to train a deep reinforcement learning model based on historical new energy output data and load data using a deep reinforcement learning framework to generate an intelligent agent for evaluating new energy absorption capacity;

[0054] The determination module is used to use the new energy consumption capacity assessment intelligent body to conduct consumption assessment on the new energy output data and load data of the evaluation scene to be evaluated, and determine the evaluation result of the new energy consumption capacity of the evaluation scene to be evaluated.

[0055] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0056] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0057] Therefore, the method of the present invention combines the Lagrangian multiplier method with the flexible action-evaluation algorithm, does not need to rely on the traditional mixed integer programming model, has high computational efficiency and strong training stability, and can realize accurate quantitative evaluation of the new energy absorption capacity; by introducing the Lagrangian network, the patented method can effectively solve the conflict problem between the intelligent agent's immediate reward and the power balance equation constraint, and can effectively reduce the system power imbalance, and improve the system's new energy absorption capacity on the basis of ensuring the system power supply balance; in addition, the method of the present invention can take into account the regulating role of the energy storage system, optimize the energy storage operation strategy, and increase the new energy absorption by increasing the energy storage call efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0059] Figure 1 It is a flow chart of a new energy consumption capacity evaluation method provided by an exemplary embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of a neural network parameter updating process in a Lag-SAC algorithm provided by an exemplary embodiment of the present invention;

[0061] Figure 3 It is a schematic diagram of a consumption evaluation training process based on a Lag-SAC algorithm provided by an exemplary embodiment of the present invention;

[0062] Figure 4 It is a schematic diagram of a new energy consumption capacity evaluation process based on the Lag-SAC algorithm provided by an exemplary embodiment of the present invention;

[0063] Figure 5 is a schematic diagram comparing system production simulation results under typical scenarios in each season provided by an exemplary embodiment of the present invention;

[0064] Figure 6 It is a structural schematic diagram of a new energy consumption capacity evaluation device provided by an exemplary embodiment of the present invention;

[0065] Figure 7 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0066] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0067] It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0068] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0069] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0070] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0071] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0072] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0073] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0074] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0075] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0076] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0077] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0078] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0079] Exemplary Methods

[0080] Figure 1 is a flow chart of a new energy consumption capacity evaluation method provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the new energy consumption capacity assessment method 100 includes the following steps:

[0081] Step 101, building a deep reinforcement learning model for evaluating new energy consumption capacity;

[0082] Step 102, constructing a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm;

[0083] Step 103, training the deep reinforcement learning model using a deep reinforcement learning framework based on historical new energy output data and load data to generate a new energy consumption capacity assessment intelligent agent;

[0084] Step 104, using the new energy consumption capacity assessment intelligent body to conduct consumption assessment on the new energy output data and load data of the scenario to be assessed, and determine the new energy consumption capacity assessment result of the scenario to be assessed.

[0085] Specifically, the present invention proposes a method for evaluating the new energy consumption capacity based on the Lagrange soft actor-critic (Lag-SAC) deep reinforcement learning framework. Compared with the prior art, the method of the present invention combines the Lagrange multiplier method with the flexible action-evaluation algorithm, does not need to rely on the traditional mixed integer programming model, has high computational efficiency and strong training stability, and can achieve accurate quantitative evaluation of the new energy consumption capacity; by introducing the Lagrangian network, the patented method can effectively solve the conflict between the intelligent agent's immediate reward and the power balance equation constraint, can effectively reduce the system power imbalance, and improve the system's new energy consumption capacity on the basis of ensuring the system power supply balance; in addition, the method of the present invention can take into account the regulation of the energy storage system, optimize the energy storage operation strategy, and increase the new energy consumption by increasing the energy storage call efficiency.

[0086] The present invention relates to a method for evaluating the absorption capacity of new energy based on the Lag-SAC deep reinforcement learning framework. First, a deep reinforcement learning model for evaluating the absorption capacity of new energy is built; then a deep reinforcement learning framework based on the Lag-SAC algorithm is constructed; then the intelligent agent is trained using historical new energy output data and load data to optimize the decision-making ability of the intelligent agent; finally, the absorption capacity of the new energy power system is evaluated using the new energy output data and load data in future scenarios to obtain the results. The main steps are as follows:

[0087] (1) Construction of a deep reinforcement learning model for evaluating new energy consumption capacity

[0088] ① Absorption capacity assessment model

[0089] The absorption capacity assessment model refers to a simulation environment used to assess the absorption capacity of new energy in the power system, including constraints such as system power balance constraints, system reserve constraints, wind and solar output constraints, thermal power unit operation constraints, and energy storage power station operation constraints. Through these constraints, it can be ensured that the operation mode of the power system source, grid, load and storage components can meet the actual physical limitations.

[0090] ②Agent action space

[0091] The action space refers to the set of all possible actions that the agent may take in each operating state, including decision variables such as the output of thermal power units, the number of thermal power units started, and the energy storage charging and discharging power, as shown in Formula 1.1.

[0092] A t = {P g,t ,N t ,P g,c,t ,P g,d,t} (1.1)

[0093] ③Agent state space

[0094] The state space refers to the set of all possible operating states at a certain moment in the power system environment, including daily load demand, available wind and solar power generation, wind and solar power generation grid-connected power, thermal power unit output, number of thermal power units in operation, energy storage power station charge state, action at the previous moment and other state variables, as shown in Formula 1.2.

[0095]

[0096] ④Agent Reward and Punishment Mechanism

[0097] The reward and punishment mechanism is used to determine the feedback obtained by the intelligent agent after taking a certain action in a specific state, so as to achieve the optimization guidance of the intelligent agent's learning behavior. Specifically, it includes the operation cost of thermal power units, the cost of power abandonment and load shedding, the penalty cost of system imbalance, the penalty cost of thermal power ramp-up exceeding the limit, and the penalty cost of energy storage power station charging and discharging power exceeding the limit, as shown in formula 1.3.

[0098] C PES,reward =C PES,TP,op +C PES,cur,op +C imb,pe +C ramp,pe +C ESS,pe (1.3)

[0099] (2) Construction of deep reinforcement learning framework based on Lag-SAC algorithm

[0100] ① Basic theoretical model

[0101] The Lag-SAC algorithm belongs to the policy gradient method based on maximum entropy in deep reinforcement learning. Its objective function is used to explore and find the optimal strategy π*, so as to maximize the expected value of cumulative returns under the maximum entropy framework, as shown in Formula 2.1.

[0102]

[0103] In the formula, r(s t ,a t ) is the immediate reward function; H(π(·|s t )) is state s t Next, take action a t The entropy regularization term; α is the temperature coefficient, which is used to balance the relationship between exploration and utilization. When α is larger, the agent pays more attention to exploration, and when α is smaller, the agent pays more attention to utilization. t is the Lagrange multiplier, c(s t ,a t) is an auxiliary cost function representing the degree of constraint violation. The more serious the constraint violation, the larger the value of the auxiliary cost function. By maximizing the objective function shown in Formula 2.1, a reasonable trade-off between the immediate reward and the degree of constraint violation of the power balance equation can be achieved.

[0104] The Lag-SAC algorithm not only uses the Bellman equation to describe the expected reward value of the intelligent agent taking a certain action in each state, as shown in Formula 2.2, but also further adds an auxiliary cost state value function, thereby effectively taking into account the degree of violation of the power balance equation constraint, as shown in Formula 2.3.

[0105]

[0106] In the formula, Q π (s t ,a t ) is the state-action value function of the agent at time t; C π (s t ,a t ) is the auxiliary cost state-action value function of the agent at time t; γ is the discount factor; is the state-value function of the agent at time t; is the state-auxiliary cost function of the agent at time t+1.

[0107] ②Network module

[0108] The network module includes the policy network, value network, Lagrangian network and target network. The network parameter update process is as follows: Figure 2 The specific instructions are as follows:

[0109] The policy network is used to select the optimal action of the agent in each state. The policy network takes the current system operation state as input and outputs the corresponding action mean and its logarithmic probability. The network parameter update is guided by minimizing the loss function shown in Equation 2.4.

[0110]

[0111] The value network is used to evaluate the expected return of each state-action pair of the agent. The value network takes the current state-action pair (s t ,a t ) as input and output the corresponding Q value. The network parameter update is guided by minimizing the loss function shown in Equation 2.5.

[0112]

[0113] The Lagrangian network adjusts the Lagrangian multiplier in real time to ensure that the system power balance constraint is met as much as possible. The Lagrangian network takes the current state action (s t ,at ) as input and output the corresponding C value. The network parameter update is guided by minimizing the loss function shown in Equation 2.6.

[0114]

[0115] The target network is used to provide a stable target value to reduce fluctuations during training. The target network is the next state-action pair (s t+1 ,a t+1 ) as input and output the target Q value q t and target C value c t , as shown in Equation 2.7 and Equation 2.8. The target network uses a soft update strategy to update the parameters, that is, gradually update the target grid parameters with a smaller learning rate, as shown in Equation 2.9.

[0116]

[0117] θ'=τθ+(1-τ)θ' (2.9)

[0118] In the formula, θ is the current network parameter; θ' is the target network parameter; τ is the soft update learning rate, which is 0.001.

[0119] (3) Agent training process

[0120] The training process of new energy consumption capacity evaluation based on Lag-SAC algorithm is as follows: Figure 3 The specific instructions are as follows:

[0121] Step 1: State initialization. One day is extracted from the historical wind, solar and load sequences as the system operation state P0 at each moment of the current operation day D0, and the unit operation state at moment t0 of the day is taken as the initial state value s0.

[0122] Step 2: Collect experience pairs. During the interaction between the policy network and the system simulation environment, collect a set of "experience pairs" (including state, action, reward, cost, next state and other data), and determine whether the "experience pairs" in the experience replay pool have reached the specified number.

[0123] If the specified number is not reached, the group of "experience number pairs" is stored in the experience replay pool, and the time t=t+1 is set, and then further judge whether t≤24 is established. If so, step 2 is re-executed; if not, the number of days d=d+1 is set, and step 1 is returned. If the specified number is reached, the "experience number pairs" stored at the front of the experience replay pool are deleted, and the latest "experience number pairs" are stored at the end of the experience replay pool, and step 3 is continued.

[0124] Step 3: Neural grid parameter update. First, randomly sample n sets of experience pairs from the experience replay pool. Extract the running state set {s1, s2, …, s n} is used as the input of the policy network, outputs the corresponding action probability distribution parameters, and further samples and selects the current specific action.

[0125] Secondly, the state-action pair (s k ,a k ), k=1,2,…,n are input into the value network Q1, Q2 and the Lagrangian network C1, C2 in sequence, and the corresponding Q values ​​are calculated respectively. and C value Among them, in order to reduce the over-estimation problem, the value network Q is selected i and the Lagrangian network C i The smaller value of is used in subsequent calculations.

[0126] Then, the target value networks TQ1, TQ2 and the target Lagrangian networks TC1, TC2 are used to calculate the target Q value q k With the target C value c k .

[0127] Finally, calculate the loss function J π (θ π ), J Q (θ Q ), J C (θ C ) to update the parameters of the policy network, value network and Lagrangian network. At the same time, the soft update strategy is used to update the parameters of the target network.

[0128] Step 4: Iterate the loop. Determine whether the agent training round has reached the specified number of days. If the number of days The iteration cycle ends; if the number of days Then let time t=t+1, and further determine whether t≤24 holds.

[0129] If yes, return to step 2; if no, set the number of days d=d+1 and return to step 1.

[0130] (4) Using intelligent agents to evaluate consumption

[0131] When using an intelligent agent to evaluate the consumption, the predicted output data of the new energy and the load forecast data of the scenario to be evaluated are first input into the simulation environment, and then the intelligent agent receives the state s from the simulation environment. t , and then select action a according to the optimal strategy π* obtained during training t , and transmit it back to the simulation environment to obtain the next state s t+1 , and so on, until the consumption assessment is completed.

[0132] Figure 4 It is an implementation flow chart of the present invention, and the specific steps are as follows:

[0133] (1) Build a deep reinforcement learning model for evaluating new energy absorption capacity, including absorption capacity evaluation model, intelligent agent action space, intelligent agent state space, and intelligent agent reward and punishment mechanism.

[0134] (2) Construct a deep reinforcement learning framework based on the Lag-SAC algorithm, including policy network, value network, Lagrangian network, and target network.

[0135] (3) Use historical renewable energy output data and load data to train the intelligent agent and save the parameters after the model converges.

[0136] (4) Use the renewable energy output data and load data of the scenario to be evaluated to evaluate the power system and obtain the results.

[0137] (5) Attachment Figure 5 The results of evaluating the renewable energy consumption capacity of a typical power grid in different scenarios using the method of the present invention are presented. The results show that the method proposed by the present invention can quickly adjust the strategy to cope with uncertain operating conditions under different renewable energy output scenarios and energy storage configuration capacities, showing good adaptability.

[0138] Therefore, the present invention proposes a new energy consumption capacity evaluation method based on the Lagrangian flexible action-evaluation deep reinforcement learning framework, which can realize the consumption evaluation of the new energy power system on the basis of taking into account the regulation effect of energy storage. It has the following advantages: the present invention combines the Lagrangian multiplier method with the SAC algorithm, and does not need to process the mixed integer variables when evaluating the new energy consumption capacity, thereby greatly reducing the computational complexity. For the same scenario, the solution time of the method of the present invention is only 3.3% of the solution time of the traditional mathematical programming method, which greatly improves the computational efficiency; when facing a large number of new energy scenarios, the method of the present invention only needs to update the test set data to evaluate the consumption of the new energy power system, and has strong adaptability; compared with deep reinforcement learning algorithms such as SAC and DDPG, the method of the present invention can effectively solve the conflict problem between the immediate reward and the power balance equation constraint of the intelligent body, and the system unbalanced electricity is reduced from 41.3% of the traditional SAC method to 7.5%, thereby improving the new energy consumption capacity on the basis of safe operation of the system; in addition, the method of the present invention can optimize the operation strategy of the energy storage system on the basis of considering the operation constraints related to energy storage, and further improve the utilization efficiency of the energy storage equipment and the new energy consumption capacity of the system. By applying the energy storage system operation strategy of this method, the energy storage utilization rate is increased from 10.6% of the traditional SAC method to 17.8%, while the new energy consumption rate is increased from 69.8% when no energy storage is configured to 88.9% when 1000MW·h energy storage capacity is configured, an increase of 27.4%.

[0139] Exemplary Devices

[0140] Figure 6 Schematic diagram of the structure of a new energy consumption capacity evaluation device provided by an exemplary embodiment of the present invention. Figure 6 As shown, the device 600 includes:

[0141] Building module 610, for building a deep reinforcement learning model for evaluating new energy consumption capacity;

[0142] A construction module 620 is used to construct a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm;

[0143] A generation module 630 is used to train a deep reinforcement learning model using a deep reinforcement learning framework based on historical new energy output data and load data to generate a new energy consumption capacity evaluation agent;

[0144] The determination module 640 is used to use the new energy consumption capacity assessment intelligent body to conduct a consumption assessment on the new energy output data and load data of the scene to be assessed, and determine the new energy consumption capacity assessment result of the scene to be assessed.

[0145] Optionally, building module 610 includes:

[0146] Construct a simulation environment for evaluating the absorption capacity of new energy in the power system. The absorption capacity evaluation model includes: system power balance constraints, system reserve constraints, wind and solar output constraints, thermal power unit operation constraints, and energy storage power station operation constraints.

[0147] Construct an agent action space of all the action sets taken by the agent in each operating state, where the action set includes: thermal power unit output, number of thermal power units in operation, and energy storage charging and discharging power;

[0148] Construct the agent state space of all the operating states of the agent at each moment, where the operating state set includes: daily load demand, available wind and solar power generation, wind and solar power generation grid-connected power, thermal power unit output, number of thermal power units in operation, energy storage power station charge state and the action at the previous moment;

[0149] Construct an agent reward and punishment mechanism to determine the feedback obtained after the agent takes a certain action in a specific state, where the feedback includes: the operating cost of thermal power units, the cost of power abandonment and load shedding, the penalty cost of system imbalance, the penalty cost of thermal power ramp-up exceeding the limit, and the penalty cost of energy storage power station charging and discharging power exceeding the limit;

[0150] A deep reinforcement learning model is constructed based on the absorption capacity evaluation model, agent action space, agent state space and agent reward and punishment mechanism.

[0151] Optionally, the building block 620 includes:

[0152] Construct the basic theoretical model of Lagrangian flexible motion evaluation algorithm;

[0153] Construct a network model of Lagrangian flexible action evaluation algorithm;

[0154] Based on the basic theoretical model and network model, a deep reinforcement learning framework is constructed.

[0155] Optionally, the objective function of the basic theoretical model is:

[0156]

[0157] In the formula, π* is the optimal strategy; r(s t ,a t ) is the immediate reward function; H(π(·|s t )) is state s t Next, take action a t The entropy regularization term; α is the temperature coefficient, λ t is the Lagrange multiplier, c(s t ,a t) is an auxiliary cost function representing the degree of constraint violation.

[0158] Optionally, the network module includes: a policy network, a value network, a Lagrangian network and a target network, wherein

[0159] The loss function of the policy network guiding the network parameter update is:

[0160]

[0161] In the formula, J π (θ π ) indicates that the parameter θ π The loss function under Represents the state s sampled in the experience pool D t And according to the strategy The sampled action a t The expectation under the sampling (s t ,a t ) is the current state action pair, α is the temperature coefficient, λ t is the Lagrange multiplier, Q π (s t ,a t ) is the state-action value function of the agent at time t; is the state-value function of the agent at time t; C π (s t ,a t ) is the auxiliary cost state-action value function of the agent at time t; is the state-auxiliary cost value function of the agent at time t+1; γ is the discount factor;

[0162] The loss function of the value network guiding the network parameter update is:

[0163]

[0164] In the formula, q t is the target Q value;

[0165] The loss function of the Lagrangian network to guide the update of network parameters is:

[0166]

[0167] In the formula, c t is the target C value;

[0168] The loss function of the target network to guide the network parameter update is:

[0169]

[0170] θ'=τθ+(1-τ)θ'

[0171] Where, d t is the termination mark. When the training is finished, d t is 1, otherwise it is 0; (s t+1 ,a t+1 ) is the state-action pair for the next moment; is the target network’s state-action pair for the next moment (s t+1 ,a t+1 )’s estimated Q value; is the instantaneous cost value at time t; is the target network’s state-action pair for the next moment (s t+1 ,a t+1 )’s estimated C value; π θ (a t+1 |s t+1 ) in a given state s t+1 When , the action a is generated by the strategy π (with parameter θ) t+1 The probability of; θ is the current network parameter; θ' is the target network parameter; τ is the soft update learning rate.

[0172] Optionally, the generating module 630 includes:

[0173] Step 1: State initialization: extract one day from the historical wind, solar and load sequences as the system operation state P0 at each moment of the current operation day D0, and use the unit operation state at moment t0 of the day as the initial state value s0;

[0174] Step 2: Collect experience pairs. During the interaction between the policy network and the system simulation environment, collect a new set of experience pairs and determine whether the experience pairs in the experience replay pool have reached the specified number. The experience data pairs include running status, action, reward, cost, and next step status.

[0175] If the specified number is not reached, the new experience number pair is stored in the experience replay pool, and the time t=t+1 is set, and then further judge whether t≤24 is established. If so, step 2 is re-executed; if not, the number of days d=d+1 is set, and step 1 is returned to execute; if the specified number is reached, the experience number pair stored in the front row of the experience replay pool is deleted, and the new experience number pair is stored at the end of the experience replay pool, and step 3 is continued;

[0176] Step 3: Neural grid parameter update: randomly sample n sets of experience pairs from the experience replay pool, and extract the running state set {s1, s2, …, s n} as the input of the policy network, output the corresponding action probability distribution parameters, and further sample and select the current specific action; the state-action pair (sk ,a k ), k=1,2,…,n are input into the value network Q1, value network Q2, Lagrangian network C1 and Lagrangian network C2 in turn, and the corresponding Q values ​​are calculated respectively. and C value The target Q value q is calculated by using the target value network TQ1 and the target value network TQ2 and the target Lagrangian network TC1 and the target Lagrangian network TC2 respectively. k With the target C value c k ; Calculate the loss function J respectively π (θ π ), J Q (θ Q ), J C (θ C ), realize the parameter update of the policy network, value network and Lagrangian network, and use the soft update strategy to realize the parameter update of the target network;

[0177] Step 4: Iterate the loop to determine whether the agent training rounds have reached the specified number of days. The iteration cycle ends and the new energy consumption capacity evaluation intelligent entity is generated; if the number of days Then set the time t=t+1, and further determine whether t≤24 holds; if so, return to step 2; if not, set the number of days d=d+1, and return to step 1.

[0178] Optionally, the determination module 640 includes:

[0179] Input the predicted output data of new energy and load forecast data of the scenario to be evaluated into the simulation environment;

[0180] Using the new energy consumption capacity to evaluate the intelligent agent to accept the state s from the simulation environment t ;

[0181] According to the acceptance status s t And the optimal strategy π* obtained in advance selects action a t ;

[0182] The selected action a t Transmitted back to the simulation environment to get the state s at the next moment t+1 , repeat the selection action until the consumption assessment is completed, and determine the evaluation result of the new energy consumption capacity of the scenario to be evaluated.

[0183] Exemplary Electronic Devices

[0184] Figure 7 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 7As shown, the electronic device 70 includes one or more processors 71 and a memory 72 .

[0185] The processor 71 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0186] The memory 72 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 71 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 73 and an output device 74, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0187] In addition, the input device 73 may also include, for example, a keyboard, a mouse, and the like.

[0188] The output device 74 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0189] Of course, to simplify, Figure 7 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0190] Exemplary computer program products and computer-readable storage media

[0191] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0192] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0193] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0194] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0195] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0196] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0197] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0198] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0199] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0200] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for evaluating the new energy consumption capacity, characterized in that: include: Build a deep reinforcement learning model for evaluating new energy consumption capacity; Construct a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm; Based on historical new energy output data and load data, the deep reinforcement learning model is trained using the deep reinforcement learning framework to generate a new energy consumption capacity assessment intelligent agent; The new energy consumption capacity assessment intelligent body is used to conduct consumption assessment on the new energy output data and load data of the scenario to be assessed, and determine the new energy consumption capacity assessment result of the scenario to be assessed.

2. The method according to claim 1, characterized in that Build a deep reinforcement learning model for evaluating new energy consumption capacity, including: Constructing a simulation environment for evaluating the absorption capacity of new energy in the power system, wherein the absorption capacity evaluation model includes: system power balance constraints, system reserve constraints, wind and solar output constraints, thermal power unit operation constraints, and energy storage power station operation constraints; Constructing an agent action space of all action sets taken by the agent in each operating state, wherein the action set includes: thermal power unit output, the number of thermal power units started, and energy storage charging and discharging power; Construct an agent state space of all the operating state sets of the agent at each moment, wherein the operating state set includes: daily load demand, available wind and solar power generation power, wind and solar power generation grid-connected power, thermal power unit output, number of thermal power units in operation, energy storage power station charge state and the action at the previous moment; Construct an agent reward and punishment mechanism for determining the feedback obtained after the agent takes a certain action in a specific state, wherein the feedback includes: the operating cost of thermal power units, the cost of power abandonment and load shedding, the penalty cost of system imbalance, the penalty cost of thermal power ramp-up exceeding the limit, and the penalty cost of energy storage power station charging and discharging power exceeding the limit; The deep reinforcement learning model is constructed according to the absorption capacity evaluation model, the agent action space, the agent state space and the agent reward and punishment mechanism.

3. The method according to claim 1, characterized in that Construct a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm, including: Construct the basic theoretical model of Lagrangian flexible motion evaluation algorithm; Construct a network model of Lagrangian flexible action evaluation algorithm; The deep reinforcement learning framework is constructed based on the basic theoretical model and the network model.

4. The method according to claim 3, characterized in that The objective function of the basic theoretical model is: In the formula, π * is the optimal strategy; r(s t ,a t ) is the immediate reward function; H( π (·|s t )) is state s t Next, take action a t The entropy regularization term; α is the temperature coefficient, λ t is the Lagrange multiplier, c(s t ,a t ) is an auxiliary cost function representing the degree of constraint violation.

5. The method according to claim 3, characterized in that: The network module includes: a strategy network, a value network, a Lagrangian network and a target network. The loss function of the policy network guiding the network parameter update is: In the formula, J π (θ π ) indicates that the parameter θ π The loss function under Represents the state s sampled in the experience pool D t And according to the strategy The sampled action a t The expectation under the sampling (s t ,a t ) is the current state action pair, α is the temperature coefficient, λ t is the Lagrange multiplier, Q π (s t ,a t ) is the state-action value function of the agent at time t; is the state-value function of the agent at time t; C π (s t ,a t ) is the auxiliary cost state-action value function of the agent at time t; is the state-auxiliary cost value function of the agent at time t+1; γ is the discount factor; The loss function of the value network guiding the network parameter update is: In the formula, q t is the target Q value; The loss function of the Lagrangian network guiding the network parameter update is: In the formula, c t is the target C value; The loss function of the target network guiding the network parameter update is: θ'=τθ+(1-τ)θ' Where, d t is the termination mark. When the training is finished, d t is 1, otherwise it is 0; (s t+1 ,a t+1 ) is the state-action pair for the next moment; is the target network’s state-action pair for the next moment (s t+1 ,a t+1 )’s estimated Q value; is the instantaneous cost value at time t; is the target network’s state-action pair for the next moment (s t+1 ,a t+1 )’s estimated C value; π θ (a t+1 |s t+1 ) in a given state s t+1 When , the action a is generated by the strategy π (with parameter θ) t+1 The probability of; θ is the current network parameter; θ' is the target network parameter; τ is the soft update learning rate.

6. The method according to claim 1, characterized in that Based on the historical new energy output data and load data, the deep reinforcement learning model is trained using the deep reinforcement learning framework to generate a new energy consumption capacity assessment agent, including: Step 1: State initialization: extract one day from the historical wind, solar and load sequences as the system operation state P0 at each moment of the current operation day D0, and use the unit operation state at moment t0 of the day as the initial state value s0; Step 2: Collect experience pairs. During the interaction between the policy network and the system simulation environment, collect a new set of experience pairs and determine whether the experience pairs in the experience replay pool have reached the specified number. The experience data pairs include running status, action, reward, cost, and next step status. If the specified number is not reached, the new experience number pair is stored in the experience replay pool, and the time t=t+1 is set, and then further judge whether t≤24 is established. If so, step 2 is re-executed; if not, the number of days d=d+1 is set, and step 1 is returned to execute; if the specified number is reached, the experience number pair stored in the front row of the experience replay pool is deleted, and the new experience number pair is stored at the end of the experience replay pool, and step 3 is continued; Step 3: Neural grid parameter update: randomly sample n sets of experience pairs from the experience replay pool, and extract the running state set {s1, s2, …, s n } as the input of the policy network, output the corresponding action probability distribution parameters, and further sample and select the current specific action; the state-action pair (s k ,a k ), k=1,2,…,n are input into the value network Q1, value network Q2, Lagrangian network C1 and Lagrangian network C2 in turn, and the corresponding Q values ​​are calculated respectively. and C value The target Q value q is calculated by using the target value network TQ1 and the target value network TQ2 and the target Lagrangian network TC1 and the target Lagrangian network TC2 respectively. k With the target C value c k ; Calculate the loss function J respectively π (θ π ), J Q (θ Q ), J C (θ C ), realize the parameter update of the policy network, value network and Lagrangian network, and use the soft update strategy to realize the parameter update of the target network; Step 4: Iterate the loop to determine whether the agent training rounds have reached the specified number of days. The iteration cycle ends and the new energy consumption capacity evaluation agent is generated; if the number of days Then set the time t=t+1, and further determine whether t≤24 holds; if so, return to step 2; if not, set the number of days d=d+1, and return to step 1.

7. The method according to claim 1, characterized in that The new energy consumption capacity assessment intelligent agent is used to conduct consumption assessment on the new energy output data and load data of the scenario to be assessed, and the new energy consumption capacity assessment result of the scenario to be assessed is determined, including: Input the predicted output data of new energy and load forecast data of the scenario to be evaluated into the simulation environment; The intelligent agent receives the state s from the simulation environment by using the new energy consumption capacity evaluation agent t ; According to the acceptance status s t And the optimal strategy π* obtained in advance selects action a t ; The selected action a t Transmitted back to the simulation environment to obtain the state s at the next moment t+ 1. Repeat the selection action until the consumption evaluation is completed, and determine the evaluation result of the new energy consumption capacity of the scenario to be evaluated.

8. A new energy consumption capacity assessment device, characterized in that: include: Build a module for building a deep reinforcement learning model for evaluating new energy absorption capacity; Building blocks for constructing a deep reinforcement learning framework based on the Lagrangian flexible action evaluation algorithm; A generation module, used to train the deep reinforcement learning model using the deep reinforcement learning framework based on historical new energy output data and load data to generate a new energy consumption capacity evaluation intelligent agent; The determination module is used to use the new energy consumption capacity assessment intelligent body to conduct consumption assessment on the new energy output data and load data of the scene to be assessed, and determine the new energy consumption capacity assessment result of the scene to be assessed.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 7.

Images