Power system dynamic economic dispatching decision-making method and system oriented to reliability improvement

Abstract

The invention discloses a power system dynamic economic dispatching decision-making method and system oriented to reliability improvement. The method comprises the following steps: constructing a Markov decision-making process of dynamic economic dispatching; constructing a reinforcement learning environment; updating the neural network of distributed reinforcement learning; obtaining feasible power output under the constraint condition; obtaining the total power generation cost through an average consensus algorithm; a greedy strategy epsilon-greedy algorithm is introduced, and the intelligent agent is utilized to maximize a reward which can be obtained currently, so that the intelligent agent is ensured to learn an optimal strategy in a complex environment; and defining # imgabs0 # as a historical optimal strategy, and updating a local operation strategy, so that the optimal scheduling strategy is monotonically decreased, and finally, the optimal scheduling strategy is achieved. The method has high robustness under the condition of fluctuating power demand.

Description

Background Art

[0002] In the power system, the Economic Dispatch Problem (EDP) is of great significance. Its main task is to seek the optimal distribution of power generation under various constraints to minimize the total power generation cost. The main methods to solve EDP are classical optimization algorithms, intelligent optimization algorithms and reinforcement learning. Many classical optimization algorithms are only applicable to convex optimization problems, while EDP is difficult to meet the convex requirements in mathematical modeling. Although intelligent optimization algorithms can handle more complex situations, they converge slowly and are prone to fall into local optimal solutions. They can usually only find static optimal solutions with low efficiency and weak robustness. In contrast, reinforcement learning focuses on developing flexible strategies in unknown environments, adapting to dynamic changes through continuous exploration and learning, and providing more effective solutions. Depending on whether a single time period is considered, EDP can be divided into dynamic economic dispatch problems (DEDP) and static economic dispatch problems (SEDP). SEDP only considers economic dispatch within a certain time period. It belongs to the optimization problem of studying a specific time period on a single time section, ignoring the connection between the dynamic characteristics of the power system and each time period, and it is difficult to cope with the volatility brought by renewable energy generation. DEDP makes full use of the connections between the various characteristic quantities of the power system at each time section to better cope with the volatility on the load side and the randomness on the source side caused by the access of new energy.

[0003] Although existing research uses distributed reinforcement learning algorithms to solve dynamic economic scheduling problems, it does not consider the problem of quickly deciding the most economical generator output and achieving active power output balance in uncertain scenarios with a high proportion of new energy. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a dynamic economic dispatch decision method and system for power system with improved reliability in view of the deficiencies in the above-mentioned prior art, so as to solve the technical problem of rapid dispatch of power system under strong uncertain source and load fluctuations.

[0005] The present invention adopts the following technical solutions:

[0006] A method for dynamic economic dispatch decision-making of a power system for improving reliability includes the following steps:

[0007] S1. Construct a Markov decision process for dynamic economic scheduling, define the state transition probability through the Markov chain, and use it to calculate the strategy and value function of reinforcement learning;

[0008] S2. Construct a reinforcement learning environment to provide simulation scenarios for the power system generator set agent. The power system generator set agent makes scheduling decisions based on the system status and schedules the generator set to minimize the total power generation cost.

[0009] S3, updating the neural network of distributed reinforcement learning, and the power system generator set agent adjusts its strategy according to environmental feedback;

[0010] S4, obtaining feasible power output under constraint conditions;

[0011] S5. Obtain the total cost of power generation through the average consensus algorithm, and return the total cost of power generation to the power system generator agent as a reward for reinforcement learning;

[0012] S6. Introduce the greedy strategy ε-greedy algorithm, use the power system generator set agent to maximize the current reward that can be obtained, and ensure that the power system generator set agent learns the optimal strategy in a complex environment;

[0013] S7. Definition The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

[0014] Preferably, in step S1, a multi-agent Markov decision process for designing dynamic economic dispatch of a high proportion of new energy system is defined as follows:

[0015]

[0016] in, is a global state space shared by all agents, set to the maximum output of new energy With total load D t , m is the number of new energy sources; is the action space of agent i, which is designed as the active output p of the generator set i,t ; is the action reward value of agent i; γ∈[0,1] is the discount factor. In multi-agent decision-making, the agent obtains information about the global state space. Predict the action strategy of local agent i Get local rewards at the same time

[0017] Preferably, in step S2, the reinforcement learning environment includes a state space s t , generator set output constraints;

[0018] Each unit agent estimates the total power demand through the average consensus algorithm. The consensus algorithm for the total demand is expressed as:

[0019]

[0020] Among them, L is the graph The Laplace matrix of the communication topology between central nodes, Among them, max i l ii is the maximum value of the diagonal elements of the Laplace matrix L, is the average power obtained in the kth iteration;

[0021] The consensus protocol yields:

[0022]

[0023] Among them, 1 N is an N-dimensional column vector, the matrix elements are 1, D t is the total load value of the power grid at time t, and N is the total number of power grid nodes;

[0024] definition To represent the communication topology degree estimated by the central node i, Each central node obtains the same ∈ through a distributed algorithm;

[0025] After N iterations, there exists Equal to the degree of the communication topology.

[0026] Preferably, step S3 specifically includes:

[0027] S301. Design a dual neural network consisting of an action network and an evaluation network. The action network is responsible for the output action to be taken under a given grid state; the evaluation network evaluates the expected cost of the system after taking a certain action. Define As the state-action value function under the policy action π at time slot t, the reinforcement learning value network estimation is established, ω t is the parameter of the network, and the mean square error with the target network is used as the loss function to update ω t Parameters, obtain the minimum value of the state action value function to obtain an approximate solution for economic dispatch; use a distributed method to obtain the minimum value of the state action value function Solve problems about approximate state-value functions;

[0028] S302, each generator set's agent has an action network Using parameters Approximate optimal policy; Select a fully connected multilayer perceptron with two hidden layers to approximate the optimal policy; Reinforcement learning state Actions with Agents are the input and output of the action network respectively; along Relative to The negative gradient is updated.

[0029] Preferably, the loss function is defined as:

[0030]

[0031] in, Refers to the expected value, γ is the discount factor, is the estimated value of the action-value function of the current state;

[0032] Update t :

[0033]

[0034] in, is the learning rate; C t Indicates that in state s t The minimum total cost from time t to t+1; yes The derivative with respect to t;

[0035] The action network of generator unit i is expressed as:

[0036]

[0037] in, is the weight matrix, is the bias vector, h1 and h2 are the number of neurons in the two hidden layers; is a non-linear activation function.

[0038] Preferably, step S4 is specifically:

[0039] S401, each intelligent agent decides the power output of each generator Take samples;

[0040] S402, adding a projection controller to the power output distribution; if the constraints are not met, a joint action that meets the constraints is obtained through projection, and the problem is transformed into an optimization problem, which is solved by a projection optimization algorithm. If the communication topology is undirected and connected, the equilibrium point of this formula is the optimal solution to the optimization problem.

[0041] Preferably, in step S5, define and is the local estimate of the average electricity cost at time t, and c i,t =F(p i,t );

[0042] Applying the average consensus algorithm, the time costs satisfy:

[0043]

[0044] get The local estimated generation cost of the average value converges to the actual average generation cost at time slot t, and the total generation cost is Nc t .

[0045] Preferably, step S6 specifically includes:

[0046] Introduce the greedy strategy ε-greedy algorithm, the probability of the optimal action is 1-ε, and the probability of random exploration action is ε; design exploration strategy For the current k-th training; when the number of explorations k≤K0, the agent's decision power is determined by:

[0047]

[0048] When the number of explorations k>K0, the agent's decision-making power is determined by:

[0049]

[0050] in, Indicates in s t The probability of choosing action a;

[0051] Initially, the agent's decision action is given by the evaluation network, which has added some noise inside the algorithm before outputting the action, and adopts a greedy strategy to ensure exploration performance;

[0052] After the training set, the actor network gives a deterministic decision.

[0053] Preferably, in step S7, it is assumed that p i,t An action performed in a real environment; definition:

[0054]

[0055] Update the local operation policy:

[0056]

[0057] in, is the optimal action generated by the evaluation network; p i,t =p i ' ,t It is an exploration of the interaction between action and real power grid environment. It is the optimal scheduling record during the training process; or p i,t When the decision output combination is more economical, the optimal scheduling strategy Updated to the best.

[0058] In a second aspect, an embodiment of the present invention provides a power system dynamic economic dispatch decision system for improving reliability, including:

[0059] Construct modules to build the Markov decision process of dynamic economic dispatch, define the state transition probability through the Markov chain, and use it to calculate the strategy and value function of reinforcement learning; build a reinforcement learning environment to provide simulation scenarios for the power system generator set agent, and the power system generator set agent makes dispatch decisions based on the system status and dispatches the generator set to minimize the total power generation cost;

[0060] Update module, update the neural network of distributed reinforcement learning, and the power system generator set agent adjusts the strategy according to the environmental feedback to obtain the feasible power output under the constraint conditions;

[0061] The calculation module obtains the total cost of power generation through the average consensus algorithm and returns the total cost of power generation to the power system generator agent as a reward for reinforcement learning;

[0062] The strategy module introduces the greedy strategy ε-greedy algorithm, which uses the power system generator set agent to maximize the current reward that can be obtained, ensuring that the power system generator set agent learns the optimal strategy in a complex environment;

[0063] Output module, definition The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

[0064] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for dynamic economic dispatch decision-making of power system for improving reliability when executing the computer program.

[0065] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for dynamic economic dispatch decision-making of a power system for improving reliability.

[0066] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for dynamic economic dispatch decision-making of power system for improving reliability when executing the computer program.

[0067] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned method for dynamic economic dispatch decision-making of power system for improving reliability.

[0068] Compared with the prior art, the present invention has at least the following beneficial effects:

[0069] A dynamic economic dispatch decision method for power system with improved reliability can better and more quickly cope with the volatility and randomness brought by high proportion of renewable energy generation, and minimize the power generation cost under the premise of ensuring load supply. In the design process, the advantages of reinforcement learning and optimization algorithms are used to the greatest extent to coordinate the output of each generator set; the specific work is as follows: the optimal feasible power output is gradually obtained by reinforcement learning training. After the training process, when the total power demand fluctuates or changes, the algorithm can provide a feasible power output distribution of the generator set through the neural network (NN) prediction and projection optimization algorithm, so as to effectively absorb the renewable energy. In addition, the update of the strategy depends not only on the optimal solution of the approximate state action value function, but also on the optimal strategy previously explored. This method ensures that the decision cost decreases monotonically.

[0070] Furthermore, a Markov decision process for dynamic economic dispatch is designed to describe how the agent interacts with the environment and learns the optimal strategy. This step clarifies that the agent's goal is to minimize the cost of power generation, the decision content is the generator output, and the environment is the maximum output of new energy and the total cost required by the power grid.

[0071] Furthermore, the method of obtaining each element required for reinforcement learning is described in detail. In the distributed power system, the total load is obtained by the average consensus algorithm, thus forming the state space of reinforcement learning.

[0072] Furthermore, for dynamic economic dispatch problems, the design of dual neural networks is of great significance. The action network is responsible for generating output decisions under a given grid state. This design enables the system to respond quickly to real-time changes in grid states, such as load fluctuations and fluctuations in new energy output, to ensure the stability and economy of grid operation. The evaluation network is used to evaluate the expected future costs or returns of the grid system after a decision is made, thereby effectively evaluating the long-term effects of each action.

[0073] Furthermore, the projection constraint algorithm can be used to effectively map the decision output of reinforcement learning to a feasible safety constraint range. The method of the present invention ensures that the intelligent agent not only pursues the optimization goal when making control decisions, but also complies with the physical and safety constraints of the system.

[0074] Furthermore, the consensus method is used to obtain the total cost of the generator set, ensuring that each agent aims to minimize the total cost.

[0075] Furthermore, a greedy strategy is used to add random fluctuations after each reinforcement learning decision to prevent the generator set from converging to a local optimal solution.

[0076] Furthermore, the update of the local operation strategy depends on the historical optimal strategy to ensure that the total cost decreases monotonically. This technology stores the optimal strategy in the experience set, which will indirectly guide the learning process, thereby improving the training efficiency of reinforcement learning and accelerating convergence.

[0077] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0078] In summary, the present invention can quickly make an optimal output power prediction and improve the decision-making efficiency of the generator in the face of strong uncertainty in the output of new energy units and the power demand on the load side.

[0079] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is the topological diagram of information exchange of 4 generator nodes;

[0081] Figure 2 It is a topological diagram of information exchange of 14 load nodes;

[0082] Figure 3 This is a graph showing the evolution of the total cost of the agent’s decisions during training;

[0083] Figure 4 The graphs (1-6) show the evolution of the crew strategy at each moment as the training progresses;

[0084] Figure 5 The diagram (7-12) shows the evolution of crew strategy at each moment as training progresses;

[0085] Figure 6 This is the new energy consumption situation (12h) diagram;

[0086] Figure 7 This is the new energy consumption situation (48h) diagram;

[0087] Figure 8 This is a graph showing the evolution of strategy cost with training at different exploration rates;

[0088] Fig. 9 This is the robustness analysis diagram;

[0089] Fig.10 A schematic diagram of a computer device provided by an embodiment of the present invention;

[0090] Fig.11 A block diagram of a chip provided according to an embodiment of the present invention;

[0091] Fig.12It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0094] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0095] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0096] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0097] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0098] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0099] The key to solving the DEDP of a high-proportion renewable energy power system is to find the optimal power distribution of generator sets, so as to fully and effectively absorb the volatility of renewable energy while minimizing the total power generation cost. Dynamic economic dispatch can be described as:

[0100]

[0101] Among them, F i (·) is the power generation cost function of unit i; γ t is the discount factor at time t and γ∈(0,1); p i,t is the power output of generator set i at time t; D t is the total power demand over time; The ramp rate limit of thermal power unit i; p i and are the minimum and maximum power outputs of the generator sets, respectively. The introduction of the discount factor γ∈(0,1) allows the definition of the economic dispatch problem in infinite time and a unified expression form; it also corresponds to the discount factor used to measure future rewards in reinforcement learning, making it easier to apply reinforcement learning theory.

[0102] make is the allowable power output setting of thermal power unit i, then the constraint condition is simplified to:

[0103]

[0104] The power generation cost function can be expressed in various forms, including linear function, logarithmic function, etc. The most common cost function is a quadratic function, such as:

[0105]

[0106] Among them, a i ,b i and c i are some coefficients of agent i. For new energy units, without considering operation and maintenance costs, the power generation cost can be regarded as 0.

[0107] In the problem of dynamic economic dispatch, the operating cost of the power system is usually determined by multiple factors, including fuel costs, maintenance costs, and environmental costs. Traditional optimization methods usually rely on known cost function forms and assume that these functions are convex functions. However, in practical applications, the actual changes in power generation costs may be complex and nonlinear, and it is difficult to accurately describe them with simple convex functions. Therefore, it is necessary to consider more complex cost function forms, such as a combination of quadratic functions and trigonometric functions, to more accurately reflect the changes in actual power generation costs. The present invention considers a more complex cost function in order to better fit the power generation cost, namely:

[0108]

[0109] Among them, e i With f i It is related to the capacity of the thermal power generator set. This function is designed to be the goal of this model solution and guide the actions of the intelligent agent.

[0110] In order to ensure the effective supply of load and better solve the economic dispatch model, the load satisfaction formula is set as follows:

[0111]

[0112] By designing a complex cost function, the actual changes in power generation costs can be better fitted, and the accuracy and practicality of the dynamic economic dispatch model can be improved. In view of the problem DEDP required to be solved in this invention, the following assumptions are proposed:

[0113] Feasibility and optimality assumptions:

[0114] At each moment in the considered time period, there is at least one feasible power output combination (FPOC) that satisfies the constraints: 1,1 ,...,p N,1 ,...,p 1,T ,...,p N,1 ) T satisfy:

[0115]

[0116] Topological connectivity assumptions:

[0117] Each unit can be regarded as an intelligent agent. The topological structure of the information exchange diagram of these intelligent agents and the information exchange diagram of the main power grid is undirected and interconnected, ensuring that the information between the intelligent agents can be exchanged with each other to achieve the goal of minimizing the overall cost. At each time t, each main power grid can obtain its own power demand and the total power demand D t; Each agent i can access the previously obtained total power demand D of the power grid t , adjust the local power output p i,t , and obtain the local power generation cost F i (p i,t ), the total power generation cost can be obtained through information exchange.

[0118] See also Fig.12 The present invention provides a power system dynamic economic dispatch decision method for improving reliability, comprising the following steps:

[0119] S1. Constructing the Markov decision process for dynamic economic dispatch;

[0120] Dynamic economic dispatch can be regarded as a Markov decision process (MDP). The goal of economic dispatch is to find the minimum cost of the output combination within a period of time, so the output of the unit needs to be used as a decision variable, and each unit with output is regarded as an intelligent agent. The dynamic economic dispatch problem (DEDP) can be described as a system decision MDP. The present invention will design a multi-agent Markov decision process (Distributed Multiagent MDP, MAMDP) for dynamic economic dispatch of a high-proportion new energy system.

[0121] Define the MAMDP of DEDP as a five-tuple:

[0122]

[0123] in, is a global state space shared by all agents, set to the maximum output of new energy With total load D t , to ensure full utilization of new energy, that is m is the number of new energy sources; is the action space of agent i, which is designed as the active output p of the generator set i,t ; is the action reward value of agent i, which is considered to be the total power generation cost; γ∈[0,1] is the discount factor. In this invention, it is considered that the power generation cost in the future period is as important as the current power generation cost, so let γ=1.

[0124] In multi-agent decision making, the overall action space is the superposition of the action spaces of each agent, so Also reward In the state space Execute the action of agent i The local reward obtained, therefore, the agent obtains information about the global state space Predict the action strategy of local agent i Get local rewards at the same time

[0125] S2, build a reinforcement learning environment;

[0126] The environment of power system reinforcement learning is the basic requirements and various information of the power grid, including the state space s t (The maximum output of new energy generator set P j,t With total load D t , that is, ie,s t =<P j,t ,D t >,j=1,2,...,m), generator output constraints. Among them, the maximum output of each new energy unit is directly transmitted, and the total load demand needs to be obtained using the average consensus algorithm.

[0127] Since the communication topology is undirected and interconnected, each unit agent estimates the total power demand through an average consensus algorithm. in, The i-th term of represents the average local power demand of all units estimated on each main line in the grid topology.

[0128] The consensus algorithm for total demand is expressed as:

[0129]

[0130] Among them, L is the graph The Laplace matrix of the communication topology between central nodes,

[0131] From the consensus protocol, we get:

[0132]

[0133] Among them, 1 N is an N-dimensional column vector whose elements are all 1.

[0134] Therefore, the local estimate of the average power demand converges to the actual average power demand.

[0135] definition To represent the communication topology degree estimated by the central node i, Each central node obtains the same ∈ through a distributed algorithm.

[0136]

[0137] Among them, a i,j are the entries of the adjacency matrix of the communication topology.

[0138] After N iterations, there exists in, Equal to the degree of the communication topology.

[0139] S3, update the neural network of distributed reinforcement learning;

[0140] The design consists of a dual neural network consisting of an action network and an evaluation network. The action network is responsible for generating decisions, that is, the output action that should be taken under a given grid state; while the evaluation network evaluates the expected cost of the system after taking a certain action. This separate architecture allows the system to make decisions quickly during execution and effectively evaluate the long-term effects of decisions during the training phase.

[0141] S301, update evaluation network

[0142] The evaluation network is used to update the state action value function. Definition As the state-action value function under the policy action π at time slot t. Establish a reinforcement learning value network estimate, where ω t is the parameter of the network. The mean square error with the target network is used as the loss function to update ω t parameter.

[0143] The loss function is defined as:

[0144]

[0145] Among them, P t i is the set of active outputs of all thermal power units except agent i, that is, P t =<P1,...,P n >. To minimize the loss function, update ω t :

[0146]

[0147] in, is the learning rate; C t Indicates that in state s t The minimum total cost from time t to t+1; It's Q t (s t ,P t i ;w t ) with respect to t.

[0148] To improve the stability of reinforcement learning, ensure that the parameter ω t Slow Update:

[0149] ω t '←τω t +(1-τ)ω t '

[0150] in, is the feature vector. Accordingly, the design and evaluation of the neural network parameters definition:

[0151]

[0152] Where i = 1, 2, ..., N, N is the total number of agents, then the state-action value function can be obtained to satisfy:

[0153]

[0154] Obtain the minimum value of the state action value function to obtain an approximate solution for economic dispatch. Obtain the minimum value of the state action value function using a distributed method That is to say, it solves the problem of approximate state-value function.

[0155]

[0156] Among them, for each unit agent, there is

[0157] make The problem of representation can be solved under the following standard assumptions.

[0158] Assumption 3: There is a finite optimal solution P for this optimization problem t a* , which satisfies Slater's constraint condition, that is, there exists Make For the optimization problem to be solved, have:

[0159]

[0160] Using the Laplace matrix L and Set up the constrained optimization problem.

[0161]

[0162] stLΛ=0 N

[0163] With multiplier The augmented Lagrangian duality of is:

[0164]

[0165] The distributed algorithm of agent i is expressed as:

[0166]

[0167] Based on the Karush-Kuhn-Tucker (KKT) condition, the equilibrium point of this equation is the optimal solution to the minimum problem. t a* ,Λ a* ,Z a* ) is expressed as the vector P t a* ,Λ a* ,Z a* The stacked column vectors can then be obtained The value of .

[0168] S302, for each generator set agent, there is an action network Using parameters To approximate the optimal policy; a fully connected multilayer perceptron (MLP) with two hidden layers is selected to approximate the optimal policy, and the action network of generator unit i is expressed as:

[0169]

[0170] in, is the weight matrix, is the bias vector, h1 and h2 are the number of neurons in the two hidden layers; is a nonlinear activation function; reinforcement learning state Actions with Agents are the input and output of the action network respectively; the weight matrix and bias vector are the parameters that need to be fitted. Define the J function, whose gradient is expressed as:

[0171]

[0172] In order to minimize the approximation error, Along Relative to The negative gradient is updated, that is:

[0173]

[0174] The present invention designs an action network and uses a neural network to approximate the optimal strategy of each unit, thereby improving the generalization ability of the strategy. t and θ t All have the subscript t, indicating that data from different times are trained separately. The value functions at different times may be different, so data from different time slots are trained separately. Similarly, since the optimal strategies at different time periods are different, the action networks at different time periods also need to be trained separately.

[0175] S4, obtaining feasible power output under constraint conditions;

[0176] S401, each intelligent agent decides the power output of each generator Take samples;

[0177] S402. Power output distribution due to direct decision If the load constraints, power upper and lower limit constraints and ramp constraints required by the power grid are not met, a projection controller is added to the power output distribution before interacting with the real environment of the power grid; if the constraints are not met, the joint action that meets the constraints is obtained through projection.

[0178]

[0179] Among them, a t =(a 1,t ,...,a N,t ) T It is the joint action after projection.

[0180] The equivalent replacement objective function is:

[0181]

[0182] Then the problem is transformed into an optimization problem, as follows:

[0183]

[0184] Among them, x i is the local decision variable determined by agent i; h i (x i ) is a locally convex function; d i is a local constant.

[0185] Use projection optimization algorithm to solve the problem:

[0186]

[0187] Among them, λ i and z i is an auxiliary variable; Yes i (x i ) relative to x i The derivative of is to project a vector onto Ω i The projection operator on .

[0188] If the communication topology is undirected and connected, the equilibrium point of this equation is the optimal solution to the optimization problem. The optimal solution for the unit output is expressed as:

[0189]

[0190] Among them, λ i and z i is an auxiliary variable, is the projection operator; the optimal solution for adjusting the unit output is P t ′=(p′ 1,t ,...,p′ N,t ) T . i,t The equilibrium point is expressed as

[0191] S5. Obtain the total cost of power generation through the average consensus algorithm;

[0192] The reward function of economic dispatch is set to consider the total power generation cost of each generator unit in the period. After each generator agent makes a power output decision, it returns the local power generation cost. In order to minimize the total power generation cost, each agent needs to communicate with each other to obtain the total power generation cost of the power grid, and at the same time set their respective reward functions to the total cost to ensure that the decision-making goal of each agent is not limited to its own cost.

[0193] definition and is a local estimate of the average electricity generation cost at time t, where c i,t =F(p i,t ). Applying the average consensus algorithm, the time costs satisfy:

[0194]

[0195] get The local estimated generation cost of the average value converges to the actual average generation cost at time slot t. Therefore, the total generation cost is Nc t .

[0196] S6, introduce the greedy strategy ε-greedy algorithm to balance "exploitation" and "exploration";

[0197] The decision of the generator unit agent is to make output decisions based on the original experience. Through utilization, the agent can maximize the current reward that can be obtained, and to a certain extent, ensure the stability and efficiency of the algorithm, and continuously explore possible optimal solutions. In the basic distributed multi-agent deep deterministic policy gradient algorithm framework, random perturbations are only added to the decision actions of the evaluation network, but there are no random perturbations in the action network decision. This will make the decision of the power grid generator unit easy to converge to the local optimal solution and difficult to explore a better solution. Therefore, it is necessary to reasonably balance utilization and exploration to ensure that the agent can learn the optimal strategy in a complex environment. Therefore, the greedy strategy ε-greedy algorithm is introduced to balance "utilization" and "exploration", where the probability of the optimal action is 1-ε, and the probability of random exploration actions is ε. Design exploration strategy For the current k-th training, when the number of explorations k≤K0, the agent’s decision power is determined by:

[0198]

[0199] When the number of explorations k>K0, the decision-making power of the agent is determined by:

[0200]

[0201] in, Indicates in s t The probability of choosing action a.

[0202] Initially, the agent's decision action is given by the evaluation network, which has added some noise inside the algorithm before outputting the action, and adopts a greedy strategy to ensure exploration performance;

[0203] After the training set, the action network directly gives a deterministic decision, adopting a greedy strategy in the hope that the algorithm will provide a better exploration decision.

[0204] S7. To help the action network make better decisions and obtain the most economical actions, define The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

[0205] Let p i,t An action performed in a real environment; definition:

[0206]

[0207] Update the local operation policy:

[0208]

[0209] in, is the optimal action generated by the evaluation network; at the same time, p i,t =p i ' ,t It is an interaction between an exploration action and the real grid environment, and it is the power output obtained by the decision made by the neural network after the ε-greedy strategy; is the optimal scheduling record during the training process; only or p i,t When the decision output combination is more economical, the optimal scheduling strategy This strategy also ensures that the optimal scheduling strategy decreases monotonically and eventually reaches the optimal state.

[0210] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0211] In another embodiment of the present invention, a dynamic economic dispatch decision system for an electric power system for improving reliability is provided. The system can be used to implement the above-mentioned dynamic economic dispatch decision method for an electric power system for improving reliability. Specifically, the dynamic economic dispatch decision system for an electric power system for improving reliability includes a construction module, an update module, a calculation module, a strategy module and an output module.

[0212] Among them, the construction module constructs the Markov decision process of dynamic economic dispatch, defines the state transition probability through the Markov chain, and is used to calculate the strategy and value function of reinforcement learning; the reinforcement learning environment is constructed to provide a simulation scenario for the power system generator set intelligent agent, and the power system generator set intelligent agent makes dispatch decisions according to the system state, and dispatches the generator set to minimize the total power generation cost;

[0213] Update module, update the neural network of distributed reinforcement learning, and the power system generator set agent adjusts the strategy according to the environmental feedback to obtain the feasible power output under the constraint conditions;

[0214] The calculation module obtains the total cost of power generation through the average consensus algorithm and returns the total cost of power generation to the power system generator agent as a reward for reinforcement learning;

[0215] The strategy module introduces the greedy strategy ε-greedy algorithm, which uses the power system generator set agent to maximize the current reward that can be obtained, ensuring that the power system generator set agent learns the optimal strategy in a complex environment;

[0216] Output module, definition The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

[0217] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the dynamic economic dispatch decision method for power system with improved reliability, including:

[0218] Construct a Markov decision process for dynamic economic dispatch, define the state transition probability through the Markov chain, and use it to calculate the strategy and value function of reinforcement learning; construct a reinforcement learning environment to provide a simulation scenario for the power system generator set agent, and the power system generator set agent makes scheduling decisions based on the system state, and dispatches the generator set to minimize the total power generation cost; update the distributed reinforcement learning neural network, and the power system generator set agent adjusts the strategy according to the environmental feedback; obtain the feasible power output under the constraints; obtain the total power generation cost through the average consensus algorithm, and return the total power generation cost to the power system generator set agent as a reinforcement learning reward; introduce the greedy strategy ε-greedy algorithm, and use the power system generator set agent to maximize the current reward that can be obtained to ensure that the power system generator set agent learns the optimal strategy in a complex environment; define The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

[0219] See also Fig.10, the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the power system dynamic economic dispatch decision method for improving reliability in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the power system dynamic economic dispatch decision system for improving reliability in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0220] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Fig.10 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0221] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0222] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0223] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0224] See also Fig.11 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0225] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Fig.12 Follow the steps shown in .

[0226] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0227] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0228] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0229] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0230] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (Non-VolatileMemory), such as at least one disk memory.

[0231] One or more instructions stored in a computer-readable storage medium may be loaded and executed by a processor to implement the corresponding steps of the power system dynamic economic dispatch decision method for reliability improvement in the above embodiment; one or more instructions in the computer-readable storage medium are loaded and executed by the processor as follows:

[0232] Construct a Markov decision process for dynamic economic dispatch, define the state transition probability through the Markov chain, and use it to calculate the strategy and value function of reinforcement learning; construct a reinforcement learning environment to provide a simulation scenario for the power system generator set agent, and the power system generator set agent makes scheduling decisions based on the system state, and dispatches the generator set to minimize the total power generation cost; update the distributed reinforcement learning neural network, and the power system generator set agent adjusts the strategy according to the environmental feedback; obtain the feasible power output under the constraints; obtain the total power generation cost through the average consensus algorithm, and return the total power generation cost to the power system generator set agent as a reinforcement learning reward; introduce the greedy strategy ε-greedy algorithm, and use the power system generator set agent to maximize the current reward that can be obtained to ensure that the power system generator set agent learns the optimal strategy in a complex environment; define The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

[0233] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0234] Construct two neural network frameworks, action network and evaluation network. When k≤K0, select the action decided by the action network. Otherwise, choose to evaluate the network decision action In order to balance the exploration of new strategies and make full use of the original experience, the greedy strategy is used to update the original power. However, this decision action may not meet the required load constraints, power upper and lower limits, and power climbing constraints. Therefore, the constrained projection algorithm is used to adjust the power. When there is a new energy unit intelligent agent, the projection optimization algorithm is improved to:

[0235]

[0236] Among them, m is the number of new energy units, n is the total number of units, is the maximum output of the new energy unit i at time t. Set To ensure the consumption of new energy.

[0237] If k>K00 , the quadratic function coefficients of the evaluation network are output to the state-action value function optimizer to obtain another set of power output distributions, so that the state-action value function value is minimized. The costs of these two sets of unit combinations are compared, and the smaller unit combination output and its state transfer variables are selected and stored in the experience set for subsequent use of experience. 00 When the learning process is repeated, a small part of the data set is extracted from the experience set to update the parameters of the neural network to ensure that the decision-making of the reinforcement learning agent gradually converges to the optimal result.

[0238] Example Consider a system consisting of four power generation units, where the communication topology is shown in the figure. Figure 1 , 1 is a photovoltaic generator, 2 is a wind generator, 3 and 4 are thermal power generators G1 and G2. In this section, we consider using the IEEE14 standard topology that lacks one unit to average the total load required by the power grid. The topology is as follows Figure 2 As shown, PVG represents photovoltaic generator set, WPG represents wind generator set, and TPG represents thermal power generator set.

[0239] In the experiment, a total of 12 time periods are set, one time period every 2 hours, and the total power demand corresponding to each time period is set to 359, 350.8, 374.9, 399.3, 426.9, 446.1, 449.0, 491.5, 421.9, 404.0, 363.4, and 352.6 MW respectively. The total number of time slots is set to T = 12. The exploration rate is set to ε = 0.2. The main parameters of the thermal power unit are shown in Table 1:

[0240] Table 1 Main parameters of the thermal power unit in the example

[0241]

[0242] The selection of DEDP parameters is shown in Table 2. The selection of parameters is simplified in order to improve the calculation efficiency.

[0243] Table 2 Main parameters of reinforcement learning

[0244]

[0245] After 1,000 trainings, the new energy units were fully powered in each time period, and the decisions of the thermal power units are shown in Table 3:

[0246] Table 3 Decision-making of thermal power units

[0247]

[0248] Table 3 (Continued) Decision-making of thermal power units (Continued)

[0249]

[0250] The entire training process is shown in Figure *. After 130 trainings, the total cost is relatively stable, and the corresponding total power generation cost is 32,098.78. The cost function of the example is a nonlinear function with a relatively complex form. The MILP algorithm with piecewise linearized cost is used as the control group. After relaxing the start-stop constraints, the cost calculated by the strategy is 30,654.38, which is 4.71% different, and the error is small. At this time, the new energy has been completely absorbed, and the action decision of the intelligent agent is easy to explore near this better solution, and it is difficult to find a better solution. It can be inferred that when the training step size is increased, a better strategy will be tried, and the error will be further reduced. Although the accuracy of reinforcement learning has not reached the optimization algorithm, it can make decisions quickly when dealing with new situations and reduce time costs.

[0251] See also Figure 3 , as the exploration process of the calculation method of the present invention gradually deepens, the cost of the strategy changes. In addition to the algorithm designed by the present invention, Algorithm 1 is also used for comparison. Algorithm 1 is obtained by other optimization methods such as the original MADDPG algorithm, the optimization of embedded bias adjustment, the Actor-Critic neural network with 40 neurons and 2 hidden layers, and the greedy strategy without ε-greedy. The comparison between the two shows that the algorithm designed by the present invention converges faster and is better. And during the execution of the algorithm, it is not difficult to find that the optimization method of bias adjustment is difficult to achieve the bias σ = 0, and the decision made may not meet the load demand D t , proving the effectiveness of the constrained projection algorithm. To better demonstrate the evolution of the strategy during training, Figure 4 and Figure 5 , showing how the output of thermal power units changes with the training process at each moment.

[0252] Make a line graph of load demand and maximum output of renewable energy, and a stacked column graph of output of thermal power units, such as Figure 6 As shown in the figure, the algorithm analyzes the consumption of new energy. It can be found that in the calculation example, the output decision of the thermal power unit fully adapts to the fluctuations of load and new energy, and completes the task of absorbing a high proportion of new energy system. In order to more clearly show the consumption of new energy by this algorithm, the final output result is made with a step length of 48 hours and each hour as a period of time. Figure 7 The figure can more clearly show the algorithm's absorption of new energy units.

[0253] In order to consider the impact of different ε greedy strategies on the results of this algorithm, the present invention adopts ε = 1 / k, ε = 0.2, ε = 0.4, ε = 0.6, ε = 0.8 and ε = 0.9 × 0.99 k And ε ≥ 0.1, where k is the number of training steps, so that the operation strategy in the ε greedy strategy has an infinite exploration greedy limit. Figure 8 As shown, Figure 8 The evolution of the total generation cost for 1,000 training runs for each updated policy is shown. The distributed reinforcement learning optimization algorithm converges fastest when ε = 0.2, yielding a good policy.

[0254] Next, the robustness of the proposed method is verified when the power demand suddenly fluctuates. The maximum training time is set to K = 1000. When the training set k = 900, the total power demand at time 12 suddenly changes to 432.60MW, an increase of 80MW on the original load. The experiment is conducted using the proposed algorithm. After 1000 trainings, the total power generation cost is learned to be 38692.94. Figure 4 The total cost of the policy learned during the training process is shown, and the evolution of the power output learned during the training process at time 6 is shown. From the results, it can be seen that when k = 1500, the method of the present invention can quickly adjust the output power distribution to meet the power demand. The output of the G2 generator suddenly changes from 59.70143MW to 139.70143MW, and the total power output becomes 432.60MW. The corresponding Fig. 9 The cost of electricity generation by China Total has suddenly increased.

[0255] Experimental results show that the algorithm is robust under fluctuating power demand conditions.

[0256] In summary, the present invention provides a method and system for dynamic economic dispatch decision-making of power systems for improving reliability, which uses reinforcement learning to train a model that can predict the optimal output of generator sets. During the training process, the algorithm learns how to adjust the output of the generator sets according to changes in electricity demand, while taking into account the cost of power generation. The reinforcement learning model gradually approaches the optimal power output strategy through continuous iteration and learning; while ensuring the stability and reliability of the power system, it efficiently utilizes new energy power generation resources, reduces dependence on traditional fossil fuel power generation, and reduces overall power generation costs and environmental impacts.

[0257] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0258] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0259] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0260] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0261] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0262] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0263] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0264] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0265] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0266] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0267] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A dynamic economic dispatch decision method for power system for reliability improvement, characterized in that: The following steps are involved: S1. Construct a Markov decision process for dynamic economic scheduling, define the state transition probability through the Markov chain, and use it to calculate the strategy and value function of reinforcement learning; S2. Construct a reinforcement learning environment to provide simulation scenarios for the power system generator set agent. The power system generator set agent makes scheduling decisions based on the system status and schedules the generator set to minimize the total power generation cost. S3, updating the neural network of distributed reinforcement learning, and the power system generator set agent adjusts its strategy according to environmental feedback; S4, obtaining feasible power output under constraint conditions; S5. Obtain the total cost of power generation through the average consensus algorithm, and return the total cost of power generation to the power system generator agent as a reward for reinforcement learning; S6. Introduce the greedy strategy ε-greedy algorithm, use the power system generator set agent to maximize the current reward that can be obtained, and ensure that the power system generator set agent learns the optimal strategy in a complex environment; S7. Definition The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

2. The power system dynamic economic dispatch decision-making method for reliability improvement according to claim 1 is characterized in that: In step S1, a multi-agent Markov decision process for dynamic economic dispatch of a high-proportion renewable energy system is designed, which is defined as follows: in, is a global state space shared by all agents, set to the maximum output of new energy With total load D t , re i =1,2,...,m, where m is the number of new energy sources; is the action space of agent i, which is designed as the active output p of the generator set i,t ; is the action reward value of agent i; γ∈[0,1] is the discount factor. In multi-agent decision-making, the agent obtains information about the global state space. Predict the action strategy of local agent i Get local rewards at the same time t i .

3. The power system dynamic economic dispatch decision-making method for reliability improvement according to claim 1 is characterized in that: In step S2, the reinforcement learning environment includes the state space s t , generator set output constraints; Each unit agent estimates the total power demand through the average consensus algorithm. The consensus algorithm for the total demand is expressed as: Among them, L is the graph The Laplace matrix of the communication topology between central nodes, Among them, max i l ii is the maximum value of the diagonal elements of the Laplace matrix L, is the average power obtained in the kth iteration; The consensus protocol yields: Among them, 1 N is an N-dimensional column vector, the matrix elements are 1, D t is the total load value of the power grid at time t, and N is the total number of power grid nodes; definition To represent the communication topology degree estimated by the central node i, Each central node obtains the same ∈ through a distributed algorithm; After N iterations, there exists Equal to the degree of the communication topology.

4. The power system dynamic economic dispatch decision method for reliability improvement according to claim 1 is characterized in that: Step S3 is specifically as follows: S301. Design a dual neural network consisting of an action network and an evaluation network. The action network is responsible for the output action to be taken under a given grid state; the evaluation network evaluates the expected cost of the system after taking a certain action. Define As the state-action value function under the policy action π at time slot t, the reinforcement learning value network estimation is established, ω t is the parameter of the network, and the mean square error with the target network is used as the loss function to update ω t Parameters, obtain the minimum value of the state action value function to obtain an approximate solution for economic dispatch; use a distributed method to obtain the minimum value of the state action value function Solve problems about approximate state-value functions; S302, each generator set's agent has an action network Using parameters Approximate optimal policy; Select a fully connected multilayer perceptron with two hidden layers to approximate the optimal policy; Reinforcement learning state re i =1,2,...,m and agent actions are the input and output of the action network respectively; along Relative to The negative gradient is updated.

5. The power system dynamic economic dispatch decision method for reliability improvement according to claim 4 is characterized in that: The loss function is defined as: in, Refers to the expected value, γ is the discount factor, is the estimated value of the action-value function of the current state; Update t : in, is the learning rate; C t Indicates that in state s t The minimum total cost from time t to t+1; It's Q t (s t ,P t i ;w t ) with respect to t; The action network of generator unit i is expressed as: in, is the weight matrix, is the bias vector, h1 and h2 are the number of neurons in the two hidden layers; is a non-linear activation function.

6. The power system dynamic economic dispatch decision-making method for reliability improvement according to claim 1 is characterized in that: Step S4 is specifically as follows: S401, each intelligent agent decides the power output of each generator Take samples; S402, adding a projection controller to the power output distribution; if the constraints are not met, a joint action that meets the constraints is obtained through projection, and the problem is transformed into an optimization problem, which is solved by a projection optimization algorithm. If the communication topology is undirected and connected, the equilibrium point of this formula is the optimal solution to the optimization problem.

7. The power system dynamic economic dispatch decision-making method for reliability improvement according to claim 1 is characterized in that: In step S5, define and is the local estimate of the average electricity cost at time t, and c i,t =F(p i,t ); Applying the average consensus algorithm, the time costs satisfy: get The local estimated generation cost of the average value converges to the actual average generation cost at time slot t, and the total generation cost is 8. The power system dynamic economic dispatch decision method for reliability improvement according to claim 1 is characterized in that: Step S6 is specifically as follows: Introduce the greedy strategy ε-greedy algorithm, the probability of the optimal action is 1-ε, and the probability of random exploration action is ε; design exploration strategy For the current k-th training; when the number of explorations k≤K0, the agent's decision power is determined by: When the number of explorations k>K0, the agent's decision-making power is determined by: in, Indicates in s t The probability of choosing action a; Initially, the agent's decision action is given by the evaluation network, which has added some noise inside the algorithm before outputting the action, and adopts a greedy strategy to ensure exploration performance; After the training set, the actor network gives a deterministic decision.

9. The power system dynamic economic dispatch decision method for reliability improvement according to claim 1, characterized in that: In step S7, let p i,t An action performed in a real environment; definition: Update the local operation policy: in, is the optimal action generated by the evaluation network; p i,t =p i ' ,t It is an exploration of the interaction between action and real power grid environment. It is the optimal scheduling record during the training process; or p i,t When the decision output combination is more economical, the optimal scheduling strategy Updated to the best.

10. A power system dynamic economic dispatch decision system for reliability improvement, characterized in that: include: Building modules, constructing the Markov decision process of dynamic economic scheduling, defining the state transition probability through the Markov chain, and calculating the strategy and value function of reinforcement learning; Construct a reinforcement learning environment to provide simulation scenarios for the power system generator set agent. The power system generator set agent makes scheduling decisions based on the system status and schedules the generator set to minimize the total power generation cost. Update module, update the neural network of distributed reinforcement learning, and the power system generator set agent adjusts the strategy according to the environmental feedback to obtain the feasible power output under the constraint conditions; The calculation module obtains the total cost of power generation through the average consensus algorithm and returns the total cost of power generation to the power system generator agent as a reward for reinforcement learning; The strategy module introduces the greedy strategy ε-greedy algorithm, which uses the power system generator set agent to maximize the current reward that can be obtained, ensuring that the power system generator set agent learns the optimal strategy in a complex environment; Output module, definition The local operation strategy is updated to be the historical optimal strategy, so that the optimal scheduling strategy decreases monotonically and finally reaches the optimal state.

Images