Power network economic dispatching method based on parallel training of PSO and DDPG algorithms

Abstract

The invention belongs to the technical field of artificial intelligence and power grid dispatching. The invention provides a power network economic dispatching method based on parallel training of PSO and DDPG algorithms. According to the embodiment of the invention, the power system network is scheduled by adopting the DDPG algorithm, the model and training parameters in the DDPG algorithm are optimized by adopting the PSO algorithm, and the DDPG intelligent agent is trained by adopting the optimal network model and the optimal parameters, so that the adaptability of the intelligent agent to the scheduling task of the power network system is enhanced; the method has lower calculation complexity and avoids dimension disasters for high-dimensional data. The intelligent agent parallel training has randomness, so that the robustness of the final intelligent agent can be ensured.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical fields of artificial intelligence and electric power grid dispatching, and in particular to an electric power network economic dispatching method based on parallel training of PSO and DDPG algorithms. Background Art

[0002] In the context of vigorous social and economic development, the power system, as one of the core components of infrastructure, plays a vital role in promoting sustainable development. With the surge in modern industrial electricity demand and the increase in the proportion of new energy grid connection, the instability and complexity of the power network continue to rise, and traditional power energy dispatching methods are difficult to meet the current power system's needs for stability and economy. In particular, the major market changes caused by the gradual opening of the power market and the increasing frequency of power transactions have made it urgent to overcome the difficulties in optimizing the allocation of power generation resources, reducing economic expenditures at the power generation end, and improving the economic benefits of power generation and transmission while ensuring the stability and safety of the transmission network.

[0003] At present, the main problems encountered in the research on economic dispatch of power networks of large power grids are the large amount of data and the long time period for data collection and calculation. It is difficult to reflect the operation of the power grid in real time, making it difficult to achieve economic dispatch. The economic dispatch of power systems is a high-dimensional, non-convex, nonlinear constrained optimization problem, so it is particularly difficult to solve this problem. my country's power system has long insisted on centralized dispatch, which will make the solution of economic dispatch of power systems more difficult. It is urgent to find an effective method for solving the economic dispatch of large power grids. Summary of the invention

[0004] In order to avoid the shortcomings of the prior art, the present application provides a power network economic dispatching method based on parallel training of PSO and DDPG algorithms to solve the problem of nonlinear optimization faced by smart grids in the prior art, which is constrained by multiple uncertain factors and has high solution complexity.

[0005] According to an embodiment of the present disclosure, a method for economic dispatching of a power network based on parallel training of PSO and DDPG algorithms is provided, the method comprising:

[0006] Using the power network topology model, respectively obtaining link topology information, line parameter information, and linear and nonlinear constraints, and constructing a power grid topology environment based on the link topology information, the line parameter information, and the linear and nonlinear constraints;

[0007] Respectively obtaining an action space and a state space in the power grid topology environment;

[0008] Constructing an overall reward function; wherein the overall power network system environmental reward includes an environmental reward and a negative economic reward of the power network system;

[0009] Initializing an initial DDPG agent using the reward function, the action space, and the state space, and initially defining initial hyperparameters;

[0010] The PSO algorithm is used to optimize the number of neural network layers and the number of neurons in each layer of the strategy network in the DDPG algorithm to obtain the strategy network model;

[0011] Optimizing the initial hyperparameters using the PSO algorithm to obtain target hyperparameters that are adapted to the policy network model;

[0012] Using the policy network model as the policy network of the initial DDPG agent, and using the target hyperparameters to train the initial DDPG agent in parallel to obtain a trained target DDPG agent;

[0013] Input the current power network environment status into the trained target DDPG agent to obtain the scheduling result.

[0014] Furthermore, the method also includes:

[0015] Obtain a benchmark capacity baseMVA, and use the benchmark capacity baseMVA as a measurement benchmark for power network scheduling to normalize the data.

[0016] Further, the step of respectively acquiring link topology information, line parameter information, and linear and nonlinear constraints using the power network topology model, and constructing a power grid topology environment based on the link topology information, the line parameter information, and the linear and nonlinear constraints includes:

[0017] Acquire the link topology information and the line parameter information between each network node from the power network topology model;

[0018] Using the power network topology model to obtain the linear and nonlinear constraints of each device in the power network;

[0019] The link topology information, the line parameter information and the linear and nonlinear constraints are combined to obtain the power grid topology environment.

[0020] Furthermore, the steps of respectively obtaining the action space and the state space in the power grid topology environment include:

[0021] The linear and nonlinear constraints of each device in the power grid topology environment are read to obtain N a Information about schedulable equipment; among them, N a The dispatchable device information includes Power generation equipment, Adjustable load devices, Energy storage devices and A reactive power compensation device;

[0022] Obtain n from each of the power generation equipment, the adjustable load equipment, the energy storage equipment and the reactive power compensation equipment respectively gen adjustable action dimensions, n l adjustable action dimensions, n sto adjustable action dimensions and nq adjustable action dimensions;

[0023] According to the n gen adjustable action dimensions, the n l adjustable action dimensions, the n sto adjustable action dimensions and the n q adjustable action dimensions to construct the action space A of the initial DDPG agent * ;

[0024] The link topology information and the line parameter information in the power grid topology environment are read to obtain N s running status device information; wherein the running status device information includes Line nodes and transmission branches;

[0025] Obtain n from each of the line nodes and the transmission branches respectively bus observable state dimensions and n b ru observable state dimensions;

[0026] According to the n bus observable state dimensions and the n bru observable state dimensions, constructing the state space S of the initial DDPG agent * .

[0027] Furthermore, the steps of constructing the overall reward function include:

[0028] The power network system simulation environment is used to calculate the power network flow state at time t, and the state S of each device, node, and branch at time t is obtained. t ;

[0029] The linear and nonlinear constraints in the power network system are used as a benchmark to check whether the current state violates the linear and nonlinear constraints, and whether the constraints are violated is used as the environmental reward R 1 ;

[0030] Calculate the overall operation cost M of the network under time t t , and weighted calculation to obtain the negative economic reward R2 ;

[0031] According to the environment reward R 1 and the negative economic reward R 2 Get the final overall reward function R; where R = R 1 +R 2 .

[0032] Furthermore, the PSO algorithm includes:

[0033] Define the parameters in the particle swarm optimization algorithm, including: particle initial velocity v i , the initial position of the particle x i , particle self-learning factor c 1 , global learning factor c 2 , the particle's own random factor r 1 , global random factor r 2 , maximum inertia weight ω max , minimum inertia weight ω min and the number of iterations K;

[0034] The fitness of each position passed by the particle is calculated to obtain the fitness value f(x i );

[0035] The fitness value f(x i ) and the historical best fitness value f of this particle i，best Compare, if the fitness value f(x i ) is greater than the historical best fitness f i，best , then the fitness value f(x i ) is assigned to the historical best fitness f i，best , and the particle's optimal fitness position P i，best Modify to current position;

[0036] Traverse the best fitness f of all particles j，best , and the optimal fitness f j，best and the global optimal fitness value f g，best For comparison, if the optimal fitness f of the kth particle k，best Greater than the global optimal fitness value f g，best ,but:

[0037] f g，best =f k，best

[0038] P g，best =P k，best

[0039] Among them, Pg，best is the global optimal fitness position coordinate;

[0040] According to the current speed v i , the optimal fitness position P of the particle history i，best and the global optimal fitness position coordinate P g，best , update the travel direction and speed information of all particles.

[0041] Furthermore, the step of using the PSO algorithm to optimize the number of neural network layers and the number of neurons in each layer of the policy network in the DDPG algorithm to obtain an optimized policy network model includes:

[0042] The number of input neural network layers L and the number of neurons in each layer n corresponding to the current particle position coordinates l The strategy network model is trained, and the total amount of rewards obtained in the preset rounds is used as the fitness to evaluate the current position of the particle to obtain the optimized strategy network model; wherein the reward and the fitness are linearly proportional.

[0043] Furthermore, the step of optimizing the initial hyperparameters by using the PSO algorithm to obtain target hyperparameters adapted to the policy network model includes:

[0044] Taking the policy network model as the basic model, the PSO algorithm is used to optimize the initial hyperparameters to obtain the target hyperparameters adapted to the policy network model; wherein the hyperparameters P * Including the number of training rounds Episode, the maximum number of steps per round Step, the learning rate Lr, the discount factor Gamma, the soft update coefficient Tau and the experience replay pool capacity BUFFERCAPACITY.

[0045] Further, the step of using the policy network model as the policy network of the initial DDPG agent and training the initial DDPG agent in parallel using the target hyperparameters to obtain a trained target DDPG agent includes:

[0046] Using the same initial DDPG agent to schedule different independent power network simulation environments;

[0047] The experiences obtained from each independent environment action are collected respectively, and after reaching a preset number, the initial DDPG agent is trained according to the DDPG algorithm based on the strategy network model and the target hyperparameters;

[0048] Each of the trained initial DDPG agents is tested and soft-updated and integrated to obtain the target DDPG agent.

[0049] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0050] In the embodiment of the present disclosure, through the above-mentioned power network economic dispatch method based on parallel training of PSO and DDPG algorithms, on the one hand, the DDPG algorithm is used to dispatch the power system network, the PSO algorithm is used to optimize the model and training parameters in the DDPG algorithm, and the optimal network model and optimal parameters are used to train the DDPG agent, so as to enhance the adaptability of the agent to the power network system dispatching task, have lower computational complexity and avoid the dimensionality disaster for high-dimensional data. The parallel training of agents is beneficial to ensure the robustness of the final agent because of its randomness. On the other hand, when facing high-dimensional and non-convex problems, a large amount of iterative calculations are not required, and the practicality in real-time system application scenarios is high. The particle swarm optimization algorithm is used to avoid the agent strategy network from falling into the local optimal solution to ensure good global convergence. Using the reinforcement learning algorithm to find the global optimal strategy has a great advantage over the classical mathematical method in non-convex environments. The agent deep neural network has a strong fitting and adaptability ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0052] Figure 1 A step diagram showing a method for economic dispatching a power network based on parallel training of PSO and DDPG algorithms in an exemplary embodiment of the present disclosure;

[0053] Figure 2 A schematic diagram showing a structural diagram of a power grid topology environment in an exemplary embodiment of the present disclosure;

[0054] Figure 3 A schematic diagram showing the structure of the action space in an exemplary embodiment of the present disclosure;

[0055] Figure 4 A schematic diagram showing the structure of the state space in an exemplary embodiment of the present disclosure;

[0056] Figure 5 A schematic diagram showing the structure of a reward function R in an exemplary embodiment of the present disclosure is shown;

[0057] Figure 6 A flowchart showing a POS algorithm in an exemplary embodiment of the present disclosure;

[0058] Figure 7A DDPG algorithm structure diagram in an exemplary embodiment of the present disclosure is shown;

[0059] Figure 8 A schematic diagram showing parallel training in an exemplary embodiment of the present disclosure;

[0060] Fig. 9 A result diagram showing the global optimal fitness optimization result of the POS algorithm in the exemplary embodiment of the present disclosure;

[0061] Fig.10 A result diagram showing the optimal fitness of particles optimized by the POS algorithm in an exemplary embodiment of the present disclosure;

[0062] Fig.11 A schematic diagram showing a customized parameter training reward in an exemplary embodiment of the present disclosure;

[0063] Fig.12 A schematic diagram showing optimization of parameter training rewards in an exemplary embodiment of the present disclosure;

[0064] Fig.13 A specific flow chart of a method for economic dispatching of a power network based on parallel training of PSO and DDPG algorithms in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0065] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0066] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0067] This example implementation provides a method for economic dispatch of power networks based on parallel training of PSO and DDPG algorithms. Figure 1 As shown in , the power network economic dispatch method based on parallel training of PSO and DDPG algorithms may include: steps S101 to S108.

[0068] Step S101: using the power network topology model to respectively obtain link topology information, line parameter information, and linear and nonlinear constraints, and constructing a power grid topology environment based on the link topology information, the line parameter information, and the linear and nonlinear constraints;

[0069] Step S102: respectively obtaining an action space and a state space in the power grid topology environment;

[0070] Step S103: constructing an overall reward function; wherein the overall power network system environmental reward includes an environmental reward and a negative economic reward of the power network system;

[0071] Step S104: Initializing an initial DDPG agent using the reward function, the action space, and the state space, and initially defining initial hyperparameters;

[0072] Step S105: using the PSO algorithm to optimize the number of neural network layers and the number of neurons in each layer of the policy network in the DDPG algorithm to obtain a policy network model;

[0073] Step S106: optimizing the initial hyperparameters using the PSO algorithm to obtain target hyperparameters adapted to the policy network model;

[0074] Step S107: using the policy network model as the policy network of the initial DDPG agent, and using the target hyperparameters to train the initial DDPG agent in parallel to obtain a trained target DDPG agent;

[0075] Step S108: Input the current power network environment status into the trained target DDPG agent to obtain the scheduling result.

[0076] Through the above-mentioned power network economic dispatch method based on parallel training of PSO and DDPG algorithms, on the one hand, the DDPG algorithm is used to dispatch the power system network, the PSO algorithm is used to optimize the model and training parameters in the DDPG algorithm, and the optimal network model and optimal parameters are used to train the DDPG agent, so as to enhance the adaptability of the agent to the power network system dispatching task, have lower computational complexity and avoid the dimensionality disaster for high-dimensional data. The parallel training of agents is beneficial to ensure the robustness of the final agent due to its randomness. On the other hand, when facing high-dimensional and non-convex problems, a large number of iterative calculations are not required, and it has high practicality in real-time system application scenarios. The particle swarm optimization algorithm is used to avoid the agent strategy network from falling into the local optimal solution to ensure good global convergence. The use of reinforcement learning algorithm to find the global optimal strategy has a great advantage over classical mathematical methods in non-convex environments. The agent deep neural network has a strong fitting and adaptability ability.

[0077] Next, we will refer to Figures 1 to 13 The various steps of the above-mentioned power network economic dispatch method based on parallel training of PSO and DDPG algorithms in this example implementation are described in more detail.

[0078] In step S101, the power data base capacity baseMVA of the power network topology model is collected and used as the data base in the agent training process to avoid the problem of inconsistent unit magnitude during model training. As the measurement base of power network dispatch, baseMVA no longer uses actual physical units for system parameters, which simplifies calculations and reduces numerical stability issues in calculations.

[0079] Get the power grid topology environment E p Mathematical logic data abstracted from real networks, generally designed for computer systems to simulate power networks, including: link topology information E between network nodes in the power network pnet , which contains the link information between each node in the power network system, that is, the transmission line link information in the network structure, which is presented here with nodes as the main body; the line parameter information E between network nodes pY Including: Line resistance R n and circuit conductance G n At least one of the two, line reactance X n and line susceptance B n At least one of the two, where the line resistance R n and circuit conductance G n The amount of information carried is the same, the line reactance X n and line susceptance B n The amount of information carried is the same, so you can choose one of them and combine them to calculate the power network node admittance matrix Y.

[0080] Obtain linear and nonlinear constraints F n Linear and nonlinear constraints are terms used in mathematical optimization, which stipulate the feasible range of various parameter data. In the power network system, linear and nonlinear constraints F n Including the equipment performance limitations in the system: Generator performance limitation F gen，n , mainly including the upper and lower limits of power generation equipment performance, power generation equipment work climbing rate, etc.; adjustable load performance limit F l，n , mainly including the upper and lower limits of adjustable load performance, the adjustable power consumption ramp rate; the performance limit of energy storage equipment F s，n , mainly including the maximum and minimum values ​​of energy storage equipment, the storage power range of energy storage equipment, the power loss of energy storage equipment over time; reactive power compensation performance limit F q，n , mainly including the upper and lower limits of each reactive compensation equipment performance; the compensation climbing performance climbing rate; the performance limit F of each node Node，n , mainly including the upper and lower voltage limits of each network node, the voltage phase limit of each network node; the performance limit of each line F b，n, including the transmission power limit of each line and the transmission loss of the transmission line.

[0081] The topological information of the links between nodes in the power network pnet and line parameter information E between network nodes pY , linear and nonlinear constraints F n The three constitute the power grid topology environment E p ={E pnet , E pY , F n}, the grid topology information is a constant in the dispatching process. Grid topology environment E p Structure Figure 2 shown.

[0082] In step S102, the power grid topology environment E is obtained. p A dispatchable action n , which can dispatch action A n Output actions for the agent, which can schedule actions A in the current specific environment n Specifically reflected in the adjustable amount of adjustable equipment in the power system: adjustable amount of generator set A gen，n， Indicates the adjustable quantity of the power supply end in the power network system; adjustable load equipment A l，n , which represents the adjustable amount of the power consumption end in the power network system, and its main function is to maintain the balance of various data in the power network. s，n , which means that the energy storage device in the power network system can adjust the amount, mainly used to absorb energy when the network active power output is high, and release energy when the active power generation does not meet the consumption, but the stored energy will gradually consume over time. Reactive power compensation equipment A q，n , which is mainly used for on-site reactive power compensation, thereby reducing the transmission of reactive power in the power network and saving various losses. The subscript n represents an indefinite number and is modified according to the number of devices in different power network system environments.

[0083] The specific acquisition obtains the power network topology model N a The dispatchable equipment information is classified into the following categories according to their equipment categories: Power generation equipment, Adjustable load devices, Energy storage devices, Reactive power compensation equipment obtains n reactive power from each power generation equipment, adjustable load equipment, energy storage equipment and reactive power compensation equipment. gen adjustable action dimensions, n l adjustable action dimensions, n sto adjustable action dimensions and n q adjustable action dimensions, according to n genadjustable action dimensions, n l adjustable action dimensions, n sto adjustable action dimensions and n q The agent action space is constructed by the adjustable action dimension. The action space is The upper and lower limits of each dimension are the upper and lower limits of the adjustment capacity of each adjustment dimension of the corresponding schedulable equipment A i，max , A i，min , where N a is the total number of schedulable devices. The number of action space dimensions is:

[0084]

[0085] A i，max , A i，min are the upper and lower limits of the adjustable capabilities of the corresponding device in the i-th dimension, respectively.

[0086] Action Space A * Dispatch space for power generation equipment Adjustable load equipment adjustable space Dispatchable space for energy storage equipment Dispatch space of reactive power compensation equipment The construction results in:

[0087]

[0088] Among them, A * Each dimension of is a normalized mapping value, which can help the agent fit the network faster. Action space A * Structure Figure 3 shown.

[0089] Get observable data S n The data required by the agent during training and scheduling. The agent uses the observable data S n Make inference decisions to get scheduling action a n , a n ∈A n Observable data S n In the power network system, it specifically includes the observation values ​​of the equipment that can be continuously observed in the system: the operating status S of the generator set a，n , including the current active power output, reactive power output, whether the generator can be started, and shutdown status; power network node status S N，n, which represents the mathematical expression format of each network node - mainly including three forms: PQ node form, PV node form, and balance node form (V, θ). Among them, the subscript n represents an indefinite number and is modified according to the number of observable devices in different power network system environments.

[0090] The specific acquisition obtains the power network topology model N s The available operating status device information is classified into: Line nodes, transmission branches, and obtain n from each line node and transmission branch. bus observable state dimensions and n bru observable state dimensions, according to n bus observable state dimensions and n bru The state space of the agent is constructed by the observable state dimension. Dimension, the upper and lower limits of each dimension are the physical upper and lower limits S of the corresponding observation information of the observable object i，max , S i，min In addition, line node information It also includes the current operation status of dispatchable equipment and load demand information of power network users; s is the number of observable devices. The number of state space dimensions is:

[0091]

[0092] S i，max , S i，min The upper and lower limits of the observable quantity corresponding to the node or line of the i-th dimension are physically defined, that is, the upper and lower limits of the possible values, where

[0093] State space S * Get the observable state space for the line node Transmission branch obtains observable state space The construction results in:

[0094]

[0095] Among them, S * Each dimension of is also a normalized mapping value. This operation and action space A * The normalization of has the same reason. The state space S * Structure Figure 4 shown.

[0096] In step S103, the reward function is a way for the environment to provide learning direction to the intelligence. The environment provides a reward feedback for the current action of the agent through the reward function, and the agent adjusts the internal network of the agent based on this. Therefore, the reward function directly reflects the goal of the agent's learning task and helps the agent shape its action strategy and reasoning network. The specific method of constructing the reward function Reward here is: first, the power network flow state at time t is calculated through the power network system simulation environment, and the state S of each device, node, and branch at time t can be obtained. t The linear and nonlinear constraints F in the power network system n Use the benchmark to check whether the current state violates the linear and nonlinear constraints F n , using whether the constraint is violated as the power system environmental reward R 1 ; Then calculate the overall network operation cost M at time t t , including the dispatching cost of each device and the consumption cost of each device to maintain the status quo, and weighted calculation to obtain the negative economic reward R of the power network system 2 =k M *M t , where k M is a negative weight; the sum of the two parts of the reward gives the final overall power network system environment reward R: R = R 1 +R 2 The final structure of the overall power network system environmental reward R is as follows: Figure 5 shown.

[0097] In step S104 to step S106, the initial parameter P * Under this condition, the particle swarm algorithm is used to optimize the number of neural network layers L and the number of neurons in each layer n of the strategy network. l , l = 1, ..., L. The particle swarm algorithm is inspired by the idea of ​​birds searching for food in nature. Its search capability can be used to search for the best network structure and optimize the intelligent agent network model.

[0098] In nature, bird flocks search for food by sharing information within the flock, thereby guiding the direction of food and helping the flock find places where food is abundant. Bird flocks need to find the place with the largest amount of food in the forest. All individual birds cannot determine where the food is, but they can sense the direction of the food by smell, so each bird will search along the direction determined by the individual. At the same time, each bird will remember the place where the most food was found in the process of searching for food. And each bird will share the coordinates of the food found and the amount of food during the search process, so all individuals in the flock will know the coordinates of the current largest amount. Each bird can adjust the new search direction based on the place with the largest amount of food in memory, the coordinates with the highest amount of food in the flock, and its own search direction. Because the distribution of food is not completely random, this foraging behavior can help the flock find the place with the most food in the forest.

[0099] In the PSO algorithm, the particle corresponds to the bird mentioned above, the solution space is the entire forest, the objective function (fitness) is the amount of food at each location in the forest, the particle position (solution) corresponds to the location of the bird, and the global optimal solution is the coordinate position with the largest amount of food, and the coordinate can be composed of N dimensions, corresponding to N values ​​that need to be optimized. Assume that there are k particles in the N-dimensional search space. First, define the position of each particle based on experience or sampling method:

[0100] x i =(x i1 , x i2 , ..., x in )

[0101] Speed ​​(moving direction and distance information):

[0102] v i =(v i1 , v i2 , ..., v in )

[0103] After the particle starts to move, the fitness of each position will be calculated and evaluated to obtain the fitness value f(x i ), fitness value f(x i ) indicates the extent to which the particle meets the requirements at this position. i ) is usually given as the desired optimization target. The fitness f(x i ), we need to convert f(x i ) is compared with the historical best fitness of this particle. If f(x i ) is greater than the historical best fitness value f i，best , then f(x i ) is assigned to fi，best , and at the same time set the particle's optimal fitness position P i，best Modify to the current position. After each particle has been traversed forward once, the best fitness of all particles is traversed. j，best , and the global optimal fitness value f g，best By comparison, if the best fitness f of the kth particle k，best Greater than the global optimal fitness value f g，best ,but:

[0104] f g，best =f k，best

[0105] P g，best =P k，best

[0106] Among them, P g，best is the global optimal fitness position coordinate.

[0107] After updating the fitness information, it is necessary to calculate the direction and speed information of all particles. This information is obtained by integrating the current speed v i 、The best fitness position P of this particle in history i，best and the global best fitness position P g，best Three items were updated:

[0108]

[0109] Among them, ω is the particle inertia weight, c 1 is the particle’s own learning factor, c 2 is the global learning factor, r 1 is the random factor of the particle itself, r 2 is the global random factor. This formula consists of the sum of three parts:

[0110] Own inertia: This part is composed of the inertia weight ω and the current speed of the particle The purpose of this design is to reduce the ability of the particle to maintain its own speed and direction from large to small during the process of continuous searching, so that a larger search ability can be used in the early stage of optimization, and in the later stage, it can help the particle to have a higher convergence accuracy. Because when optimizing problems, it is generally expected that the algorithm will first use random search on the global level, so that the search space can be quickly locked to a small area to reduce performance waste, and then the local area can be finely searched to obtain a high-precision optimization solution. Therefore, the inertia weight ω will gradually decrease with the increase of the number of iterations to achieve this goal. According to the formula:

[0111]

[0112] Among them, ω max ,ωmin The distribution is the maximum and minimum values ​​of the inertia weight, k and K are the current number of iterations and the maximum number of iterations respectively.

[0113] Independent knowledge: This represents the particle’s own experience, which is actually reflected in the particle’s current position The particle's optimal fitness position The difference between represents the distance of the particle from the optimal fitness position. This part is to allow the particle to use the optimal fitness position in its own experience To guide the particles, they wander or roam back and forth in a small area around them, guiding them to move to a position with better fitness.

[0114] Group communication: represents the map information sharing within the particle group, with the best fitness position in the whole map As a guide, the limited particles can be guided to the position where the better fitness may appear, thereby saving performance and increasing the optimization efficiency and accuracy. and the global best fitness position P g，best The difference can also guide the particles to wander around the point in search of the best solution.

[0115] These three parts constitute the idea of ​​updating the position and speed of particles. The change of position only needs to specify the particle travel benchmark for a single time step, which is set as 1 here, that is, the travel distance per time step is v i , the position update formula is:

[0116]

[0117] The principle of the PSO algorithm is intuitive and relatively simple to implement. It does not require complex mathematical derivation or heavy computing resources. Because each particle searches independently, the PSO algorithm can be calculated in parallel, which means that in a multi-core computing system, the PSO algorithm can be significantly accelerated. At the same time, the PSO algorithm also has good optimization capabilities. The flow chart of the particle swarm algorithm is as follows: Figure 6 shown.

[0118] Specifically, we need to optimize the number of input neural network layers L and the number of neurons in each layer n l Therefore, the position corresponding to each particle in the particle swarm optimization method should be the number of neural network layers L and the number of neurons in each layer n l The value of is used as the coordinate of the particle in the two-dimensional space. The optimization process uses the cumulative reward within a certain round of the DDPG algorithm for power network system scheduling training test as an indicator to evaluate the fitness of each particle in the current PSO algorithm, that is, the number of input neural network layers L and the number of neurons in each layer n corresponding to the current particle position coordinates lAs a network model for training, the total amount of rewards that can be obtained in a specific round is used as the fitness to evaluate the current position of the particle, and the reward and fitness are linearly proportional.

[0119] Specifically define the fitness calculation function f(x i ) is as follows: In the particle swarm algorithm, particles are always looking for the optimal fitness function f(x i ) value, so the fitness function f(x i ) should be the performance indicator of the required optimization algorithm. Here it should be reflected as the total reward value obtained by the agent when scheduling a certain number of steps, that is:

[0120]

[0121] Among them, R t is the reward obtained by the agent in the tth round, and T is the time step defined for calculating the fitness value.

[0122] Define the parameters required in the particle swarm optimization algorithm: particle initial velocity v i , the initial position of the particle x i , particle self-learning factor c 1 , global learning factor c 2 , the particle's own random factor r 1 , global random factor r 2 、Maximum and minimum values ​​of inertia weight ω max ,ω min , the number of iterations K. After a certain number of PSO optimization iterations, the position coordinates of the particle with the optimal fitness are obtained. At this time, the optimal fitness two-dimensional position coordinates of the particle correspond to the number of layers of the optimized neural network L * and the number of neurons in each layer The network model N with the best fitting ability for the current power environment can be determined a，pso , this model is the intelligent agent network model that can obtain the most environmental rewards under the current training parameters, and it has the optimal scheduling mapping capability for the power network system environment.

[0123] After optimizing and determining the number of neural network layers L * and the number of neurons in each layer And combine to determine the strategic network model N that has the best fitting ability for the current power environment a，pso Afterwards, the policy network model N with the best fitting ability a，pso Based on the model, the particle swarm optimization algorithm PSO is used again to optimize the hyperparameters P required in the agent training process. * .

[0124] Hyperparameter P during agent training* Specifically, it includes: the number of training rounds Episode, that is, the total number of rounds required to train a DDPG agent, usually the sum of training rounds and test rounds; the maximum number of steps per round Step, that is, the maximum number of operation steps that will be performed if the environment does not give an end signal during each round of training or testing. It is usually used in an infinite time step environment, and the power network dispatching environment is this type of environment; the learning rate Lr, which can be generally understood as the rate at which each network performs learning and training. Too fast will easily lead to unstable learning, and too slow will cause data and performance waste; the discount factor Gamma, its role is to quantify the importance of the agent to future rewards, The discount factor Gamma is between 0 and 1. The closer it is to 1, the more the agent values ​​future rewards, and the closer it is to 0, the more the agent values ​​current rewards. The soft update coefficient Tau means that the target policy network and the target action value network are updated by mixing the parameters of the target network and the behavior network according to a certain ratio Tau each time, so as to achieve smooth network updates and ensure the stability of the training process. The experience replay pool capacity BUFFER_CAPACITY corresponds to the capacity of the experience replay pool in the DDPG algorithm. The experience in the pool will be sampled and sent to the optimizer for training during model training. Its advantage is that it can disrupt the temporal relationship between experiences and improve the utilization rate of excellent experiences.

[0125] In the DDPG algorithm, there are four deep neural networks: policy network (P-net), target policy network (target P-net), action value network (Q-net), and target action value network (target Q-net). There is usually also an experience replay pool to improve experience utilization and assist the four networks to converge quickly.

[0126] Let the policy network be μ θ (s), where θ represents the network parameters and s is the current input state. Policy network μ θ (s) is essentially a deep neural network. Mathematical analysis of it can be considered as a deterministic function, that is, given a state s, the policy network μ θ (s) will definitely map and output the same action a t There is no randomness in this process, so in the policy network μ θ During the training process of (s), it is necessary to manually introduce a certain amount of randomness into the DDPG agent so that the DDPG agent has the ability to explore the environment and try new behaviors. θ The role of (s) in the DDPG algorithm is: responsible for calculating the current state s at each time step t, according to the weight parameter mapping of the deep neural network, the action a that the DDPG agent should take at this moment is calculated t .

[0127] The target policy network is in, represents the parameters of the target policy network, s is the target policy network The input state, usually the state s at the next moment t+1 Target Policy Network and the policy network μ θ (s) have the same structure, both are deep neural networks, with the same number of network layers and neurons, and can also be considered as a deterministic function, that is, given a state s t+1 , target policy network The same action a will be mapped and outputted t+1 At this time, action a t+1 It is not used directly by the DDPG agent, but is used to estimate future values.

[0128] Let the action value network be Q ω (a, a), where ω represents the parameters of the action-value network and s is the action-value network Q ω (s, a) is the input state, a is the action value network Q ω (s, a) Input action. Action value network Q ω (s, a) is also a deep neural network, but it can be different from the policy network in terms of the number of layers and neurons. The action value network Q ω (s, a) Input a state s from the power network environment t and the policy network μ in the DDPG agent θ (s t ) corresponds to the output action a t , action value network Q ω (s, a) will map and output an action value estimate of the action in the current state Here the action value estimator The meaning of it is the estimated value of the expected cumulative reward that can be obtained in the future. The value is an estimate of the long-term reward of taking a specific action a in a given state s.

[0129] The target action value network is in, represents the parameters of the target action value network, s is the target action value network The input state, a is the target action value network Input action. Target action value network It is also a deep neural network, whose network structure and action value network Q ω The network structure of (s, a) is also consistent, the target action value network Enter a next state s from the power network environment t+1 and the target policy network in the DDPG agent In status t+1 The corresponding output action a t+1 , target action value network It will map and output the action a in the next time step state t+1 The action value estimator Here the action value estimator It represents the estimated value of the expected cumulative reward that can be obtained in the next step.

[0130] In the DDPG algorithm, the information flow between the four networks is as follows: First, the DDPG agent obtains the current state s from the power network system simulation environment. t , and the current state s t Input to the policy network μ θ (s), policy network μ θ (s) will output the current action a of the DDPG agent t to the power network system simulation environment, and the DDPG agent sends the current state s t and the current action a t The input to the action-value network is Q ω (s, a) to get the action value estimate Then, the power network system simulation environment enters the next state s according to the state transition function t+1 ; At this point, the DDPG agent obtains the next time step state s t+1 , and the next time step state s t+1 Input to the target policy network So we get the next time step action a t+1 , at this time s t+1 and a t+1 Will be input into the target action value network Get the action value estimate for the next time step At this point, the information flow between the four networks in one time step is completed, and the DDPG agent obtains two states s t and t+1 , two actions a t and a t+1 , two action value estimators and The overall DDPG algorithm input is s t, the corresponding overall output is a t The overall structure of the DDPG algorithm and the schematic diagram of data information flow are as follows: Figure 7 shown.

[0131] In the DDPG agent, the policy network μ is trained θ (s), the Q network can be regarded as the policy network μ θ (s) is scored, so the policy network μ θ The fitting direction of (s) is the direction with the highest score of Q network. The purpose of DDPG algorithm is to solve the action value function Q ω (s, a) is the highest action. So in training the policy network μ θ (s), the construction of the loss function should maximize Q ω (s, a) is the principle, theoretically, we only need to follow Q ω (s, a) can be monotonically decreased. Among them, the simplest and most effective way is to directly use Q ω (s, a) takes a negative value as loss for training:

[0132]

[0133] Training a deep Q network Q in the DDPG algorithm ω (s, a), its training method is to use the reward r of the training environment t and the next action state value Q of the agent ω (s t+1 , μ θ (s t+1 )) to calculate the current action state value Q′ ω (s t , a t ), and then make the estimated value Q output by the Q network ω (s t , a t ) Approximate the current action state value Q′ ω (s t , a t ). Usually in the DDPG algorithm, the loss function used to optimize the deep Q network is the mean square error of these two values:

[0134]

[0135] Among them, μ θ (s t+1 ) is the strategy network mentioned above in s t+1 Output action a t+1 .

[0136] It can be seen that in the DDPG algorithm training deep Q network Q ω(s, a), the problem of unstable training target will occur, that is, [r t +Q ω (s t+1 , μ θ (s t+1 ))] is unstable, after all, Q in the formula ω (s t+1 , μ θ (s t+1 )) is also an estimate. So we need to use the idea of ​​the target network to deal with this problem. In DDPG, we need to use the idea of ​​the target network for the policy network μ θ (s) and the action-value network Q ω (s, a) create a corresponding target network respectively and By temporarily locking down the target network and The stable training target is obtained in this way.

[0137] Target Network technology means that when learning an agent, the fitting target is fixed and only changed once in a period of time, rather than updated after each interaction. When learning the Q function, at time t, the agent is in state s t , after the action strategy takes action a t , get reward r t , the state transition is s t+1 , at this time:

[0138] Q π (s t , a t )=r t +Q π (s t+1 ,π(s t+1 ))

[0139] Among them, Q π (s t+1 ,π(s t+1 )) is variable, and this application hopes that Q π (s t , a t ) and Q π (s t+1 ,π(s t+1 )) differs only by r t In fact, this situation is not easy to learn. When the deep neural network is doing back propagation, the parameters of Qπ will be updated, causing the training to become unstable. t +Q π (s t+1,π(s t+1 )) as the target of deep neural network fitting, and Q π (s t , a t )When the network fitting training is performed as the output of the deep neural network, the target that the network needs to fit is constantly changing, which causes the network to always be chasing a changing target, making the network training very difficult.

[0140] The target network technique cleverly alleviates this problem. In the target network technique, the agent's Q network is first initialized as two identical deep neural networks; then one of them is fixed, here Q A network refers to, and another network Q B The network is not fixed; Q A The network is usually responsible for generating the target, so it is called the target network. At this time, a fixed fitting target value r can be obtained. t +Q π (s t+1 ,π(s t+1 ));Q B The network uses Q A The target generated by the network is trained for network fitting at each time step; B After the network is trained to a certain extent, Q B The parameters of the network are copied to Q A network, thus achieving Q A Network update; then with the new Q A The target training Q generated by the network B Network, and so on. Network Q under the target network skills A It is designed to be a relatively stable network, and its parameter update frequency is lower than that of the behavior Q B This makes the target Q value relatively stable over a period of time, greatly alleviating the difficulty of deep neural network fitting caused by frequent changes in the target Q value. A It provides a relatively stable and steadily improving baseline for agent learning, making the training objectives clear and stable, thus making the learning and training process of the DDPG agent smoother.

[0141] Specifically, at each step in the DDPG algorithm, the policy network μ is updated θ (s) and the deep Q network Q ω (s, a); the two target networks are fixed and will only be modified after a certain number of interaction rounds, not after each interaction. and The modification is to directly change the policy network μ θ (s) and the action-value network Qω (s, a) After a certain number of training steps, the network model parameters can be synchronized. However, in practical applications, the so-called soft update method is usually used to achieve a smooth transition of the target network:

[0142]

[0143] Among them, ω is the parameter of the Q network when updating the target network, is the target Q network parameter, is the parameter of the updated target Q network; θ is the parameter of the policy network when updating the target network, is the target strategy network parameter, are the parameters of the updated target strategy network; It is the soft update parameter, which controls the smoothness of soft update.

[0144] Since deep reinforcement learning data is obtained through autonomous exploration by the agent, the data acquisition efficiency is low, and the number of deep neural networks that need to be trained in the DDPG algorithm is large, which results in the data volume not being able to meet the training needs, so the experience replay technology is used here. Experience replay technology refers to building an experience replay pool in the process of training the agent, and using the experience in the experience replay pool for ruminant training to improve the utilization rate of the agent's training trajectory experience. Experience replay refers to collecting the data obtained when the existing strategy π interacts with the environment and storing it in the experience replay pool. After accumulating a certain amount of experience, a batch of experience will be extracted from the experience replay pool to fit the agent's deep neural network for training. Among them, each experience includes at least: the current state s t 、The action a taken by the current strategy π t , the reward r t , enter the state s t+1 . The process of training the deep neural network of the agent after extracting a batch of experience is similar to the process of fitting and training the deep neural network in the field of deep learning. This is a manifestation of the combination of deep learning and reinforcement learning. The capacity of the experience replay pool is generally several times the experience collected by a single strategy π, so it usually does not include only the experience of one strategy, so the agent trained by the experience replay pool is usually more robust. And because the experience in the experience replay pool used in training is randomly or according to certain rules, it can break the temporal connection between trajectory experiences, making the training efficiency of the deep neural network higher.

[0145] In step S107 to step S108, the strategy network model N of the combined agent a，pso and hyperparameters adapted to the current network model Further training is performed through the DDPG algorithm combined with parallel training. Parallel training here refers to using the same DDPG agent to schedule different independent power network simulation environments. Because noise is artificially added to the output actions during the training process to enhance the exploration ability of the DDPG agent, different noises will cause the power network system to enter different new states, and the situations faced by the DDPG agent will also be different. The experience gained from each independent environment action is then collected separately. After reaching a certain number, the agents are trained according to the DDPG algorithm, and then each agent is tested. After achieving a certain effect, each agent is soft-updated and integrated into one agent. Figure 8 shown.

[0146] After the agent test results are stable and meet the requirements, take out the strategy network N a，pso Deployed on terminal computing devices, it monitors and dispatches network status in real time.

[0147] In a specific embodiment, the parameters required in the particle swarm optimization algorithm are defined as follows: The initial particle velocity v is defined based on experience and Gaussian noise initialization i ; Initial position of particle x i ; Two types of learning factors c 1 =c 2 =2; two types of random factors r 1 , r 2 Take random numbers in the interval [0, 1] to increase the randomness of exploration; the maximum and minimum values ​​of inertia weight ω max =5,ω min =0.05; number of iterations K = 50.

[0148] Taking the standard power network system dispatch calculation model IEEE30 model and the data of a certain place as an example, the specific implementation process of this application is introduced.

[0149] First, we obtain the benchmark capacity baseMVA. Here, the apparent power of the No. 1 generator set connected to the node bus1 is used as the benchmark capacity baseMVA. Then, we per-unit standardize other data. The power flow calculation and DDPG model training of the overall power network system are performed in per-unit standardization to eliminate the influence of units.

[0150] Then read the link topology information E between each network node in the standard IEEE30 model pnet , line parameter information E between network nodes pY and the linear and nonlinear constraints F of each device in the IEEE30 standard model n , in order to construct the grid topology environment E p, which is used as the standard environment for subsequent training and testing. The Newton-Raphson method is used to gradually approximate a set of variables through iteration so that the given power balance condition is met, thereby obtaining the current feasible power flow data.

[0151] For the power grid topology environment E p Linear and nonlinear constraints for each device in the data package F n Read and get the schedulable action A n , the specific dispatchable equipment information collected is: 5 power generation equipment, 1 adjustable load equipment, 1 energy storage equipment, and 5 reactive power compensation equipment, namely:

[0152]

[0153] The adjustable dimension of each device is 1. Read the linear and nonlinear constraints F of each device again n , obtain the upper and lower limits of the adjustable capabilities of each dimension of each device A i，max , A i，min . Scheduling action A n Each dimension is normalized:

[0154]

[0155] Then combine the action space A as the DDPG agent * Action Space A * The number of dimensions is:

[0156]

[0157] For the power grid topology environment E p The link topology information E between each network node in the data packet pnet and line parameter information E between network nodes pY Read and get observable data S n , the specific dispatchable equipment information collected is: 30 line nodes and 41 transmission branches, namely:

[0158]

[0159] The observable state dimension of each line node is 13, and the observable state dimension of each transmission branch is 13, namely:

[0160] n bus =13

[0161] n bru =13

[0162] Slave device linear and nonlinear constraints F nRead the maximum and minimum values ​​of each node and each branch in each dimension. i，max , S i，min Observable data S n Each dimension is normalized:

[0163]

[0164] Then combine the state space S as the DDPG agent * . State space S * The number of dimensions is:

[0165]

[0166] According to the constructed reward function R, after the environmental calculation program receives the output action of the DDPG agent, it uses whether the constraint is violated as the power system environmental reward R1. Specifically:

[0167] R 1 =r 1 +r 2 +r 3 +r 4 +r 5 +r 6 Among them, r 1 Rewards for violating the upper and lower limits of device actions (negative rewards); 2 Reward for violating the device action power ramp (negative reward); 3 Reward for violating the overall power network power load balance (negative reward); 4 is a reward for violating the rationality of power flow calculation (negative reward), that is, if the scheduling action causes the power flow calculation to fail to converge, r 4 =-1; r 5 is the reward for voltage stability of each node in the power network (negative reward). If the voltage of a node exceeds the upper and lower limits, r 5 The absolute value of r increases according to the weight; 6 is the reward (negative reward) for the power limit of each branch in the power network. If the power of a branch exceeds the upper limit, r 6 The absolute value of increases according to the weight. The above rewards all belong to (-1, 0), the specific expression is:

[0168]

[0169] r 3 =-B balance

[0170] r 4 =-B pf

[0171]

[0172] Among them, B limit A Boolean value indicating whether the device action exceeds the upper and lower limits. If exceeded, it is 1; if not exceeded, it is 0. ramp A Boolean value indicating whether the device violates the power ramping rule. If it violates the rule, it is 1; if it does not violate the rule, it is 0. balance A Boolean value indicating whether the power network is balanced, 0 for balanced and 1 for unbalanced. pf A Boolean value indicating whether the power flow calculation has converged. If it has converged, it is 0; if it has not converged, it is 1. bus A Boolean value indicating whether the node voltage exceeds the upper and lower limits. If exceeded, it is 1; if not exceeded, it is 0. branch A Boolean value indicating whether the branch power exceeds the upper limit. If it exceeds, it is 1; if it does not exceed, it is 0.

[0173] Calculate the overall network operation cost M under time t t , including the dispatching costs of each device and the consumption costs of each device to maintain the status quo, and weighted calculation to obtain the negative economic reward of the power network system:

[0174] R 2 =k M *M t

[0175] where k M is a negative weight, where k is taken M =-10 -5 , reduce the cost weight to ensure that the agent takes the stability of the power network system as the primary task during training, and then weighs the economic benefits; M t Includes the operating costs of various power generation and regulation equipment and the maintenance costs of the power system, which are based on the definition in the IEEE30 standard.

[0176] The final overall power grid system environmental reward R is obtained by adding the two parts of the reward:

[0177] R=R 1 +R 2

[0178] Then with A * As the action space, S * As the state space, R as the environment reward initial DDPG agent, and initially define the hyperparameters P * :

[0179] EPISODES=50000

[0180] STEPS=500

[0181] BATCH SIZE=64

[0182] BUFFER_CAPACITY = 100000

[0183] GAMMA = 0.99

[0184] TAU=0.005

[0185] E_GREEDY=0.2

[0186] ACTOR_LR=0.001

[0187] CRITIC_LR=0.002

[0188] Among them, EPISODES is the total number of training rounds; STEPS is the number of training steps per round, which needs to be configured in such an infinite step environment; BATCH_SIZE is the number of experience points extracted from the experience replay pool for each training; BUFFER_CAPACITY is the capacity of the experience replay pool; GAMMA is the discount factor, and the corresponding update formula in the DDPG algorithm is:

[0189]

[0190] Among them, γ is the controllable agent's trade-off between future rewards and current rewards; TAU is the target network soft update coefficient, corresponding to the target network update algorithm:

[0191]

[0192] in E_GREEDY is the threshold for adding noise, which means that during the training process, noise is used to replace the action output by the policy network with a probability of 0.2 to give the agent the ability to explore autonomously; ACTOR_LR and CRITIC_LR are the learning rates of the policy network and Q network respectively.

[0193] Then the random number generation algorithm is used to define the initial position x of the particle in the particle swarm algorithm. i and initial velocity v i , initial position x i The first dimension represents the number of policy network layers L, and the value range is defined as (20, 200); the initial position x i The second dimension represents the number of neurons in each hidden layer n l , define the value range as (64, 4096). Define the initial parameters:

[0194] P_SIZE=50

[0195] W_MAX=5

[0196] W_MIN=0.05

[0197] C1=1.5

[0198] C2=1.5

[0199] MAX ITERATION=100

[0200] Among them, P_SIZE is the number of particles in the population; W_MAX and W_MIN are the maximum and minimum values ​​of the inertia weight; C1 is the cognitive parameter; C2 is the social parameter; MAX_ITERATION is the maximum number of iterations. According to step S105 and step S106, the optimized network structure N is obtained. a，pso .like Fig. 9 As shown in , it is the result diagram of particle swarm optimization global optimal fitness. Fig.10 As shown in Figure 1, it is the result diagram of the optimal fitness of particles in particle swarm optimization.

[0201] After obtaining the optimized network structure, the particle dimension in the particle swarm optimization algorithm is modified to 9 dimensions, and the hyperparameter P is optimized. * . So as to obtain the most suitable current network model structure N a，pso Hyperparameters like Fig.11 As shown in , it is a custom parameter training reward. Fig.12 As shown, the reward for optimizing parameter training.

[0202] Based on the agent's policy network model N a，pso and hyperparameters adapted to the current network model Conduct multi-agent parallel training to speed up the training of the DDPG agent model, making it more stable and enhancing its robustness.

[0203] After the agent test is stable, it can be deployed to the terminal device. The overall power network system dispatching agent training process is as follows: Fig.13 shown.

[0204] Through the above-mentioned power network economic dispatch method based on parallel training of PSO and DDPG algorithms, on the one hand, the DDPG algorithm is used to dispatch the power system network, the PSO algorithm is used to optimize the model and training parameters in the DDPG algorithm, and the optimal network model and optimal parameters are used to train the DDPG agent, so as to enhance the adaptability of the agent to the power network system dispatching task, have lower computational complexity and avoid the dimensionality disaster for high-dimensional data. The parallel training of agents is beneficial to ensure the robustness of the final agent due to its randomness. On the other hand, when facing high-dimensional and non-convex problems, a large number of iterative calculations are not required, and it has high practicality in real-time system application scenarios. The particle swarm optimization algorithm is used to avoid the agent strategy network from falling into the local optimal solution to ensure good global convergence. The use of reinforcement learning algorithm to find the global optimal strategy has a great advantage over classical mathematical methods in non-convex environments. The agent deep neural network has a strong fitting and adaptability ability.

[0205] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0206] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for economic dispatching of power networks based on parallel training of PSO and DDPG algorithms, characterized in that: The method includes: Using the power network topology model, respectively obtaining link topology information, line parameter information, and linear and nonlinear constraints, and constructing a power grid topology environment based on the link topology information, the line parameter information, and the linear and nonlinear constraints; Respectively obtaining an action space and a state space in the power grid topology environment; Constructing an overall reward function; wherein the overall power network system environmental reward includes an environmental reward and a negative economic reward of the power network system; Initializing an initial DDPG agent using the reward function, the action space, and the state space, and initially defining initial hyperparameters; The PSO algorithm is used to optimize the number of neural network layers and the number of neurons in each layer of the strategy network in the DDPG algorithm to obtain the strategy network model; Optimizing the initial hyperparameters using the PSO algorithm to obtain target hyperparameters that are adapted to the policy network model; Using the policy network model as the policy network of the initial DDPG agent, and using the target hyperparameters to train the initial DDPG agent in parallel to obtain a trained target DDPG agent; Input the current power network environment status into the trained target DDPG agent to obtain the scheduling result.

2. The power network economic dispatching method based on parallel training of PSO and DDPG algorithms according to claim 1 is characterized in that: The method further includes: Obtain a benchmark capacity baseMVA, and use the benchmark capacity baseMVA as a measurement benchmark for power network scheduling to normalize the data.

3. The power network economic dispatching method based on parallel training of PSO and DDPG algorithms according to claim 1 is characterized in that: The steps of respectively acquiring link topology information, line parameter information, and linear and nonlinear constraints by using the power network topology model, and constructing a power grid topology environment based on the link topology information, the line parameter information, and the linear and nonlinear constraints include: Acquire the link topology information and the line parameter information between each network node from the power network topology model; Using the power network topology model to obtain the linear and nonlinear constraints of each device in the power network; The link topology information, the line parameter information and the linear and nonlinear constraints are combined to obtain the power grid topology environment.

4. The power network economic dispatching method based on parallel training of PSO and DDPG algorithms according to claim 3 is characterized in that: The steps of respectively obtaining the action space and the state space in the power grid topology environment include: The linear and nonlinear constraints of each device in the power grid topology environment are read to obtain N a Information about schedulable equipment; among them, N a The dispatchable device information includes Power generation equipment, Adjustable load devices, Energy storage devices and A reactive power compensation device; Obtain n from each of the power generation equipment, the adjustable load equipment, the energy storage equipment and the reactive power compensation equipment respectively gen adjustable action dimensions, n l adjustable action dimensions, n sto adjustable action dimensions and n q Adjustable action dimensions; According to the gen adjustable action dimensions, the n l adjustable action dimensions, the n sto adjustable action dimensions and the n q adjustable action dimensions to construct the action space A of the initial DDPG agent * ; The link topology information and the line parameter information in the power grid topology environment are read to obtain N s running status device information; wherein the running status device information includes Line nodes and transmission branches; Obtain n from each of the line nodes and the transmission branches respectively bus observable state dimensions and n bru observable state dimensions; According to the bus observable state dimensions and the n bru observable state dimensions, constructing the state space S of the initial DDPG agent * .

5. The power network economic dispatching method based on parallel training of PSO and DDPG algorithms according to claim 4 is characterized in that: The steps to construct the overall reward function include: The power network system simulation environment is used to calculate the power network flow state at time t, and the state S of each device, node, and branch at time t is obtained. t ; Taking the linear and nonlinear constraints in the power network system as a reference, checking whether the current state violates the linear and nonlinear constraints, and using whether the constraints are violated as the environmental reward R1; Calculate the overall operation cost M of the network under time t t , and weighted calculation to obtain the negative economic reward R2; The final overall reward function R is obtained according to the environmental reward R1 and the economic negative reward R2; wherein R=R1+R2.

6. The method for economic dispatching of power networks based on parallel training of PSO and DDPG algorithms according to claim 5 is characterized in that: The PSO algorithm includes: Define the parameters in the particle swarm optimization algorithm, including: particle initial velocity v i , the initial position of the particle x i , particle's own learning factor c1, global learning factor c2, particle's own random factor r1, global random factor r2, maximum inertia weight ω max , minimum inertia weight ω min and the number of iterations K; The fitness of each position passed by the particle is calculated to obtain the fitness value f(x i ); The fitness value f(x i ) and the historical best fitness value f of this particle i,best Compare, if the fitness value f(x i ) is greater than the historical best fitness f i,best , then the fitness value f(x i ) is assigned to the historical best fitness f i,best , and the particle's optimal fitness position P i,best Modify to current position; Traverse the best fitness f of all particles j,best , and the optimal fitness f j,best and the global optimal fitness value f g,best For comparison, if the optimal fitness f of the kth particle k,best Greater than the global optimal fitness value f g,best ,but: f g,best =f k,best P g,best =P k,best Among them, P g,best is the global optimal fitness position coordinate; According to the current speed v i , the optimal fitness position P of the particle history i,best and the global optimal fitness position coordinate P g,best , update the travel direction and speed information of all particles.

7. The method for economic dispatching of power networks based on parallel training of PSO and DDPG algorithms according to claim 6, characterized in that: The steps of using the PSO algorithm to optimize the number of neural network layers and the number of neurons in each layer of the policy network in the DDPG algorithm to obtain the optimized policy network model include: The number of input neural network layers L and the number of neurons in each layer n corresponding to the current particle position coordinates l The strategy network model is trained, and the total amount of rewards obtained in the preset rounds is used as the fitness to evaluate the current position of the particle to obtain the optimized strategy network model; wherein the reward and the fitness are linearly proportional.

8. The method for economic dispatching of power networks based on parallel training of PSO and DDPG algorithms according to claim 7, characterized in that: The step of optimizing the initial hyperparameters by using the PSO algorithm to obtain target hyperparameters adapted to the policy network model includes: Taking the policy network model as the basic model, the PSO algorithm is used to optimize the initial hyperparameters to obtain the target hyperparameters adapted to the policy network model; wherein the hyperparameters P * Including the number of training rounds Episode, the maximum number of steps per round Step, the learning rate Lr, the discount factor Gamma, the soft update coefficient Tau and the experience replay pool capacity BUFFER_CAPACITY.

9. The method for economic dispatching of power networks based on parallel training of PSO and DDPG algorithms according to claim 8, characterized in that: The step of using the policy network model as the policy network of the initial DDPG agent and training the initial DDPG agent in parallel using the target hyperparameters to obtain a trained target DDPG agent includes: Using the same initial DDPG agent to schedule different independent power network simulation environments; The experiences obtained from each independent environment action are collected respectively, and after reaching a preset number, the initial DDPG agent is trained according to the DDPG algorithm based on the strategy network model and the target hyperparameters; Each of the trained initial DDPG agents is tested and soft-updated and integrated to obtain the target DDPG agent.

Images