Distribution network distributed energy scheduling optimization method and related device

Abstract

The invention belongs to the technical field of energy scheduling optimization, and provides a power distribution network distributed energy scheduling optimization method and a related device, and the method comprises the steps: taking the operation cost of a power distribution network as a target function, taking the operation limitation of all equipment in the power distribution network as a constraint condition, and constructing a power distribution network distributed energy operation optimization problem model; based on a power distribution network distributed energy operation optimization problem model, constructing a multi-agent reinforcement learning agent set, and converting an active power distribution network distributed energy operation optimization problem into a Markov decision process; and a multi-agent reinforcement learning framework is constructed, multi-agent training is carried out, and an optimization strategy is obtained after training is completed. According to the multi-agent training framework, the decision-making process of multiple agents can be processed in parallel, and the time required for strategy generation is greatly shortened. And in a dynamically changing power grid environment, the possibility is provided for immediately adjusting and optimizing an operation strategy, so that the overall operation efficiency of the power distribution network is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of energy dispatch optimization, and in particular relates to a method for optimizing the dispatch of distributed energy in a distribution network and related devices. Background Art

[0002] With the construction of power grids with a high proportion of renewable energy, the diversity and geographical distribution of distributed energy make the power system face greater volatility and challenges. How to improve the renewable energy consumption capacity and dispatch and optimize the operation of distributed energy has become an increasingly important issue for distribution networks. Traditional power grid design relies more on centralized power generation. The optimization of distributed energy operation in distribution networks needs to adapt to the intermittent and volatile nature of distributed energy. The problem of optimizing the operation of distributed energy in distribution networks can be expressed as a mixed integer nonlinear programming problem, which is an NP-hard problem, considering switches, photovoltaic wind power, and energy storage. With the advancement of artificial intelligence technology, artificial intelligence algorithms are expected to play an increasingly important role in distribution network control optimization strategies. It is expected to use data-driven analysis and optimization algorithms to achieve efficient scheduling and operation of distributed energy and maximize the use of renewable energy.

[0003] The multi-agent deep deterministic policy gradient (MADDPG) algorithm is an algorithm proposed by the OpenAI team in 2017 to solve multi-agent problems. It enables multiple agents to make collaborative decisions to complete more complex tasks in a high-dimensional and dynamic environment through communication between agents and interaction between agents and the environment. It is an excellent application of distributed computing methods in the field of multi-agents. In addition, it can also use the observation information of other agents for centralized training. The MADDPG algorithm adopts the idea of ​​a centralized training with decentralized execution framework during training.

[0004] MADDPG is an extension of the DDPG framework algorithm in the field of multi-agents. Each network learns the strategy function and action value function. At the same time, it has a target network and uses the Q-learning algorithm for different strategy learning.

[0005] Each MADDPG agent has its own policy network, which has high computational overhead, especially when the number of agents increases, the training time increases significantly; MADDPG may be low in sample efficiency, especially in the early stages of training, when agents may need more interactions to learn effectively. During the training process, especially in high-dimensional or complex environments, training instability may be a problem, especially in scenarios with strong interactions between agents. The strategy of each MADDPG agent is learned independently, which may lead to poor interpretability of the overall strategy, especially in collaborative tasks.

[0006] DQfD (Deep Q-learning from Demonstrations) is a deep reinforcement learning algorithm first proposed by Todd Hester and others from Google Deep Mind. Its main feature is to apply expert solutions and experience to intelligent agent learning, "guide" the intelligent agent to make actions similar to those of the expert solutions and experience, and achieve the purpose of reducing the exploration cost and training overhead of reinforcement learning and improving the performance of the intelligent agent.

[0007] DQfD requires human demonstration data for training, which may be difficult to obtain in some applications; the training process of DQfD is relatively complex, requiring the simultaneous processing of demonstration data and its own experience, which may make hyperparameter adjustment more difficult; due to the introduction of demonstration data, DQfD may overfit these demonstrations, especially when the amount of demonstration data is insufficient. The single-agent-based method may have the problem of dimensionality curse when dealing with power grid systems with higher system complexity, and therefore it is difficult to adapt to larger-scale distribution network systems. Summary of the invention

[0008] The purpose of the present invention is to provide a method and related devices for optimizing the dispatching of distributed energy in a distribution network, so as to solve the problem of dimensionality disaster that may exist when a single-agent method is used to process a power grid system with high system complexity.

[0009] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for optimizing the dispatching of distributed energy in a distribution network, comprising: Taking the distribution network operation cost as the objective function and the operation restrictions of each device in the distribution network as the constraints, a distribution network distributed energy operation optimization problem model is constructed; Based on the distributed energy operation optimization problem model of the distribution network, a multi-agent reinforcement learning agent set is constructed to transform the distributed energy operation optimization problem of the active distribution network into a Markov decision process. Build a multi-agent reinforcement learning framework to train multiple agents. After the training is completed, the optimization strategy is obtained.

[0010] Optionally, taking the distribution network operation cost as the objective function specifically includes: Define the total cost objective function of the distribution network in the entire time range. Specifically, take 24h as the analysis period and 1h as the time granularity. The objective function is:

[0011] The total cost function of the distribution network is , is the electricity price at time t, is the load active power, , , , They are the costs of photovoltaic power generation unit, wind turbine power generation unit, energy storage unit and external grid power supply at corresponding powers.

[0012] Optionally, the constraint condition of using the operation limit of each device in the distribution network as a constraint condition specifically includes: Equality constraints: The network parameters are , The network and power flow constraints of the power system are:

[0013]

[0014] Among them, for node i, is the injected complex power, is the node voltage; is the ij node phase difference; For node i:

[0015]

[0016] in , The active and reactive loads carried by the node; For energy storage, when charging:

[0017] When discharging:

[0018] The energy storage state should be consistent at the beginning and end of a cycle:

[0019] Inequality constraints: For the network, the node voltage and branch current should be kept within a certain range:

[0020]

[0021] For photovoltaic power generation units, ignoring their reactive power:

[0022] For wind turbine power generation units:

[0023]

[0024] For energy storage units, the energy storage state has upper and lower limits: .

[0025] Optionally, the agent set of multi-agent reinforcement learning is constructed based on the distribution network distributed energy operation optimization problem model, including: The set of state spaces, the set of action spaces, the set of observation spaces, and the reward function for agent reinforcement learning.

[0026] Optional, specific: Divide all devices with action behavior states on the power distribution network, and set up an intelligent agent model for each device: switch, photovoltaic power generation unit, wind turbine power generation unit, energy storage unit; Collect the distribution network operation status information and construct a global state space set, which includes the network node voltage, line current, switch status, restored load power, photovoltaic maximum power, wind turbine maximum power, and energy storage status at the time node; By mapping the action behaviors of the intelligent agent, the action space of the intelligent agent is constructed; the switch action space is {0,1}, that is, whether it is turned on or not, and the switch controlled by the intelligent agent that controls the switch is divided according to the topological relationship and the station relationship; for the photovoltaic active power, wind turbine active and reactive power, and energy storage operating power with continuous action space, they are discretized and normalized according to the maximum output power, that is, [0,1]; The state space that each intelligent agent can observe is set. For switches, the state information of the station area managed by the observation is observed, including the current state of the entire microgrid and the activity state of each intelligent agent; for photovoltaic power generation units, wind turbine power generation units, and energy storage units, their observation space is the state information of the microgrid where they are located and the activity state of each intelligent agent under the microgrid; Setting up reward functions for multi-agent reinforcement learning, immediate rewards Set to the running profit at time t and the power grid operation over-limit penalty function sum;

[0027]

[0028]

[0029]

[0030] In the formula, , are reference voltage and reference current, and a, b, c, d are weight parameters.

[0031] Optionally, the multi-agent reinforcement learning framework is constructed to train the multi-agents, and after the training is completed, an optimization strategy is obtained, including: The MADDPG algorithm is used to build a multi-agent reinforcement learning framework, and a pre-training process is introduced. After pre-training, an experience replay pool is built, and data is randomly sampled from the experience replay pool to complete the update of all agents.

[0032] Optional, specific: Construct a teaching experience pool and pre-train the switch agent. First, based on the area divided by each switch agent and the distribution network topology, traverse to obtain all switch action groups that can complete the power supply of each node without electromagnetic ring network as the teaching experience pool; then pre-train and repeat. Second-rate; Construct an experience replay pool; for each agent i, according to the current strategy and observation Select Action ; Execute action And get the r value and the new state x Will Store it in the experience pool D, and repeat until the number in the experience playback pool reaches a predetermined value; Randomly sample data from the experience replay pool, use the action values ​​and observation value sets of all agents as the Critic network input to get the Q value, then use the next action value and the next observation value set of all agents as the Critic target network input to get the target value, and then substitute it into the following formula for gradient descent;

[0033]

[0034] in, For the target network; The Actor network update uses the Q value output by the updated Critic network to perform gradient ascent according to the following formula.

[0035]

[0036] According to the following formula, the parameters of the Actor and Critic networks are soft updated;

[0037] Repeat until all agents have completed the update, until the target number of rounds is reached, and the training is completed to obtain the optimized strategy.

[0038] In a second aspect, the present invention provides a distributed energy dispatch optimization system for a distribution network, comprising: The model building module is used to construct a distribution network distributed energy operation optimization problem model with the distribution network operation cost as the objective function and the operation restrictions of each device in the distribution network as the constraint conditions; The problem conversion module is used to build a multi-agent reinforcement learning agent set based on the distributed energy operation optimization problem model of the distribution network, and convert the distributed energy operation optimization problem of the active distribution network into a Markov decision process; The training output module is used to build a multi-agent reinforcement learning framework and conduct multi-agent training. After the training is completed, the optimization strategy is obtained.

[0039] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a distribution network distributed energy scheduling optimization method when executing the computer program.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a method for optimizing the dispatching of distributed energy in a distribution network are implemented.

[0041] Compared with the prior art, the present invention has the following technical effects: The present invention integrates the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm, and achieves a significant acceleration of strategy generation for the operation optimization problem of distributed energy in distribution networks. Compared with traditional methods that rely on artificial experience or single-agent algorithms, the multi-agent training framework of the present invention can process the decision-making process of multiple agents in parallel, greatly shortening the time required for strategy generation. This breakthrough not only meets the distribution network's stringent requirements for real-time strategies, but also provides the possibility for immediate adjustment and optimization of operation strategies in a dynamically changing power grid environment, thereby effectively improving the overall operation efficiency of the distribution network.

[0042] By innovatively introducing a pre-training mechanism, the present invention has achieved a qualitative leap in the performance and stability of agent strategy generation. During the pre-training phase, the agent can learn and accumulate experience in advance in a simulated environment, which greatly shortens the learning curve during formal training and enables the agent to master efficient strategies more quickly. At the same time, pre-training enhances the stability of learning, reduces fluctuations in the strategy exploration process, accelerates convergence, and reduces the cost of strategy learning. In addition, this mechanism also reduces the reliance on high-performance computing resources, making this method more economical and feasible in practical applications.

[0043] The present invention fully considers the diversity and complexity of distributed energy in the distribution network at the beginning of its design, and ensures the high adaptability and flexibility of the method by constructing a comprehensive and sophisticated optimization problem model and constraints. Whether it is the intermittent nature of renewable energy such as wind energy and solar energy, or the charging and discharging characteristics of energy storage equipment, the present invention can respond flexibly and provide customized solutions for optimizing the operation of distribution networks in different scenarios. This wide applicability makes the present invention a powerful tool for meeting the challenges of future power grid diversification.

[0044] While pursuing optimization efficiency, the present invention always puts the safe and stable operation of the power grid first. By carefully designing constraints and reward mechanisms, the present invention guides the intelligent agent to strictly abide by the safety regulations of the power grid during the optimization process, effectively avoiding the stability problems of the power grid caused by over-optimization. This safety-oriented optimization strategy not only ensures the physical safety of the power grid, but also maintains the stability and reliability of the power supply, providing users with high-quality power services.

[0045] In summary, the active distribution network distributed energy operation optimization method based on multi-agent reinforcement learning proposed in this invention not only shows excellent technical effects in terms of strategy generation speed, strategy performance and stability, system adaptability and flexibility, but also makes important contributions to ensuring the safe and stable operation of the power grid. This innovative method not only opens up a new path for the operation optimization of the distribution network, but also provides strong technical support for the development of smart grids. Its broad application prospects and promotion value are self-evident. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the present invention.

[0047] Figure 2 This is a logic block diagram of the present invention.

[0048] Figure 3 This is the MADDPG algorithm diagram of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings: Example 1, please refer to Figure 1 The present invention provides a method for optimizing the dispatching of distributed energy in a distribution network, comprising: Taking the distribution network operation cost as the objective function and the operation restrictions of each device in the distribution network as the constraints, a distribution network distributed energy operation optimization problem model is constructed; Based on the distributed energy operation optimization problem model of the distribution network, a multi-agent reinforcement learning agent set is constructed to transform the distributed energy operation optimization problem of the active distribution network into a Markov decision process. Build a multi-agent reinforcement learning framework to train multiple agents. After the training is completed, the optimization strategy is obtained.

[0050] First, the distribution network operation cost is taken as the objective function, which means that all optimization efforts are aimed at reducing operating costs and improving economic benefits. Through accurate modeling and solving, efficient use of resources can be achieved, unnecessary waste can be reduced, and the purpose of reducing operating costs can be achieved.

[0051] The scheme uses the operating limits of each device in the distribution network as constraints, which ensures that the optimization process does not exceed the physical and safety limits of the equipment. The setting of such constraints helps prevent equipment from being overloaded, damaged, or causing safety accidents, thereby ensuring the stable operation of the distribution network and the safety of electricity use for users.

[0052] By transforming the distributed energy operation optimization problem of active distribution network into a Markov decision process (MDP), the solution uses the powerful solving ability of reinforcement learning algorithm to deal with this complex problem. The introduction of MDP makes the problem solvable and can find the approximate optimal solution through iterative learning, which greatly improves the efficiency and accuracy of solving the optimization problem.

[0053] Building a multi-agent reinforcement learning framework is one of the core of the solution. Multi-agent training enables each agent to learn the optimal strategy for its responsible part, and these strategies can work together to achieve the global optimum. This collaborative optimization mechanism helps to fully consider the mutual influence and constraints between various energy sources in distributed energy scheduling, so as to formulate a more comprehensive and effective optimization strategy.

[0054] After multi-agent training, the scheme can finally obtain a set of implementable optimization strategies. These strategies are based on actual distribution network operation data and constraints, so they are highly feasible and practical. Implementing these strategies can significantly improve the operating efficiency of the distribution network, reduce operating costs, and ensure the safe and stable operation of the power grid.

[0055] This technical solution achieves effective optimization of the distribution network operation cost by constructing a distributed energy operation optimization problem model for the distribution network and solving it using a multi-agent reinforcement learning algorithm. At the same time, the solution also ensures the safety and compliance of equipment operation, improves the efficiency and accuracy of solving the optimization problem, and ultimately obtains a set of implementable optimization strategies. These technical effects together constitute the significant advantages of the solution in the field of distributed energy dispatch optimization in the distribution network.

[0056] Embodiment 2, the present invention provides a method for optimizing the dispatching of distributed energy in a distribution network, specifically comprising: Step 1: Construction of the optimization problem model of distributed energy operation in active distribution network; First, Figure 2 As shown in the left part, the comprehensive operating cost in the entire time range is used as the objective function. Then, considering the operating restrictions of each device in the network as constraints, the distributed energy operation optimization problem of the distribution network is converted into a mixed integer nonlinear programming problem, and the construction of the distributed energy operation optimization problem model of the distribution network is completed.

[0057] like Figure 2 As shown, the step 1 is specifically as follows: Step 1.1: Define the total cost objective function of the distribution network in the entire time range. Specifically, take 24h as the analysis period and 1h as the time granularity, and the objective function is:

[0058] The total cost function of the distribution network is , is the electricity price at time t, is the load active power, , , , They are the costs of photovoltaic power generation unit, wind turbine power generation unit, energy storage unit and external grid power supply at corresponding powers.

[0059] Step 1.2: After the objective function of the optimization problem of distributed energy operation in the distribution network is constructed, the constraints are constructed.

[0060] Equality constraints: The network parameters are , The network considers the power flow constraints of the power system:

[0061]

[0062] Among them, for node i, is the injected complex power, is the node voltage; is the ij node phase difference.

[0063] For node i:

[0064]

[0065] in , It is the active and reactive power of the load carried by the node.

[0066] For energy storage, when charging:

[0067] When discharging:

[0068] The energy storage state should be consistent at the beginning and end of a cycle:

[0069] Inequality constraints: For the network, the node voltage and branch current should be kept within a certain range:

[0070]

[0071] For photovoltaic power generation units, ignoring their reactive power:

[0072] For wind turbine power generation units:

[0073]

[0074] For energy storage units, the energy storage state has upper and lower limits:

[0075] Step 2: If Figure 2 As shown in the middle part, based on the established mathematical model, a set of agents for multi-agent reinforcement learning, the environment state space, the agent observation space and the agent reward mechanism are constructed to transform the distributed energy operation optimization problem of the active distribution network into a Markov decision process.

[0076] like Figure 2 As shown in the middle part, the step 2 is specifically as follows: Step 2.1: Divide all devices with action behavior states on the network, and set up an intelligent agent model for each device: switch, photovoltaic power generation unit, wind turbine power generation unit, energy storage unit.

[0077] Step 2.2: Collect the distribution network operation status information and construct a global state space set. This includes the network node voltage, line current, switch status, restored load power, photovoltaic maximum power, wind turbine maximum power, and energy storage status at the time node. For the problem that the energy storage status should be consistent at the beginning and end of a cycle, it is advisable to set the energy storage status at the beginning of the cycle to the lowest state.

[0078] Step 2.3: Construct the action space of the agent by mapping the action behavior of the agent. The switch action space is {0,1}, that is, whether it is turned on or not. Considering that the actual power grid does not allow electromagnetic ring network and the coordination between switches, the switches controlled by the agent that controls the switch are divided according to the topological relationship and the station relationship; for the photovoltaic active power, wind turbine active and reactive power, and energy storage operating power with continuous action space, they are discretized and normalized according to the maximum output power, that is, [0,1].

[0079] Step 2.4: Set the state space that each agent can observe. For switches, the state information of the managed station area can be observed, including the current state of the entire microgrid and the activity state of each agent; for photovoltaic power generation units, wind turbine power generation units, and energy storage units, their observation space is the state information of the microgrid and the activity state of each agent under the microgrid.

[0080] Step 2.5: Set the reward function for multi-agent reinforcement learning. The goal of the agent is to improve the economic benefits of the operation of the distribution network containing distributed energy while ensuring the constraints of the power grid. All agents are in a cooperative relationship and share rewards. Instant Reward Set to the running profit at time t and the power grid operation over-limit penalty function sum.

[0081]

[0082]

[0083]

[0084]

[0085] In the formula, , are reference voltage and reference current, a, b, c, d are weight parameters Step 3: Use the MADDPG algorithm to build a multi-agent reinforcement learning framework and introduce a pre-training process to improve the convergence speed and performance of the agents. Figure 3 As shown in the figure. MADDPG consists of Actor and Critic neural networks, each of which uses a dual network model similar to DQN. By introducing a "common experience replay pool" and a "common Critic network", the training cost is reduced and the model efficiency is improved. Due to the different types of agents, the experience replay pool and Critic network should be classified according to the type of agent.

[0086] Step 3 is as follows: Step 3.1: Construct a teaching experience pool and pre-train the switch agent. First, based on the area divided by each switch agent and the distribution network topology, traverse and obtain all switch action groups that can complete the power supply of each node without an electromagnetic ring network as the teaching experience pool. Then perform pre-training and repeat steps 3.2 and 3.3. Second-rate Step 3.2: Construct an experience replay pool; for each agent i, based on the current strategy and observation Select Action ; Execute action And get the r value and the new state x Will Store in experience pool D. Repeat the above steps until the number of experience replay pools reaches the preset value.

[0087] Step 3.3: Randomly sample data from the experience replay pool; use the action values ​​and observation value sets of all agents as the Critic network input to obtain the Q value, and then use the next action value and the next observation value set of all agents as the Critic target network input to obtain the target value, and then substitute it into the following formula for gradient descent.

[0088]

[0089]

[0090] in, The target network.

[0091] The Actor network update uses the Q value output by the updated Critic network to perform gradient ascent according to the following formula.

[0092]

[0093] According to the following formula, the parameters of the Actor and Critic networks are soft updated.

[0094]

[0095] Repeat the above steps until all agents have completed the update.

[0096] Step 3.4: Repeat steps 3.2 and 3.3 until the target number of rounds is reached and the training is completed. After the training is completed, each agent is put into use.

[0097] Table 1 Flowchart

[0098] The present invention provides a method for optimizing the dispatching of distributed energy in a distribution network, the technical principle of which is mainly based on the application of mixed integer nonlinear programming (MINLP), multi-agent reinforcement learning (MARL) and MADDPG algorithm.

[0099] The objective function is to minimize the comprehensive operating cost over the entire time frame (e.g., 24 hours), taking into account the costs of electricity price, load power, photovoltaic power generation, wind turbine power generation, energy storage unit, and external grid power supply.

[0100] Constraints: including equality constraints (such as power system flow constraints, node power balance, and state changes of energy storage units) and inequality constraints (such as voltage and current limits, maximum output power limits of distributed energy resources, and upper and lower limits of the state of energy storage units).

[0101] Intelligent agent models are set up for key equipment in the distribution network (such as switches, photovoltaic power generation units, wind turbine power generation units, and energy storage units), and each intelligent agent is responsible for controlling the action behavior of the corresponding equipment.

[0102] Collect the operating status information of the distribution network, construct a global state space set, and set the state space that can be observed according to the type of intelligent agent.

[0103] According to the action behavior characteristics of the equipment controlled by the intelligent agent, the corresponding action space is constructed (such as the switching action of the switch, the active / reactive output of distributed energy and energy storage units).

[0104] The reward function is set to reflect the performance of the intelligent agent in optimizing the operating cost of the distribution network and improving the economic benefits, while taking into account the security and stability constraints of the grid operation.

[0105] MADDPG algorithm application: Algorithm structure: The MADDPG algorithm consists of Actor and Critic neural networks, and realizes collaborative learning and optimization among multiple agents through "common experience replay pool" and "common Critic network".

[0106] Pre-training process: Pre-train the switch agent by building a teaching experience pool to improve the initial strategy quality of the agent and accelerate the convergence speed.

[0107] Training process: Use the experience replay pool to store the agent's experience data and perform strategy updates through random sampling. The Actor network updates the strategy parameters through gradient ascent, and the Critic network optimizes the value function through gradient descent. At the same time, a soft update strategy is used to smoothly update the parameters of the Actor and Critic networks.

[0108] By accurately modeling and solving mixed integer nonlinear programming problems, the present invention can find the optimal operation strategy under the conditions of satisfying the operation constraints of the power grid, and significantly reduce the comprehensive operation cost of the distribution network.

[0109] By utilizing a multi-agent reinforcement learning algorithm, the present invention can process the decision-making process of multiple agents in parallel, accelerate the strategy generation speed, and meet the real-time requirements of the distribution network.

[0110] By introducing the pre-training process and the multi-agent collaborative optimization mechanism, the present invention can improve the performance and stability of the agent strategy and ensure the effectiveness and reliability of the optimization strategy in actual operation.

[0111] The present invention takes into account the operating characteristics and constraints of various distributed energy sources in the distribution network, and constructs a comprehensive optimization problem model and constraint conditions, so that the method can adapt to the distribution network operation optimization needs in different scenarios and enhance the adaptability and flexibility of the system.

[0112] During the optimization process, the present invention fully considers the safe and stable operation requirements of the power grid. By setting reasonable constraints and reward mechanisms, it guides the intelligent agent to perform operation optimization under the premise of ensuring the safety of the power grid, thus avoiding potential threats and damages to the power grid.

[0113] In summary, the distributed energy scheduling optimization method for distribution network proposed in the present invention is highly innovative and practical in technical principles, and can significantly improve the operating efficiency and economic benefits of the distribution network while ensuring the safe and stable operation of the power grid.

[0114] In another embodiment of the present invention, a distributed energy dispatch optimization system for a distribution network is provided, which can be used to implement the above-mentioned distributed energy dispatch optimization method for a distribution network. Specifically, the system includes: The model building module is used to construct a distribution network distributed energy operation optimization problem model with the distribution network operation cost as the objective function and the operation restrictions of each device in the distribution network as the constraint conditions; The problem conversion module is used to build a multi-agent reinforcement learning agent set based on the distributed energy operation optimization problem model of the distribution network, and convert the distributed energy operation optimization problem of the active distribution network into a Markov decision process; The training output module is used to build a multi-agent reinforcement learning framework and conduct multi-agent training. After the training is completed, the optimization strategy is obtained.

[0115] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0116] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a method for optimizing the dispatch of distributed energy in a distribution network.

[0117] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of a method for optimizing the dispatch of distributed energy resources in a distribution network in the above embodiment.

[0118] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing the dispatching of distributed energy in a distribution network, characterized in that: include: Taking the distribution network operation cost as the objective function and the operation restrictions of each device in the distribution network as the constraints, a distribution network distributed energy operation optimization problem model is constructed; Based on the distributed energy operation optimization problem model of the distribution network, a multi-agent reinforcement learning agent set is constructed to transform the distributed energy operation optimization problem of the active distribution network into a Markov decision process. Build a multi-agent reinforcement learning framework to train multiple agents. After the training is completed, the optimization strategy is obtained.

2. A method for optimizing the dispatch of distributed energy in a distribution network according to claim 1, characterized in that: The objective function of taking the distribution network operation cost as the objective function specifically includes: Define the total cost objective function of the distribution network in the entire time range. Specifically, take 24h as the analysis period and 1h as the time granularity. The objective function is: The total cost function of the distribution network is , is the electricity price at time t, is the load active power, , , , They are the costs of photovoltaic power generation unit, wind turbine power generation unit, energy storage unit and external grid power supply at corresponding powers.

3. A method for optimizing the dispatch of distributed energy in a distribution network according to claim 2, characterized in that: The constraints of the operation restrictions of each device in the distribution network are specifically as follows: Equality constraints: The network parameters are , The network and power flow constraints of the power system are: Among them, for node i, is the injected complex power, is the node voltage; is the ij node phase difference; For node i: in , The active and reactive loads carried by the node; For energy storage, when charging: When discharging: The energy storage state should be consistent at the beginning and end of a cycle: Inequality constraints: For the network, the node voltage and branch current should be kept within a certain range: For photovoltaic power generation units, ignoring their reactive power: For wind turbine power generation units: For energy storage units, the energy storage state has upper and lower limits: 。 4. A method for optimizing the dispatch of distributed energy in a distribution network according to claim 1, characterized in that: The above-mentioned intelligent agent set for multi-agent reinforcement learning is constructed based on the distributed energy operation optimization problem model of the distribution network, including: The set of state spaces, the set of action spaces, the set of observation spaces, and the reward function for agent reinforcement learning.

5. A method for optimizing the dispatch of distributed energy in a distribution network according to claim 4, characterized in that: Specific: Divide all devices with action behavior states on the power distribution network, and set up an intelligent agent model for each device: switch, photovoltaic power generation unit, wind turbine power generation unit, energy storage unit; Collect the distribution network operation status information and construct a global state space set, which includes the network node voltage, line current, switch status, restored load power, photovoltaic maximum power, wind turbine maximum power, and energy storage status at the time node; By mapping the action behaviors of the intelligent agent, the action space of the intelligent agent is constructed; the switch action space is {0,1}, that is, whether it is turned on or not, and the switch controlled by the intelligent agent that controls the switch is divided according to the topological relationship and the station relationship; for the photovoltaic active power, wind turbine active and reactive power, and energy storage operating power with continuous action space, they are discretized and normalized according to the maximum output power, that is, [0,1]; The state space that each intelligent agent can observe is set. For switches, the state information of the station area managed by the observation is observed, including the current state of the entire microgrid and the activity state of each intelligent agent; for photovoltaic power generation units, wind turbine power generation units, and energy storage units, their observation space is the state information of the microgrid where they are located and the activity state of each intelligent agent under the microgrid; Setting up reward functions for multi-agent reinforcement learning, immediate rewards Set to the running profit at time t and the power grid operation over-limit penalty function sum; In the formula, , are reference voltage and reference current, and a, b, c, d are weight parameters.

6. A method for optimizing the dispatch of distributed energy in a distribution network according to claim 1, characterized in that: The multi-agent reinforcement learning framework is constructed to train the multi-agents. After the training is completed, an optimization strategy is obtained, including: The MADDPG algorithm is used to build a multi-agent reinforcement learning framework, and a pre-training process is introduced. After pre-training, an experience replay pool is built, and data is randomly sampled from the experience replay pool to complete the update of all agents.

7. A method for optimizing the dispatch of distributed energy in a distribution network according to claim 6, characterized in that: Specific: Construct a teaching experience pool and pre-train the switch agent. First, based on the area divided by each switch agent and the distribution network topology, traverse and obtain all switch action groups that can complete the power supply of each node without electromagnetic ring network as the teaching experience pool; Then perform pre-training and repeat Second-rate; Build an experience replay pool; For each agent i, according to the current strategy and observation Select Action ; Execute action And get the r value and the new state x Will Store it in the experience pool D, and repeat until the number in the experience playback pool reaches a predetermined value; Randomly sample data from the experience replay pool, use the action values ​​and observation value sets of all agents as the Critic network input to get the Q value, then use the next action value and the next observation value set of all agents as the Critic target network input to get the target value, and then substitute it into the following formula for gradient descent; in, For the target network; The Actor network update uses the Q value output by the updated Critic network to perform gradient ascent according to the following formula; According to the following formula, the parameters of the Actor and Critic networks are soft updated; Repeat until all agents have completed the update, until the target number of rounds is reached, and the training is completed to obtain the optimized strategy.

8. A distributed energy dispatch optimization system for a distribution network, characterized in that: include: The model building module is used to construct a distribution network distributed energy operation optimization problem model with the distribution network operation cost as the objective function and the operation restrictions of each device in the distribution network as the constraint conditions; The problem conversion module is used to build a multi-agent reinforcement learning agent set based on the distributed energy operation optimization problem model of the distribution network, and convert the distributed energy operation optimization problem of the active distribution network into a Markov decision process; The training output module is used to build a multi-agent reinforcement learning framework and conduct multi-agent training. After the training is completed, the optimization strategy is obtained.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for optimizing the dispatching of distributed energy in a distribution network as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a distribution network distributed energy scheduling optimization method as described in any one of claims 1 to 7 are implemented.

Images