A multi-power router system energy scheduling method

Abstract

The present application belongs to the technical field of power optimization scheduling, and relates to a multi-electric energy router system energy scheduling method, which comprises the following steps: constructing a multi-electric energy router system model; based on the multi-electric energy router system model, performing hierarchical optimization scheduling by using a hierarchical scheduling strategy with the lowest system operation cost as the target, first performing single electric energy router energy optimization scheduling at the electric energy router layer, and then performing global optimization scheduling between electric energy routers at the distribution network layer. The present application makes the result of the hierarchical optimization scheduling of the multi-electric energy router system better than the optimization scheduling of a single electric energy router in the system, and improves the distributed energy utilization rate and the stability of the power grid operation when connected to the grid. In terms of the algorithm, the training quality of the network is improved by changing the range of the selected sample state space, the correlation between the selected samples is reduced, the instability of the training process is alleviated, the training efficiency and the distributed energy utilization rate can be effectively improved, and the system operation cost is reduced.

Description

Technical Field

[0001] This invention belongs to the field of power optimization scheduling technology, and specifically relates to an energy scheduling method for a multi-energy router system. Background Technology

[0002] The Energy Internet is a shared network that connects distributed energy devices and users through energy internet and information technologies. As a core device of the Energy Internet, the power router can collect and analyze energy information, make intelligent decisions, and facilitate multi-directional energy transmission. By acquiring output information from distributed energy sources and user energy demand information, the power router performs optimized energy scheduling, achieving efficient energy utilization while meeting user electricity needs.

[0003] Because distributed energy resources suffer from limited power generation and insufficient anti-interference capabilities, it is necessary to interconnect nearby power routers in adjacent areas to form a multi-power router system in order to alleviate these problems. By promoting energy sharing among the power routers within the system, the reliability of energy supply and the stability of system operation can be improved while reducing system energy costs. However, more complex and difficult-to-coordinate interactions can arise between multiple power routers. Therefore, based on the internal control of a single power router, the coordinated control among multiple power routers has become a key research focus in this field.

[0004] In existing technologies, considering power loss during energy transmission, ant colony optimization (ACO) is used to solve for the optimal transmission path. This effectively reduces power loss during transmission while achieving energy exchange between systems, thus saving costs. However, it's important to consider the possibility that power loss may exceed the amount of energy exchanged, meaning power loss occurs even without power interaction. To avoid this, the exchange threshold is tiered. Energy exchange occurs when the interaction power is greater than the threshold and also greater than the transmission loss; otherwise, the devices operate independently. An energy management strategy based on this tiered particle swarm optimization algorithm reduces frequent energy exchanges and lowers system operating costs. Considering that the economic efficiency and stability of the system are influenced to some extent by the capacity of the energy storage devices, a power constraint model is constructed based on the economic advantages of different types of energy storage systems. The gray wolf-sine / cosine optimization algorithm is used to solve the model, yielding an operation control strategy and an optimal configuration method for the energy storage system. Reasonable configuration of energy storage capacity enhances system stability and reduces costs. While these algorithms address several aspects of energy management, they still have limitations due to their inability to perform precise modeling.

[0005] With the development of artificial intelligence applications in power system optimization and control, methods using deep Q-network (DQN) algorithms for voltage optimization control of power routers have emerged. However, the Q-value in this algorithm is often overestimated, affecting the stability of the solution. To address the instability of the DQN algorithm, a dueling DQN approach has been developed, combined with an experience replay mechanism that integrates random, proximity, and priority sampling methods to solve energy management strategies for multi-campus power routers. However, this algorithm suffers from the curse of dimensionality when applied to continuous states or large state spaces. To improve the efficiency of processing large amounts of information and achieve continuous optimization, a deep deterministic policy gradient (DDPG) algorithm combining convolutional neural networks and deterministic policy gradient (DPG) algorithms has emerged to solve the coordinated optimization operation problem of active distribution networks. However, this algorithm does not consider the temporal correlation between sample empirical values. To overcome temporal correlation and improve the speed of experience sampling, a Sumtree data structure is used to store experience, while importance sampling (IS) is introduced to mitigate the impact of biases caused by prioritizing experience on the state distribution. In the aforementioned energy scheduling process, if a power router fails, the entire system will malfunction. To address this issue, a multi-agent reinforcement learning framework is applied to generate independent real-time market decisions. A method using multi-agent deep Q-network (MADQN) for energy internet power scheduling is proposed, which includes an optimal equilibrium selection (OES) mechanism to ensure fairness in the revenue of each device. This algorithm is a distributed execution approach, compensating for the shortcomings of centralized control, but it cannot handle high-dimensional continuous action spaces. To address this problem, the multi-agent deep deterministic policy gradient (MADDPG) algorithm is used to obtain optimal power scheduling strategies and improve the utilization rate of distributed energy. Based on the MADDPG algorithm, the concept of marginal contribution is introduced to evaluate scheduling actions, thereby reducing energy costs. In the process of system energy management, while considering the economy and stability of system operation, it is also necessary to meet the needs of individual power routers to the greatest extent. People further distribute and hierarchize the power grid system, use the MADDPG algorithm to realize the system's energy management, and build a two-level power trading model on the basis of the above, with the goal of minimizing the operating cost of each level, to achieve optimized power dispatch.The MADDPG algorithm employs an experience replay technique during model training, and the historical experience data cached in the experience replay pool is sampled with equal probability. Since different experience data have varying information value and contribute differently to policy improvement, probabilistic data sampling affects the algorithm's convergence speed. To address this issue, some researchers have proposed combining the prioritized experience replay (PER) mechanism with the MADDPG algorithm to solve the electricity price model within the system, achieving higher overall system revenue. While combining the MADDPG algorithm with PER can improve training efficiency to some extent, maintaining and updating the experience priority information increases computational complexity, especially when the experience replay pool is large. This leads to increased computational complexity during training, thereby reducing training speed.

[0006] Therefore, there is an urgent need for those skilled in the art to develop a new energy scheduling method for multi-energy router systems. Summary of the Invention

[0007] To address the aforementioned issues, the inventors propose a multi-agent deep deterministic policy gradient of state spaces scaling (MADDPG-SSS) algorithm. In this algorithm, the network training phase improves training quality by altering the state space range of samples selected from the experience pool. This reduces sample correlation and alleviates instability during training, thereby enhancing the algorithm's training efficiency and performance.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solution: A method for energy scheduling in a multi-energy router system includes the following steps: S1. Construct a multi-energy router system model; S2. Based on the multi-power router system model, a hierarchical scheduling strategy with the goal of minimizing system operating costs is used for hierarchical optimization scheduling. First, energy optimization scheduling of individual power routers is performed at the power router layer, and then global optimization scheduling among power routers at the distribution network layer is performed. The two scheduling strategies are both independent and interrelated. The hierarchical optimization scheduling adopts a Markov decision process and uses the MADDPG algorithm to learn autonomous behavior decisions. Finally, the state space of the training sample is gradually expanded by state space scaling to form a multi-agent deep deterministic policy gradient algorithm based on state space scaling (referred to as the MADDPG-SSS algorithm).

[0009] Preferably, in step S1, in order to improve the absorption rate of distributed energy in terms of space and time, interconnection lines are used to connect power routers located close to each other in adjacent areas. A multi-power router system model is constructed based on the hierarchical scheduling concept. The multi-power router system model includes an access layer, an interconnection layer, and an information layer, wherein: The access layer includes photovoltaic power generation equipment, wind turbine power generation equipment, loads, energy storage equipment and power grid, which are directly connected to the power router. The power router, as the core of the local controller, monitors the operating status of photovoltaic, wind turbine, load and energy storage equipment in real time and rationally dispatches energy to ensure the safe operation of the equipment. The interconnection layer enables energy transfer between power routers through cables, thereby reducing the impact of uncertainties in distributed energy generation and randomness in user electricity consumption, and improving the utilization rate of distributed energy. The information layer realizes the function of information exchange through the energy management center. Each power router uploads energy information to the energy management center. The energy management center adjusts the operating parameters based on the operating status. The power router dynamically adjusts the energy by changing the energy flow route according to the instructions issued by the energy management center, so as to ensure the power demand of each part.

[0010] It should be noted that the access layer, interconnection layer, and information layer in this invention are understood to describe the structural layering of a multi-power router system, such as energy generation, transmission, and information transmission. The power router layer and distribution network layer can be understood as how power is distributed, transmitted, and managed in a multi-power router system. The power router layer is responsible for the flow and management of power between the connected devices of the power routers, while the distribution network layer involves the transmission of energy between power routers and between the power routers and the power grid.

[0011] Preferably, in step S2, the power router, as the underlying device of the multi-power router system model, needs to independently control the devices directly connected to it, in addition to executing the instructions issued by the energy management center. To achieve the lowest total system operating cost, the operating cost of each power router's connected devices must be minimized. When obtaining the energy management strategy, the objective function for energy optimization scheduling of a single power router at the power router layer is as follows: (1) In equation (1), For power routers The equipment operating costs, For power routers The equipment operation and maintenance costs, For power routers Cost of loss of connected energy storage devices.

[0012] Preferably, the objective function for energy optimization scheduling of a single power router at the power router layer is constrained by power balance, i.e. Time-of-use power router Supply and demand maintain a dynamic balance, as shown in the following formula: (2) In equation (2), For power routers Purchase power from other power routers. For power routers The power output of the routers sold to other power-powered routers. For the electrical energy traded with the grid, for t Time-of-use power router The electrical power consumed by the connected load, , , They provide electrical power to photovoltaic power generation, wind turbine power generation, and energy storage systems, respectively. The output power of wind turbine power generation and photovoltaic power generation is affected by wind speed and sunlight intensity at different times, and their output power has a maximum and a minimum value, as shown in equations (3) and (4): (3) (4) In equation (3), and For power routers Connect the upper and lower limits of the wind turbine's power output; in equation (4), and For power routers Connects the upper and lower limits of photovoltaic power generation output power; Considering the wear and tear during equipment operation, Remaining power of time-limited energy storage system State of charge As shown in equations (5) and (6), to prevent overcharging and discharging, the state of charge is constrained as shown in equation (7): (5) (6) (7) In equation (5), It is the self-discharge rate of the energy storage system. The charging and discharging time of the energy storage system. It is the charging and discharging power of the energy storage system; in equation (6), The rated capacity of the energy storage system; in equation (7), and These are the maximum and minimum storage capacities of the energy storage system, respectively.

[0013] Preferably, in step S2, during each time period, each power router uploads the scheduled data information to the energy management center. The energy management center comprehensively considers the status of each power router and the total system operating cost. To achieve the lowest possible power dispatch, only power transmission losses between power routers and between power routers and the grid need to be considered. The objective function for global optimization dispatching among power routers in the distribution network layer is as follows: (8) In equation (8), and These are power routers With power router Transmission losses that occur when power is exchanged with the power grid and These are power routers The cost of purchasing electricity and the revenue generated from selling electricity.

[0014] Preferably, when the energy management center performs energy dispatch, it must not only meet the actual operating conditions of each power router, but also consider the real-time operating constraints between power routers and between power routers and the power grid. During the power exchange and transmission process, power loss will occur due to the influence of resistance and wire length, as shown in equation (9): (9) In equation (9), For power loss during transmission, For power routers and Interactive energy power, For power routers and The resistance of the transmission line between them For power routers and The distance between the transmission lines, For transmission line parameters, it refers to the magnitude of the voltage driven by the current in the current state; Because the current carrying capacity of a transmission line is limited, power routers... and power router The power of the transmission line between them satisfies the following formula: (10) In equation (10), and These are the minimum and maximum transmission power on the power router and transmission line, respectively; To facilitate power exchange between power routers and reduce power exchange with the power grid, power routers... Electricity price constraints are shown in the following formula: (11) In equation (11), and They are respectively The price of electricity purchased and sold by the power grid during specific time periods. and They are respectively Time-of-use power router Prices for buying and selling electricity; If the supply and demand of the multi-energy router system are unbalanced, it can purchase electricity from the grid to meet the load demand. The power constraint for the energy transaction between the energy router and the grid is shown in the following formula: (12) (13) In equation (12), for The active power traded between the power router and the power grid during the specified time period. and These are the minimum and maximum active power of the power grid, respectively; in equation (13), The reactive power of the power grid. This represents the apparent power of the power grid cable.

[0015] Preferably, in step S2, the multi-agent deep deterministic policy gradient algorithm based on state space scaling includes the following steps: S21. Hierarchical optimization scheduling adopts a Markov decision process, including: use Describe the Markov decision process for energy scheduling in a multi-energy router, where N The number of power routers; For the observation state space of each power router; Provides the operating space for each power router. For the power router to be in state To carry out joint operations Then it becomes the next state. The probability distribution; Let be the reward function, representing the reward value obtained by the power router after selecting a joint action from the current state to reach the next state; The S22 and MADDPG algorithms are improvements upon the DDPG algorithm, suitable for learning autonomous behavior decisions by multiple agents in continuous dynamic scenarios where cooperation and competition coexist. In the MADDPG algorithm, each agent follows the DDPG structure, but differs by employing a centralized training and distributed execution approach. Specifically, it estimates the policies of all agents, fully utilizing global information during centralized training and relying only on local information during distributed execution, thus better mitigating the environmental instability of multi-agent systems. The use of the MADDPG algorithm for learning autonomous behavior decisions includes: In the energy scheduling process of multiple power routers, each power router collaboratively controls energy. As components of the environment, they simultaneously interact with it, increasing the complexity of the environment. For each power router, the training process is entirely similar to DDPG. Each power router obtains its own state-action information through an actor network and inputs the state-action information of other power routers through a critic network to evaluate the value of the joint state-action, fit the joint state-action value function Q, and modify the actor network parameters. The joint action interacts with the environment, the environment returns a reward value, the environment state is updated, and a joint experience array is generated. ,in , , They are respectively The status, actions, and reward values ​​obtained during a given period. for The state during the +1 time period defines the Q function of the power router as follows: ,in and Therefore, the power router The state and actions during a period of time and The network equation for updating the Q function by minimizing the loss based on the DDPG algorithm is as follows: (The equation is shown below.) (19) In equation (19) Represents the loss function. i Represents a power router i ; Q Represents the Q function; θ These are the parameters of the policy network; The target Q value is calculated using the Bellman equation; The estimated value of the Q-function is obtained through the output of a neural network. M The sample size refers to the number of sample data in each batch. The policy gradient in the MADDPG algorithm is defined as follows: (20) In equation (20), This indicates calculating the gradient. Representation strategy, It is an electric router i In strategy The objective function under the given conditions; S23. In system optimization scheduling, factors such as whether the connected devices of the power router achieve power balance, the energy loss generated during power transmission between power routers and between power routers and the grid, and the power range of power trading between power routers and the grid will affect the setting of the penalty component in the reward function. The magnitude of the penalty component coefficient significantly affects the convergence and stability of model training. This application gradually expands the state space of the training sample by state space scaling, gradually learning and adapting to more complex environments, thereby improving the accuracy and robustness of decision-making. In the early stage of training, a smaller state space is randomly selected for training. After the training tends to stabilize, the state space is gradually expanded, making full use of the previously gained experience and gradually adjusting the strategy to adapt to the expanded state space.

[0016] Compared to the MADDPG algorithm, state space scaling offers advantages in terms of rapid convergence, balance between exploration and exploitation, and stability. This allows it to train high-quality policies more efficiently in certain application scenarios and better adapt to environmental changes and training requirements. Therefore, a multi-agent deep deterministic policy gradient algorithm based on state space scaling can achieve rapid convergence, stable training, high efficiency, and higher-quality decisions. In the early stages of training, randomly selecting a smaller state space allows the agent to quickly converge to a better policy. Gradually expanding the state space allows the agent to gradually improve its understanding of the environment and update based on existing experience and models, avoiding retraining and making the training process more efficient while reducing instability and oscillations.

[0017] Preferably, the Markov decision process includes the following steps: S211, Constructing the state space set During each time period, each power router uploads scheduling data to the energy management center. For the energy management center, the state space set includes the state of each power router after scheduling, the electricity price for the current time period, and the grid electricity price, as shown in the following formula: (14) in As shown in the following formula: (15) In equation (15), for Power router during the time period If the state is This indicates a power router. If there is surplus electricity, it can be sold to other power routers or the power grid. This indicates a power router. We need to purchase electricity; Indicates a power router exist Customized electricity trading prices within a specific time period; , They represent The electricity sales price and purchase price of the power grid during the time period; S212, Constructing the Action Space Set The action space is used to regulate the power routers within the system, including the trading objects of each power router, the trading power, and the trading power price, as shown in the following formula: (16) in As shown in the following formula: (17) In equation (17), for Power router during the time period Randomly selected trading partners; express Power router during the time period The amount of electrical energy traded; for Power router during the time period The price of traded electricity; S212, Setting the reward function Reinforcement learning aims to maximize long-term rewards through continuous trial-and-error feedback and evaluation, and its reward function is similar to that of a single-agent reinforcement learning model. This application sets the immediate reward in two parts: the first part is set according to the optimization objective; the second part is a penalty mechanism set according to the constraint equations (10), (11), (7), and (12), respectively expressed as follows: , , and The reward function is defined as follows: (18) In the formula , , , All are penalty coefficients. for Power router during the time period The reward value obtained.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: The proposed hierarchical optimization scheduling of a multi-energy router system outperforms that of a single energy router within the system, and also improves the utilization rate of distributed energy resources and the stability of the power grid during grid connection. In terms of algorithms, the proposed MADDPG-SSS algorithm improves the training quality of the network by changing the range of the selected sample state space, while reducing the correlation between selected samples, thus mitigating the instability of the training process. Therefore, this invention can effectively improve training efficiency and the utilization rate of distributed energy resources, while reducing the system operating cost. Attached Figure Description

[0019] Figure 1 This is a structural diagram of the multi-energy router system model of the present invention in the embodiment; Figure 2 This is a schematic diagram of the training structure of the MADDPG algorithm of the present invention in an embodiment; Figure 3 This is a diagram illustrating the state space scaling process of the present invention in an embodiment; Figure 4 This is a schematic diagram illustrating the scheduling results of a single optimized scheduling power router 1 within the system using the present invention in this embodiment; Figure 5 This is a schematic diagram illustrating the results of optimizing the scheduling of a single power router 2 within the system using the present invention in this embodiment; Figure 6 This is a schematic diagram illustrating the hierarchical optimization scheduling results of multiple power routers within the system using the present invention in this embodiment. Figure 7 This is a comparison chart of the average reward value curves of the MADDPG-SSS algorithm, the MADDPG algorithm, and the MADDPG-PER algorithm of the present invention in the embodiments; Figure 8 This is a comparison diagram of the power information of the power router 1 and power router 2 of the present invention in the embodiment; Figure 9 This is a comparison diagram of the power output of the power router 1 and power router 2 in the embodiment of the present invention.

[0020] Figure 10 The following is a detailed flowchart of the MADDPG-SSS algorithm of the present invention in an embodiment. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0022] Example This embodiment proposes an energy scheduling method for a multi-energy router system, comprising the following steps: S1, such as Figure 1 As shown, a multi-energy router system model is constructed. To improve the absorption rate of distributed energy in both space and time, interconnected power routers in adjacent areas are linked together using interconnection lines. Based on the hierarchical scheduling concept, a multi-energy router system model is constructed, which includes an access layer, an interconnection layer, and an information layer, wherein: The access layer includes photovoltaic power generation equipment, wind turbine power generation equipment, loads, energy storage equipment and power grid, which are directly connected to the power router. The power router, as the core of the local controller, monitors the operating status of photovoltaic, wind turbine, load and energy storage equipment in real time and rationally dispatches energy to ensure the safe operation of the equipment. The interconnection layer enables energy transfer between power routers through cables, thereby reducing the impact of uncertainties in distributed energy generation and randomness in user electricity consumption, and improving the utilization rate of distributed energy. The information layer realizes the function of information exchange through the energy management center. Each power router uploads energy information to the energy management center. The energy management center adjusts the operating parameters based on the operating status. The power router dynamically adjusts the energy by changing the energy flow route according to the instructions issued by the energy management center, so as to ensure the power demand of each part.

[0023] It should be noted that the access layer, interconnection layer, and information layer in this invention are understood to describe the structural layering of a multi-power router system, such as energy generation, transmission, and information transmission. The power router layer and distribution network layer can be understood as how power is distributed, transmitted, and managed in a multi-power router system. The power router layer is responsible for the flow and management of power between the connected devices of the power routers, while the distribution network layer involves the transmission of energy between power routers and between the power routers and the power grid.

[0024] S2. Based on the multi-power router system model, a hierarchical scheduling strategy with the goal of minimizing system operating costs is used for hierarchical optimization scheduling. First, energy optimization scheduling of individual power routers is performed at the power router layer, and then global optimization scheduling among power routers at the distribution network layer is performed. The two scheduling strategies are both independent and interrelated. The hierarchical optimization scheduling adopts a Markov decision process and uses the MADDPG algorithm to learn autonomous behavior decisions. Finally, the state space of the training sample is gradually expanded by state space scaling to form a multi-agent deep deterministic policy gradient algorithm based on state space scaling (referred to as the MADDPG-SSS algorithm).

[0025] As the underlying device in a multi-power router system model, the power router, in addition to executing commands issued by the energy management center, also needs to independently control the devices directly connected to it. To minimize the total system operating cost, the operating cost of each power router's connected devices must be minimized. When acquiring energy management policies, the objective function for optimizing energy scheduling of a single power router at the power router layer is as follows: (1) In equation (1), For power routers The equipment operating costs, For power routers The equipment operation and maintenance costs, For power routers Cost of loss of connected energy storage devices.

[0026] Preferably, the objective function for energy optimization scheduling of a single power router at the power router layer is constrained by power balance, i.e. Time-of-use power router Supply and demand maintain a dynamic balance, as shown in the following formula: (2) In equation (2), For power routers Purchase power from other power routers. For power routers The power output of the routers sold to other power-powered routers. For the electrical energy traded with the grid, for t Time-of-use power router The electrical power consumed by the connected load, , , They provide electrical power to photovoltaic power generation, wind turbine power generation, and energy storage systems, respectively. The output power of wind turbine power generation and photovoltaic power generation is affected by wind speed and sunlight intensity at different times, and their output power has a maximum and a minimum value, as shown in equations (3) and (4): (3) (4) In equation (3), and For power routers Connect the upper and lower limits of the wind turbine's power output; in equation (4), and For power routers Connects the upper and lower limits of photovoltaic power generation output power; Considering the wear and tear during equipment operation, Remaining power of time-limited energy storage system State of charge As shown in equations (5) and (6), to prevent overcharging and discharging, the state of charge is constrained as shown in equation (7): (5) (6) (7) In equation (5), It is the self-discharge rate of the energy storage system. The charging and discharging time of the energy storage system. It is the charging and discharging power of the energy storage system; in equation (6), The rated capacity of the energy storage system; in equation (7), and These are the maximum and minimum storage capacities of the energy storage system, respectively.

[0027] During each time period, each power router uploads the dispatched data to the energy management center. The energy management center then comprehensively considers the status of each power router and the total system operating cost. To achieve the lowest possible power dispatch, only power transmission losses between power routers and between power routers and the grid need to be considered. The objective function for global optimization dispatching among power routers in the distribution network layer is as follows: (8) In equation (8), and These are power routers With power router Transmission losses that occur when power is exchanged with the power grid and These are power routers The cost of purchasing electricity and the revenue generated from selling electricity.

[0028] When the energy management center performs energy dispatch, it must not only meet the actual operating conditions of each power router, but also consider the real-time operating constraints between power routers and between power routers and the power grid. During the power interaction and transmission process, power loss will occur due to the influence of resistance and wire length, as shown in equation (9): (9) In equation (9), For power loss during transmission, For power routers and Interactive energy power, For power routers and The resistance of the transmission line between them For power routers and The distance between the transmission lines, For transmission line parameters, it refers to the magnitude of the voltage driven by the current in the current state; Because the current carrying capacity of a transmission line is limited, power routers... and power router The power of the transmission line between them satisfies the following formula: (10) In equation (10), and These are the minimum and maximum transmission power on the power router and transmission line, respectively; To facilitate power exchange between power routers and reduce power exchange with the power grid, power routers... Electricity price constraints are shown in the following formula: (11) In equation (11), and They are respectively The price of electricity purchased and sold by the power grid during specific time periods. and They are respectively Time-of-use power router Prices for buying and selling electricity; If the supply and demand of the multi-energy router system are unbalanced, it can purchase electricity from the grid to meet the load demand. The power constraint for the energy transaction between the energy router and the grid is shown in the following formula: (12) (13) In equation (12), for The active power traded between the power router and the power grid during the specified time period. and These are the minimum and maximum active power of the power grid, respectively; in equation (13), The reactive power of the power grid. This represents the apparent power of the power grid cable.

[0029] The above-mentioned multi-agent deep deterministic policy gradient algorithm based on state space scaling includes the following steps: S21. Hierarchical optimization scheduling adopts a Markov decision process, including: use Describe the Markov decision process for energy scheduling in a multi-energy router, where N The number of power routers; For the observation state space of each power router; Provides the operating space for each power router. For the power router to be in state To carry out joint operations Then it becomes the next state. The probability distribution; Let be the reward function, representing the reward value obtained by the power router after selecting a joint action from the current state to reach the next state; Specifically, the Markov decision process includes the following steps: S211, Constructing the state space set During each time period, each power router uploads scheduling data to the energy management center. For the energy management center, the state space set includes the state of each power router after scheduling, the electricity price for the current time period, and the grid electricity price, as shown in the following formula: (14) in As shown in the following formula: (15) In equation (15), for Power router during the time period If the state is This indicates a power router. If there is surplus electricity, it can be sold to other power routers or the power grid. This indicates a power router. We need to purchase electricity; Indicates a power router exist Customized electricity trading prices within a specific time period; , They represent The electricity sales price and purchase price of the power grid during the time period; S212, Constructing the Action Space Set The action space is used to regulate the power routers within the system, including the trading objects of each power router, the trading power, and the trading power price, as shown in the following formula: (16) in As shown in the following formula: (17) In equation (17), for Power router during the time period Randomly selected trading partners; express Power router during the time period The amount of electrical energy traded; for Power router during the time period The price of traded electricity; S212, Setting the reward function Reinforcement learning aims to maximize long-term rewards through continuous trial-and-error feedback and evaluation, and its reward function is similar to that of a single-agent reinforcement learning model. This application sets the immediate reward in two parts: the first part is set according to the optimization objective; the second part is a penalty mechanism set according to the constraint equations (10), (11), (7), and (12), respectively expressed as follows: , , and The reward function is defined as follows: (18) In the formula , , , These are all penalty coefficients, initially set to 0.2, 0.3, 0.1, and 0.2. During the experiment, these coefficients can be increased or decreased by a certain percentage (e.g., ±0.1). If the curve fluctuates unstablely, the coefficients should be decreased; if the curve changes slowly, the coefficients should be increased. for Power router during the time period The reward value obtained.

[0030] The S22 and MADDPG algorithms are improvements upon the DDPG algorithm, and are suitable for learning autonomous behavioral decisions by multiple agents in continuous dynamic scenarios where cooperation and competition coexist. For example... Figure 2As shown, in the MADDPG algorithm, each agent follows the DDPG algorithm structure. The difference is that MADDPG adopts a centralized training and distributed execution approach. Specifically, it estimates the policies of all agents, fully utilizing global information during centralized training and relying only on local information during distributed execution, thus better mitigating the environmental instability of multi-agent systems. The learning of autonomous behavior decisions using the MADDPG algorithm includes: In the energy scheduling process of multiple power routers, each power router collaboratively controls energy. As components of the environment, they simultaneously interact with it, increasing the complexity of the environment. For each power router, the training process is entirely similar to DDPG. Each power router obtains its own state-action information through an actor network and inputs the state-action information of other power routers through a critic network to evaluate the value of the joint state-action, fit the joint state-action value function Q, and modify the actor network parameters. The joint action interacts with the environment, the environment returns a reward value, the environment state is updated, and a joint experience array is generated. ,in , , They are respectively The status, actions, and reward values ​​obtained during a given period. for The state during the +1 time period defines the Q function of the power router as follows: ,in and Therefore, the power router The state and actions during a period of time and The network equation for updating the Q function by minimizing the loss based on the DDPG algorithm is as follows: (The equation is shown below.) (19) In equation (19) Represents the loss function. i Represents a power router i ; Q Represents the Q function; θ These are the parameters of the policy network; The target Q value is calculated using the Bellman equation; The estimated value of the Q-function is obtained through the output of a neural network. M The sample size refers to the number of sample data in each batch. The policy gradient in the MADDPG algorithm is defined as follows: (20) In equation (20), This indicates calculating the gradient. Representation strategy, It is an electric router i In strategy The objective function under the given conditions; S23. In system optimization scheduling, factors such as whether the connected devices of the power router achieve power balance, the energy loss generated during power transmission between power routers and between power routers and the grid, and the power range of power trading between power routers and the grid will affect the setting of the penalty component in the reward function. The magnitude of the penalty component coefficient significantly affects the convergence and stability of model training. This application gradually expands the state space of the training sample by state space scaling, gradually learning and adapting to more complex environments, thereby improving the accuracy and robustness of decision-making. Figure 3 As shown, in the early stage of training, a smaller state space is randomly selected for training. After the training tends to stabilize, the state space is gradually expanded. The previously gained experience is fully utilized, and the strategy is gradually adjusted to adapt to the expanded state space.

[0031] Figure 10 The specific process of the MADDPG-SSS algorithm is shown in the figure.

[0032] Compared to the MADDPG algorithm, state space scaling offers advantages in terms of rapid convergence, balance between exploration and exploitation, and stability. This allows it to train high-quality policies more efficiently in certain application scenarios and better adapt to environmental changes and training requirements. Therefore, a multi-agent deep deterministic policy gradient algorithm based on state space scaling can achieve rapid convergence, stable training, high efficiency, and higher-quality decisions. In the early stages of training, randomly selecting a smaller state space allows the agent to quickly converge to a better policy. Gradually expanding the state space allows the agent to gradually improve its understanding of the environment and update based on existing experience and models, avoiding retraining and making the training process more efficient while reducing instability and oscillations.

[0033] To better understand the above embodiments, a specific example will be analyzed below: See Figure 1 As shown, the system includes power router 1 (power_router1) and power router 2 (power_router2), each of which is connected to photovoltaic, wind turbines, energy storage systems, loads, and the power grid. The open-source dataset of California Electricity was selected for model training. After training, the model was tested using a real-world dataset from a specific region. Table 1 shows the capacity configuration information and operation and maintenance costs of each device, while Table 2 shows the maximum and minimum states of charge of the energy storage devices.

[0034] Table 1 Equipment parameters of the multi-energy router system

[0035] Table 2 Energy Storage System Equipment Parameters

[0036] During system operation, the energy management center schedules power dispatch based on the current status of each power router and dynamic electricity prices. The power router's sales price and purchase price are set to be lower than the grid's sales price for the current time period, and higher than the grid's purchase price. The grid's purchase and sales prices are shown in Table 3.

[0037] Table 3. Electricity Prices in Different Time Periods

[0038] The structure and parameters of a neural network affect the training of the model to a certain extent. In this paper, the capacity of the experience replay pool is set to 100,000, the minimum batch size is 128, the update batch size is 50, the learning rate is 0.001, and the discount factor is 0.99.

[0039] To analyze the impact of hierarchical scheduling strategy on system energy management, the MADDPG-SSS algorithm was used to perform individual optimal scheduling and hierarchical optimal scheduling of the power routers within the system, respectively. The scheduling results during the same period are as follows: Figure 4-6 As shown.

[0040] according to Figure 4 , Figure 5It can be seen that during the period of 15:00-21:00 for power router 1 and during the periods of 00:00-02:00 and 19:00-24:00 for power router 2, the photovoltaic power generation, wind turbine power generation, and energy stored in the energy storage system cannot meet the user's energy demand, and electricity needs to be purchased from the grid to achieve supply and demand balance. In the hierarchical optimization scheduling, as can be seen from trade_between_1-2, there is energy transfer between power router 1 and power router 2. During the period of 00:00-02:00, power router 1 transfers its excess power to power router 2 to meet the energy demand of power router 2; during the period of 15:00-19:00, power router 2 transfers its excess power to power router 1 to meet its energy demand; during the period of 02:00-03:00, after storing energy in the energy storage system, power router 1 transfers its excess power to power router 2, which can meet the energy demand, so that power router 2 can charge the energy storage system. The above process improves the utilization rate of distributed energy resources and reduces the instability of grid operation caused by distributed energy grid connection to a certain extent. Meanwhile, Table 4 shows that hierarchical optimized scheduling saves 2572.26 yuan in total system operating cost compared to single optimized scheduling using an internal power router, a decrease of 32.18%. Therefore, hierarchical optimized scheduling improves the utilization rate of distributed energy resources while reducing the total system operating cost.

[0041] Table 4 Comparison of Operating Costs

[0042] To verify that the MADDPG-SSS algorithm proposed in this embodiment has better performance. Figure 7 The average reward curves for the MADDPG algorithm, MADDPG-PER algorithm, and MADDPG-SSS algorithm are shown.

[0043] from Figure 7 The curves show that in the early stages of training, the action network is in the exploratory phase, and the algorithm's reward value is low and highly volatile. As the agent learns from historical experiences in the experience replay pool, the reward value gradually increases. At 50, 100, and 150 epochs of neural network parameter updates, the average reward value curves of the MADDPG and MADDPG-PER algorithms exhibit significant fluctuations. The MADDPG-PER algorithm's curve shows smaller fluctuations after 150 epochs, indicating that a better policy has been learned. However, the MADDPG algorithm's curve still fluctuates greatly after 150 epochs, only flattening out after 200 epochs. In contrast, the MADDPG-SSS algorithm's average reward value curve gradually decreases in volatility after 100 epochs, and even after 200 epochs, no relatively large fluctuations occur. Therefore, the MADDPG-SSS algorithm proposed in this embodiment performs better.

[0044] The hierarchical optimization scheduling results under the MADDPG-SSS algorithm are as follows: Figure 8 As shown in the diagram, both Power Router 1 and Power Router 2 meet the current user energy demand and have surplus energy during the periods of 05:00-06:00 and 15:00-16:00, selling this surplus energy to the grid to reduce energy waste. During the periods of 00:00-05:00 and 11:00-15:00, Power Router 1 has a surplus of overall energy output, while Power Router 2's demand exceeds its supply. Energy interaction is achieved through instructions issued by the Energy Management Center, reducing transactions with the grid. Conversely, during the periods of 08:00-11:00 and 16:00-18:00, Power Router 1 purchases energy from Power Router 2. In this process, the Energy Management Center, based on the energy information uploaded by Power Router 1 and Power Router 2, feeds back their respective electricity price information, encouraging them to coordinate energy dispatch with each other as much as possible, reducing transactions with the grid. The Energy Management Center continuously adjusts its operational strategies based on the electricity and price information obtained from both parties until it converges to a point where the system operating cost is minimized. At this point, the final price of electricity transactions within each time period is determined by the trading partners. If energy is transferred between power routers, the transaction price is the price defined by the power provider; if the transaction is with the power grid, the transaction price is the price provided by the power grid. During other time periods, if the demand of both power router 1 and power router 2 exceeds the supply, then electricity needs to be purchased from the power grid to meet the user's energy needs.

[0045] Figure 9 The results of energy interaction between Power Router 1 and Power Router 2 are displayed. It can be seen that from 00:00 to 14:00, the energy supplied by the power supply equipment connected to Power Router 1 meets the load demand. From 00:00 to 02:00, the energy supplied by the power supply equipment connected to Power Router 2 is exactly equal to the load demand; therefore, Power Router 1 sells electricity to the grid during this period. From 03:00 to 14:00, the power supply equipment connected to Power Router 2 cannot meet the load demand and needs to purchase electricity. Given the higher purchase price of electricity from the grid, Power Router 2 is more inclined to interact with Power Router 1 to obtain more electricity. If the load demand still exceeds the available electricity after interaction, electricity must be purchased from the grid, such as from 19:00 to 20:00. In general, if power router 1 and power router 2 have excess energy when they meet their own load requirements, they will sell the excess energy to the grid after charging. If one of them cannot meet its own load requirements, it will prioritize energy trading with the other power router, and then trade with the grid, so as to reduce the trading with the grid, reduce system operating costs, and meet load requirements at the same time.

[0046] Table 4 compares the total operating costs of the system under different algorithms. It can be seen that under the same algorithm optimization, the hierarchical optimization scheduling mode reduces the operating cost of power interaction between multiple power routers compared to power interaction between a single power router and the grid, resulting in a 32.18% decrease in overall operating cost. This means that interconnecting power routers located close to each other in adjacent areas improves the system's utilization rate of distributed energy and reduces system operating costs to some extent. Table 5 shows the average computation time of the algorithms; the MADDPG-SSS algorithm is faster than the MADDPG-PER algorithm.

[0047] Table 5 Average computation time of the algorithm

[0048] In summary, this invention enables hierarchical optimization scheduling of a multi-energy router system to outperform the optimization scheduling of a single energy router within the system. Furthermore, hierarchical optimization scheduling improves the utilization rate of distributed energy resources and the stability of the power grid during grid connection. In terms of algorithm, the proposed MADDPG-SSS algorithm improves the training quality of the network by changing the range of the selected sample state space, while reducing the correlation between selected samples, thus mitigating the instability of the training process. Therefore, this invention can effectively improve training efficiency and the utilization rate of distributed energy resources, while reducing the system operating cost.

[0049] The embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. An energy scheduling method for a multi-energy router system, characterized in that, Includes the following steps: S1. Construct a multi-energy router system model; S2. Based on the multi-power router system model, a hierarchical scheduling strategy with the goal of minimizing system operating costs is used for hierarchical optimization scheduling. First, energy optimization scheduling of individual power routers is performed at the power router layer, and then global optimization scheduling among power routers at the distribution network layer is performed. The hierarchical optimization scheduling adopts Markov decision process and MADDPG algorithm for learning autonomous behavior decision. Finally, the state space of the training sample is gradually expanded by state space scaling to form a multi-agent deep deterministic policy gradient algorithm based on state space scaling. The above-mentioned multi-agent deep deterministic policy gradient algorithm based on state space scaling includes the following steps: S21. Hierarchical optimization scheduling adopts a Markov decision process, including: use Describe the Markov decision process for energy scheduling in a multi-energy router, where N The number of power routers; For the observation state space of each power router; Provides the operating space for each power router. For the power router to be in state To carry out joint operations Then it becomes the next state. The probability distribution; Let be the reward function, representing the reward value obtained by the power router after selecting a joint action from the current state to reach the next state; S22. Learning autonomous behavior decision-making using the MADDPG algorithm, including: In the energy scheduling process of multiple power routers, each power router obtains its own state-action information through an actor network and inputs the state-action information of other power routers through a critic network. This process evaluates the value of the joint state-action, fits the joint state-action value function Q, and modifies the actor network parameters. The joint actions interact with the environment, the environment returns a reward value, the environment state is updated, and a joint experience array is generated. ,in , , They are respectively The status, actions, and reward values ​​obtained during a given period. for The state during the +1 time period defines the Q function of the power router as follows: ,in and Therefore, the power router i exist The state and actions during a period of time and The network equation for updating the Q function by minimizing the loss based on the DDPG algorithm is as follows: (The equation is shown below.) （19） In equation (19) Represents the loss function. i Represents a power router ; Q Represents the Q function; θ These are the parameters of the policy network; The target Q value is calculated using the Bellman equation; The estimated value of the Q-function is obtained through the output of a neural network. M The sample size refers to the number of sample data in each batch. The policy gradient in the MADDPG algorithm is defined as follows: （20） In equation (20), This indicates calculating the gradient. Representation strategy, It is an electric router i In strategy The objective function under the given conditions; S23. Gradually expand the state space of the training sample by state space scaling, including: In the early stages of training, a smaller state space is randomly selected for training. After the training stabilizes, the state space is gradually expanded. The previously gained experience is fully utilized, and the strategy is gradually adjusted to adapt to the expanded state space.

2. The energy scheduling method for a multi-energy router system according to claim 1, characterized in that, In step S1, the multi-energy router system model includes an access layer, an interconnection layer, and an information layer, wherein: The access layer includes photovoltaic power generation equipment, wind turbine power generation equipment, loads, energy storage equipment and power grid, which are directly connected to the power router. The power router, as the core of the local controller, monitors the operating status of photovoltaic, wind turbine, load and energy storage equipment in real time, and rationally dispatches energy to ensure the safe operation of the equipment. The interconnection layer enables energy transfer between power routers through cables, thereby reducing the impact of uncertainties in distributed energy generation and randomness in user electricity consumption, and improving the utilization rate of distributed energy. The information layer realizes the function of information exchange through the energy management center. Each power router uploads energy information to the energy management center. The energy management center adjusts the operating parameters based on the operating status. The power router dynamically adjusts the energy by changing the energy flow route according to the instructions issued by the energy management center, so as to ensure the power demand of each part.

3. The energy scheduling method for a multi-energy router system according to claim 2, characterized in that, In step S2, to achieve the lowest total system operating cost, it is necessary to ensure the lowest operating cost of each power router connected device. The objective function for energy optimization scheduling of a single power router at the power router layer is as follows: （1） In equation (1), For power routers The equipment operating costs, For power routers The equipment operation and maintenance costs, For power routers Cost of loss of connected energy storage devices.

4. The energy scheduling method for a multi-energy router system according to claim 3, characterized in that, The objective function for energy optimization scheduling of a single power router at the power router layer is constrained by power balance, i.e. Time-of-use power router Supply and demand maintain a dynamic balance, as shown in the following formula: （2） In equation (2), For power routers Purchase power from other power routers. For power routers The power output of the routers sold to other power-powered routers. For the electrical energy traded with the grid, For power routers q The electrical power consumed by the connected load, , , They provide electrical power to photovoltaic power generation, wind turbine power generation, and energy storage systems, respectively. The output power of wind turbine power generation and photovoltaic power generation is affected by wind speed and sunlight intensity at different times, and their output power has a maximum and a minimum value, as shown in equations (3) and (4): （3） （4） In equation (3), and For power routers Connect the upper and lower limits of the wind turbine's power output; in equation (4), and For power routers Connects the upper and lower limits of photovoltaic power generation output power; Considering the wear and tear during equipment operation, Remaining power of time-limited energy storage system State of charge As shown in equations (5) and (6), to prevent overcharging and discharging, the state of charge is constrained as shown in equation (7): （5） （6） （7） In equation (5), It is the self-discharge rate of the energy storage system. The charging and discharging time of the energy storage system. It is the charging and discharging power of the energy storage system; in equation (6), The rated capacity of the energy storage system; in equation (7), and These are the maximum and minimum storage capacities of the energy storage system, respectively.

5. The energy scheduling method for a multi-energy router system according to claim 4, characterized in that, In step S2, during each time period, each power router uploads the dispatched data information to the energy management center. The energy management center comprehensively considers the status of each power router and the total system operating cost. To achieve the lowest possible power dispatch, only power transmission losses between power routers and between power routers and the grid need to be considered. The objective function for global optimization dispatching among power routers in the distribution network layer is as follows: （8） In equation (8), and These are power routers With power router Transmission losses that occur when power is exchanged with the power grid and These are power routers The cost of purchasing electricity and the revenue generated from selling electricity.

6. The energy scheduling method for a multi-energy router system according to claim 5, characterized in that, When the energy management center performs energy dispatch, it must not only meet the actual operating conditions of each power router, but also consider the real-time operating constraints between power routers and between power routers and the power grid. During the power interaction and transmission process, power loss will occur due to the influence of resistance and wire length, as shown in equation (9): （9） In equation (9), For power loss during transmission, For power routers and Interactive energy power, For power routers and The resistance of the transmission line between them For power routers and The distance between the transmission lines, For transmission line parameters, it refers to the magnitude of the voltage driven by the current in the current state; Power router and power router The power of the transmission line between them satisfies the following formula: （10） In equation (10), and These are the minimum and maximum transmission power on the power router and transmission line, respectively; To facilitate power exchange between power routers and reduce power exchange with the power grid, power routers... Electricity price constraints are shown in the following formula: （11） In equation (11), and They are respectively The price of electricity purchased and sold by the power grid during specific time periods. and They are respectively Time-of-use power router Prices for buying and selling electricity; If the supply and demand of the multi-energy router system are unbalanced, it purchases electricity from the grid to meet the load demand. The power constraint for the energy transaction between the energy router and the grid is as follows: （12） （13） In equation (12), for The active power traded between the power router and the power grid during the specified time period. and These are the minimum and maximum active power of the power grid, respectively; in equation (13), The reactive power of the power grid. This represents the apparent power of the power grid cable.

7. The energy scheduling method for a multi-energy router system according to claim 1, characterized in that, The Markov decision process includes the following steps: S211, Constructing the state space set During each time period, each power router uploads scheduling data to the energy management center. For the energy management center, the state space set includes the state of each power router after scheduling, the electricity price for the current time period, and the grid electricity price, as shown in the following formula: （14） in As shown in the following formula: （15） In equation (15), for Power router during the time period If the state is This indicates a power router. If there is surplus electricity, it can be sold to other power routers or the power grid. This indicates a power router. We need to purchase electricity; Indicates a power router exist Customized electricity trading prices within a specific time period; , They represent The electricity sales price and purchase price of the power grid during the time period; S212, Constructing the Action Space Set The action space is used to regulate the power routers within the system, including the trading objects of each power router, the trading power, and the trading power price, as shown in the following formula: （16） in As shown in the following formula: （17） In equation (17), for Power router during the time period Randomly selected trading partners; express Power router during the time period The amount of electrical energy traded; for Power router during the time period The price of traded electricity; S213, Setting the reward function The immediate reward is set in two parts: the first part is set according to the optimization goal; the second part is the penalty mechanism set according to the constraints (10), (11), (7), and (12), respectively, and is expressed as follows: , , and The reward function is defined as follows: （18） In the formula , , , All are penalty coefficients. for Power router during the time period The reward value obtained.

Images