A micro-grid energy scheduling method and system based on reinforcement learning

Abstract

A micro-grid energy scheduling method and system based on reinforcement learning, the method comprising: constructing an energy scheduling objective function and a micro-grid operation cost objective function; defining the state space of a power grid operator agent and a plurality of micro-grid agents respectively, performing action selection and updating the corresponding state space; based on the updated state space of the micro-grid agent, combining the micro-grid operation cost objective function value to determine an instant reward function, and updating the agent; based on the updated state space of the power grid operator agent, calculating the energy scheduling objective function value to determine an outer global reward function, and updating the agent; calculating the comprehensive benefit function of the updated agent, and when a predetermined convergence condition is met, outputting a trained strategy network to determine the optimal action of each agent. The present application constructs a hierarchical multi-agent reinforcement learning energy scheduling mechanism, which realizes the collaborative optimal control of multi-dimensional power grid operation cost while taking into account the stability of the power grid operation.

Description

Technical Field

[0001] This invention belongs to the field of microgrid scheduling technology, and relates to a microgrid energy scheduling method and system based on reinforcement learning. Background Technology

[0002] With the rapid development of new energy power generation, distributed energy storage, and smart grids, microgrids, as important units for new energy management and optimization, are gradually being widely used globally. Especially in scenarios involving interconnected multiple microgrids, achieving coordinated dispatch among them has become a key issue in improving the flexibility and stability of the power system. However, the operating environment of microgrids generally suffers from large load fluctuations, high uncertainty in renewable energy output, and complex multi-stakeholder game dynamics, posing significant challenges to traditional centralized dispatching methods.

[0003] Currently, energy management in multi-microgrid systems mainly relies on centralized optimization and heuristic algorithms for scheduling. While these methods can improve economic efficiency to some extent, they have significant limitations when facing real-time scheduling of large-scale multi-microgrid systems. First, centralized optimization methods require collecting operational information from all microgrids, resulting in huge computational and communication overhead, making it difficult to meet real-time requirements. Second, the state information between microgrids raises privacy and security issues, and over-reliance on communication can easily lead to data leakage and cyberattack risks. Finally, traditional optimization methods are not adaptable enough to dynamic changes in the system and struggle to handle complex scenarios such as renewable energy fluctuations and load surges, resulting in a lack of robustness in scheduling results.

[0004] In recent years, with the advancements in deep reinforcement learning, multi-agent systems, and distributed computing technologies, agent-based energy scheduling methods have gradually become a research hotspot. By simulating multiple microgrids as independent agents and combining them with reinforcement learning algorithms, these agents can learn optimal scheduling strategies through environmental interaction without relying on centralized control. This approach possesses advantages such as strong adaptability, fast real-time response, and outstanding distributed collaborative capabilities, effectively addressing the challenges of complex power grid operating environments. In particular, the multi-agent framework, characterized by centralized training and distributed execution, has become the mainstream approach for solving distributed energy scheduling problems in recent years.

[0005] Currently, existing microgrid scheduling methods based on reinforcement learning include: Patent CN109347149A provides a microgrid energy storage scheduling method and device based on deep Q-value network reinforcement learning. This method establishes an energy storage model for a single microgrid, trains agents using DQN, and enables them to generate battery charging and discharging strategies under dynamic electricity pricing, achieving economical operation of energy storage to a certain extent. However, this scheme is mainly limited to energy storage optimization within a single microgrid, and does not adequately consider energy interaction and collaborative scheduling across microgrids. Chinese patent CN115459359A provides a 5G base station photovoltaic-storage microgrid system scheduling method based on multi-agent deep reinforcement learning, designing a multi-agent collaborative framework that enables 5G base stations to reduce overall operating costs through energy sharing and time-of-use pricing. Although this method introduces the multi-agent concept into microgrid scheduling, its application scenarios are mainly limited to communication base station power supply issues, focusing on reducing base station energy consumption costs without fully considering the overall stability and security of the power system. Patent CN118611067A discloses a master-slave game-theoretic optimization scheduling method for distribution networks and microgrid groups based on a multi-agent reinforcement learning algorithm. It constructs a master-slave game model between the distribution network operator and the microgrid group, and uses reinforcement learning for optimization, achieving coordinated scheduling of distributed energy resources and improving the renewable energy absorption rate and overall energy efficiency. However, this scheme assumes that the distribution network and microgrid group can fully share state information and forecast data. In practical applications, microgrids often cannot fully share information due to privacy and security considerations, causing the game model to fail to achieve the expected results under incomplete information conditions. Summary of the Invention

[0006] To address the technical problems in existing technologies, such as the lack of distributed coordination capabilities across microgrids, over-reliance on communication and global data sharing, lack of real-time execution mechanisms at the edge, singular optimization objectives, and lack of stability constraints, this invention provides a microgrid energy scheduling method and system based on reinforcement learning. The method includes: constructing an energy scheduling objective function and a microgrid operating cost objective function; defining the state spaces of the distribution network operator's agent and multiple microgrid agents, selecting actions, and updating the corresponding state spaces; determining an immediate reward function based on the updated state space of the microgrid agents, combined with the microgrid operating cost objective function value, and updating the agents; calculating the energy scheduling objective function value based on the updated state space of the distribution network operator's agent, determining the outer-layer global reward function, and updating the agents; calculating the comprehensive reward function of the updated agents; and when a predetermined convergence condition is met, outputting the trained policy network to determine the optimal action for each agent. This invention, by constructing a hierarchical multi-agent reinforcement learning energy scheduling mechanism, achieves coordinated optimal control of multi-dimensional grid operating costs while considering grid operation stability.

[0007] The present invention adopts the following technical solution: A first aspect of the present invention provides a microgrid energy dispatching method based on reinforcement learning, comprising: S1. Construct an energy dispatch objective function that considers grid revenue and grid operation stability; S2. Construct a microgrid operating cost objective function that includes unit power generation cost, energy storage operation cost, demand response cost, and external power purchase and sale cost; S3. Define the state spaces of the distribution network operator's intelligent agent and multiple microgrid intelligent agents respectively; for any intelligent agent, input the corresponding initial state space into the corresponding policy network, select actions, and update the corresponding state space. S4. Based on the updated state space of each microgrid agent, calculate the microgrid operating cost objective function value at the current moment. Combined with the current state of charge of the corresponding microgrid energy storage system, determine the immediate reward function at the current moment and update the corresponding agent. Based on the updated state space of the distribution network operator agent, calculate the energy scheduling objective function value, determine the outer global reward function at the current moment, and update the corresponding agent. Weight the updated outer global reward function and immediate reward function of the agent to calculate the comprehensive revenue function and determine whether the predetermined convergence condition is met. If it is met, output the trained policy network and determine the optimal action of each agent.

[0008] Preferably, in S1, the grid benchmark revenue is calculated based on the electricity sales and purchase prices at different times, and the electricity sales and purchase power of the microgrid; the power fluctuation benchmark value is calculated based on the electricity purchase and sales power of different microgrids at different times; for each time moment, the grid revenue and power fluctuation metric at the corresponding time moment are normalized by the grid benchmark revenue and fluctuation benchmark value; the normalized grid revenue and power fluctuation metric are weighted to obtain the energy dispatch objective function.

[0009] Preferably, the process of calculating the grid benchmark revenue and grid revenue is as follows: For each time moment, the electricity sales price of all microgrids at that time moment is multiplied by the electricity sales power and then the product of the electricity purchase price and the electricity purchase power is subtracted and summed up to obtain the grid revenue of all microgrids at that time moment; the grid revenue of all time moments is summed up and divided by the benchmark sampling period to obtain the grid benchmark revenue. Subtracting the line loss factor from 1 gives the line transmission efficiency of the corresponding microgrid; multiplying the line transmission efficiency of all microgrids by the purchased power and summing them up gives the total purchased power; subtracting the wholesale electricity price of the distribution network operator from the purchased electricity price of the microgrid and then multiplying it by the dispatch time interval and the total purchased power gives the grid revenue.

[0010] Preferably, the calculation process for the fluctuation benchmark value and power fluctuation metric is as follows: For each moment, the total power exchange is obtained by multiplying the purchased power of all microgrids by the line transmission efficiency minus the ratio of the sold power to the line transmission efficiency, and summing the results. The total power exchange is then calculated by subtracting the total power exchange of adjacent moments within the reference sampling period and taking the average value. For each moment, if the difference between the total power exchange at the corresponding moment and the expected power exchange value is not less than a predetermined power exchange difference threshold, the power fluctuation measure at the corresponding moment is the power exchange difference; otherwise, the power fluctuation measure at the corresponding moment is 0.

[0011] Preferably, in S2, for each microgrid connected to by the distribution network operator, the power generation cost of different generator sets at each time moment is modeled by a quadratic function based on the output of the generator sets in the corresponding microgrid at each time moment; For each moment, the change in the state of charge of the microgrid's energy storage battery between the next moment and the current moment is multiplied by the battery capacity and a predefined influence coefficient of the change in charge amplitude to obtain the energy storage operating cost at the corresponding moment. The demand response cost at the corresponding moment is calculated by multiplying the load adjustment power of all users participating in demand response in the microgrid by the unit power adjustment compensation coefficient of the corresponding user and summing them up. The electricity purchase price of the microgrid at each time moment is multiplied by the line transmission efficiency and the purchased power to obtain the electricity purchase cost of the microgrid at the corresponding time moment; the electricity sales price of the microgrid at each time moment is multiplied by the sales power and then divided by the line transmission efficiency to obtain the electricity sales revenue of the microgrid at the corresponding time moment; the electricity purchase cost is subtracted from the electricity sales revenue to obtain the external electricity purchase and sales cost of the microgrid at the corresponding time moment. The objective function for microgrid operating costs is obtained by adding the energy storage operation cost, demand response cost, and external power purchase and sale cost to the generation cost of all units in the corresponding microgrid, and then multiplying the result by the dispatch time interval.

[0012] Preferably, in S3, the state space of each microgrid includes the local load demand of the corresponding microgrid at the current moment, the output of renewable energy, the state of charge of the energy storage system, the purchase price of electricity, the sales price of electricity, and the actions of the corresponding microgrid at the previous moment. The state space of the distribution network operator's intelligent agent includes the total power exchange of the distribution network operator, the expected exchange power reference value, the purchase price and sales price of electricity in the microgrid, and the deviation between the total power exchange and the expected exchange power reference value.

[0013] Preferably, the process of selecting an action in S3 is as follows: The distribution network operator's intelligent agent inputs its current state space into the corresponding policy network and selects actions within the action space; each microgrid intelligent agent inputs its corresponding state space into the corresponding policy network and selects actions; the action space of the distribution network operator's intelligent agent includes the adjustment amounts of the microgrid's electricity sales price, electricity purchase price, and expected exchange power reference value; the action space of the microgrid intelligent agent includes the adjustment amounts of energy storage charging and discharging power, electricity purchase power, electricity sales power, and load adjustment ratio.

[0014] Preferably, the process of updating the state space of the microgrid agent in S3 is as follows: Based on the actions selected by each microgrid agent, update the energy storage charging and discharging power, power purchase power, power sales power, and load adjustment ratio; The numerator is the ratio of the charging efficiency of the updated energy storage system to the ratio of the charging power to the discharge efficiency, and the denominator is the energy storage capacity. The degree of change in the state of charge of the energy storage system at the previous moment is calculated. The degree of change in the state of charge is multiplied by the scheduling time interval and then added to the state of charge at the current moment to update the state of charge of the microgrid energy storage system at the current moment. Add 1 to the updated load adjustment ratio and multiply by the base power of the adjustable load to obtain the adjustable load of the corresponding microgrid; add the adjustable load power to the fixed load power to update the local load power of the corresponding microgrid at the current moment. Based on the action choices of the distribution network operator's intelligent agent, the purchase price and sales price of the microgrid are updated.

[0015] Preferably, the process of determining the instantaneous reward function at the current moment in S4 is as follows: Based on the state space at the current moment, calculate the corresponding microgrid operating cost objective function value; calculate the absolute value of the difference between the state of charge of the energy storage system and the energy storage health reference value at the current moment to obtain the state of charge change of the corresponding microgrid; for each microgrid agent, weight the state of charge change of the corresponding agent with the microgrid operating cost objective function value to calculate the corresponding instantaneous reward function at the current moment.

[0016] Preferably, the process of determining the outer global reward function at the current moment in S4 is as follows: Based on the state space of the distribution network operator's intelligent agent at the current moment, the energy scheduling objective function value is calculated; the absolute value of the difference between the updated total exchange power and the expected exchange power reference value is calculated to obtain the exchange power difference; the energy scheduling objective function value and the exchange power difference are weighted and the outer global reward function is calculated.

[0017] Preferably, the process of determining the optimal action for each agent in S4 is as follows: In each scheduling cycle, after all agents complete their actions, the state space before and after the update, the action at the current moment, and the reward function value are stored in the operator's experience pool and the microgrid experience pool, respectively. According to the priority experience replay mechanism, each experience sample is assigned a priority, and high-priority samples are sampled for training. After completing a round of inner and outer layer updates, the outer global reward function and the immediate reward are weighted to obtain the comprehensive benefit function. The iteration terminates when the difference in the comprehensive reward function value between adjacent time steps is less than a predetermined reward difference threshold or the difference in parameters of the policy network between adjacent iteration steps is less than a predetermined parameter difference threshold, thus obtaining a trained policy network and an evaluation network. The current state space is then input into the trained policy network, and the corresponding optimal action is output.

[0018] A second aspect of the present invention provides a microgrid energy dispatch system based on reinforcement learning, employing a microgrid energy dispatch method based on reinforcement learning, comprising: The scheduling objective construction module constructs an energy scheduling objective function that considers grid revenue and grid operation stability. The operating cost construction module constructs a microgrid operating cost objective function that includes unit power generation cost, energy storage operating cost, demand response cost, and external power purchase and sale cost; The multi-agent state space update module defines the state spaces of the distribution network operator agent and multiple microgrid agents respectively. For any agent, the corresponding initial state space is input into the corresponding policy network to select actions and update the corresponding state space. The multi-agent optimization module calculates the microgrid operating cost objective function value at the current moment based on the updated state space of each microgrid agent. Combining this with the current state of charge of the corresponding microgrid energy storage system, it determines the immediate reward function and updates the corresponding agent. Based on the updated state space of the distribution network operator agent, it calculates the energy scheduling objective function value, determines the current outer global reward function, and updates the corresponding agent. The updated outer global reward function and immediate reward function of each agent are weighted to calculate the comprehensive revenue function, and it is determined whether the predetermined convergence condition is met. If met, the trained policy network is output to determine the optimal action for each agent.

[0019] A third aspect of the present invention provides a terminal, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute steps of a reinforcement learning-based microgrid energy scheduling method.

[0020] A fourth aspect of the invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of a reinforcement learning-based microgrid energy dispatching method.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention, through a priority experience replay mechanism, makes the training process more focused on samples with high temporal differences and policy improvement value. The agent can more quickly master strategies to cope with extreme operating conditions, thus enabling it to rapidly adjust its operating mode in practical applications when the microgrid encounters sudden load fluctuations or large-scale grid connection of new energy sources, ensuring power supply and demand balance and system stability.

[0022] 2. This invention constructs an energy dispatch objective function that simultaneously considers grid revenue and power fluctuations. It introduces a grid benchmark revenue and a power fluctuation benchmark value, and applies normalized weighting. This enables the distribution network operator's intelligent agent to not only focus on maximizing economic benefits during reinforcement learning but also to proactively suppress large fluctuations in microgrid-connected power. This significantly reduces the impact of microgrid access on the operational stability of the main distribution network, improves system security and robustness, and avoids the problem of drastic power fluctuations caused by prioritizing cost or revenue in existing technologies.

[0023] 3. This invention comprehensively considers the generation costs of generating units, the operating costs of energy storage, the demand response costs, and the costs of purchasing and selling electricity at the microgrid level. It also incorporates changes in the state of charge (SOC) of energy storage and demand response behavior into the cost and reward function design. This allows each microgrid agent to autonomously learn the optimal charging and discharging strategy, electricity purchase and sale strategy, and load regulation strategy, while ensuring the health of energy storage and user comfort. Compared to traditional rule-based or single-objective optimization methods, this invention effectively taps into the regulation potential of multi-source heterogeneous resources, reduces the overall operating cost of the microgrid, and improves the renewable energy absorption capacity.

[0024] 4. This invention sets up an outer global reward function for the distribution network operator's intelligent agent and an immediate reward function for the microgrid's intelligent agent, and weights these two functions to form a comprehensive revenue function. This constructs a hierarchical and goal-oriented multi-agent reinforcement learning framework, enabling the agents to continuously update and optimize their strategies in a dynamic environment. This allows them to maintain high dispatch performance even under conditions of fluctuating renewable energy output and frequent load changes. This adaptive capability better utilizes clean energy sources such as wind and solar power, improving absorption rates and reducing wind and solar curtailment. Simultaneously, through adaptive optimization, this invention can rapidly adjust the allocation of power sources and energy storage when facing sudden load surges, ensuring a balance between power supply and demand and preventing frequent system over-limit operations. Attached Figure Description

[0025] Figure 1The flowchart illustrates a microgrid energy scheduling method based on reinforcement learning, as provided in this invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The embodiments described in this application are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this invention.

[0027] Example 1 Embodiment 1 of the present invention provides a microgrid energy dispatching method based on reinforcement learning, see below. Figure 1 This includes the following steps: S1. Construct an energy dispatch objective function that considers grid revenue and grid operation stability.

[0028] In the context of multi-microgrid coordinated operation, the dispatch objectives of the Distribution System Operator (DSO) must consider not only grid revenue but also grid operational stability. Pursuing only revenue maximization may lead to excessive bargaining in electricity purchase and sale among microgrids, resulting in significant power fluctuations at the Point of Common Coupling (PCC). Conversely, prioritizing stability may sacrifice profitability. Therefore, these two objectives are unified into a single comprehensive function, serving as the optimization metric for the DSO agent during training.

[0029] Based on the electricity sales and purchase prices at different times, and the electricity sales and purchase power of the microgrid, the grid benchmark revenue is calculated; based on the electricity purchase and sales power of different microgrids at different times, the power fluctuation benchmark value is calculated; for each time moment, the grid revenue and power fluctuation metric at the corresponding time moment are normalized using the grid benchmark revenue and fluctuation benchmark value; the normalized grid revenue and power fluctuation metric are weighted to obtain the energy dispatch objective function, the specific formula of which is as follows: ; in, This represents the value of the energy scheduling objective function at time t; This represents the power grid revenue at time t; It is the power grid benchmark revenue, used to eliminate numerical scale differences under different operating environments, and reflects the typical revenue level of the power grid under a given operating environment; This represents a measure of power fluctuation. It is a power fluctuation benchmark value, used for normalization processing, measuring the fluctuation level of the system in an unoptimized state, and serving as a reference standard for stability optimization items; It is a weighting factor, with a value between 0 and 1, which adjusts the ratio between power grid revenue and operational stability.

[0030] In this embodiment, the energy dispatch objective function embodies the balance between profit and volatility through a two-part subtraction: the first part rewards grid revenue growth, and the second part suppresses power volatility. The second part is measured using a normalized power volatility metric, representing the ratio of current volatility to baseline volatility. If it is less than 1, it indicates that the dispatch strategy is more stable than the baseline operation and deserves a higher reward; conversely, it is penalized, thus forming a dynamic adaptive suppression mechanism. (Weight) α The value can be dynamically adjusted according to changes in the scenario. For example, the stability weight can be increased during peak grid load periods, and the economic weight can be increased when electricity market prices fluctuate drastically.

[0031] As a preferred implementation method, the grid benchmark revenue is calculated based on the electricity sales and purchase prices at different times, and the electricity sales and purchase power of the microgrid; the specific process is as follows: For each time point, the electricity sales price of all microgrids at that time is multiplied by the electricity sales power, minus the product of the electricity purchase price and the electricity purchase power, and then summed to obtain the grid revenue of all microgrids at that time point. The grid revenue of all time points is then summed and divided by the benchmark sampling period to obtain the grid benchmark revenue. The calculation formula is as follows: ; in, The baseline sampling period is typically a typical scheduling period of continuous operation (such as 24 hours or 168 hours). Number of microgrids; and These represent the electricity sales price and the electricity purchase price of the microgrid at time t, respectively. and These represent the electricity sold and purchased by microgrid m at time t, respectively.

[0032] The absolute value of returns varies significantly depending on the season, electricity price, or renewable energy output. Without normalization, reinforcement learning models may saturate in high-return scenarios, leading to vanishing gradients or overfitting. Therefore, introducing a benchmark return value for the power grid effectively eliminates the differences in operating environments, allowing the agent to focus on the relative magnitude of return increases rather than their absolute values, thus enhancing the model's generalization and transferability.

[0033] As a preferred implementation method, a power fluctuation benchmark value is calculated based on the power purchased and sold by different microgrids at different times; the specific process is as follows: Subtracting the line loss factor from 1 gives the line transmission efficiency of the corresponding microgrid. For each moment, multiplying the purchased power of all microgrids by the line transmission efficiency minus the ratio of sold power to line transmission efficiency, and summing these values, gives the total power exchange at that moment. Subtracting the total power exchange at adjacent moments within the reference sampling period and taking the average yields the power fluctuation reference value. The specific calculation formula is as follows: ; in, The total power exchange at time t reflects the real-time power interaction level between the distribution network and all microgrids; This represents the line loss factor of the m-th microgrid, used to characterize the proportion of power loss during energy transmission between the m-th microgrid and the distribution network. A smaller value indicates higher line transmission efficiency; a larger value indicates a higher proportion of line loss. This factor can be calculated based on the power flow of the lines; the specific formula is: ; In the formula, and These represent the active power and reactive power between the m-th microgrid and the distribution network, respectively. The equivalent resistance of the line; This refers to the line voltage.

[0034] In this embodiment, the power fluctuation benchmark value is calculated by the mean absolute difference of power changes, characterizing the natural fluctuation level under unscheduled conditions. If frequent load fluctuations or randomness in renewable energy output exist, the power fluctuation benchmark value will be higher. Based on this, the reinforcement learning model will focus more on fluctuation suppression, thereby automatically adjusting the scheduling strategy and increasing energy storage participation and peak shaving / valley filling capabilities. The total power exchange reflects the asymmetry between electricity purchase and sale: losses increase costs during electricity purchase, while losses reduce revenue during electricity sale. This is achieved by aggregating the transaction behavior of all microgrids. It becomes the core input for power fluctuation measurement.

[0035] In a preferred implementation, the grid revenue in the energy dispatch objective function originates from the price difference between the DSO selling electricity to the microgrid and purchasing electricity from the upstream grid, while also considering line transmission losses. The line transmission efficiency of the corresponding microgrid is obtained by subtracting the line loss factor from 1. The total purchased power is obtained by multiplying the line transmission efficiencies of all microgrids by the purchased power. The grid revenue is obtained by subtracting the wholesale electricity price of the distribution network operator from the purchased electricity price of the microgrid and then multiplying this by the dispatch time interval and the total purchased power. The formula for grid revenue is: ; in, This indicates the retail electricity price of the DSO at that moment, i.e., the unit price at which electricity is sold to the microgrid; The wholesale electricity price at time t represents the price at which the DSO purchases electricity from the upper-level grid. It is an economic signal at the upper level of the system and does not directly participate in the subsequent microgrid dispatching decisions. This indicates the scheduling time interval, used to convert power gains into periodic gains.

[0036] This formula is intended to simulate the market-oriented operation logic of the Distribution Grid Operator (DSO) and reflect the DSO's scheduling incentive orientation (the higher the DSO's revenue, the stronger its incentive to the microgrid); it represents the economic revenue of the DSO at time t, reflecting the profit margin between selling electricity to the microgrid and purchasing electricity from the upper-level grid; while the formula mentioned above is a normalized statistic of the average revenue of all microgrids during the scheduling cycle, used only for the scaling and stability control of the reward function. The two formulas correspond to different economic entities, the former belonging to the global revenue of the outer DSO layer, and the latter belonging to the relative benchmark index of the microgrid layer.

[0037] In the calculation of grid revenue, the price difference between retail and wholesale electricity prices directly reflects the revenue generated from electricity trading. This takes into account the effective electrical energy after line losses. In other words, the larger the purchased power, the higher the profit, but if the line loss factor is large, the actual profit will be reduced. This enables the agent to learn to comprehensively consider the electricity price level and line conditions during scheduling.

[0038] As a preferred implementation, the power fluctuation metric for each time moment is calculated based on the total power exchange and the expected power exchange value; the specific process is as follows: To ensure grid stability, it is necessary to suppress power fluctuations at PCC points caused by the collective behavior of microgrids. For each time point, if the difference between the total power exchange and the expected power exchange value is not less than a predetermined power exchange difference threshold, the power fluctuation measure for that time point is the power exchange difference; otherwise, the power fluctuation measure is 0. The specific formula is as follows: ; in, The power fluctuation measure at time t indicates that when the total power exchange deviates from the expected value by more than a threshold, the fluctuation penalty is the deviation value itself, otherwise it is zero; this ensures that minor fluctuations are not penalized, while severe fluctuations are significantly penalized, thereby maintaining grid stability; This represents the total power exchange at time t; This represents the expected exchange power value, i.e., the ideal level set in the planning, and is used to measure the operational volatility of the microgrid itself. A predetermined threshold for interaction power difference is set; fluctuations less than this value are considered acceptable. This is an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

[0039] S2. Construct a microgrid operating cost objective function that includes generator generation costs, energy storage operation costs, demand response costs, and external power purchase and sale costs.

[0040] After clarifying the objective function of the distribution network operator, it is necessary to further establish an operating cost model from the perspective of individual microgrids, so that the multi-agent system can optimize based on local objectives during distributed execution. For each microgrid connected to the DSO, the generation cost of all units in the corresponding microgrid is successively added to the energy storage operating cost, demand response cost, and external power purchase and sale cost, and then multiplied by the dispatch time interval to obtain the microgrid operating cost objective function; the formula is as follows: ; in, This represents the total operating cost of the m-th microgrid at time t; This represents the set of controllable generator sets within the m-th microgrid; This represents the operating cost, or generation cost, of the j-th generator unit in the m-th microgrid; This represents the energy storage operating cost of the energy storage system in the m-th microgrid; This represents the compensation cost required for the demand response of the m-th microgrid, i.e., the demand response cost. This represents the cost of purchasing and selling electricity between the m-th microgrid and the distribution network, i.e., the external cost of purchasing and selling electricity. This indicates a scheduling time interval, such as 15 minutes or 1 hour, used to convert the overhead per unit power into periodic costs.

[0041] As a preferred implementation, for any microgrid, based on the output of the generator sets in the microgrid at each time moment, the power generation cost of different generator sets at that time moment is modeled using a quadratic function; the specific process is as follows: Generator sets are one of the main energy sources for microgrids, and their operating costs often exhibit a non-linear relationship with power output. To accurately reflect the operating costs of generator sets at different power levels, this embodiment uses a quadratic function for modeling; the specific formula is as follows: ; in, This represents the output of generator j in the m-th microgrid at time t; This is a quadratic coefficient, typically determined by the unit's fuel consumption characteristics. It reflects the accelerated cost growth effect when power increases, and its typical value range is within the range... The unit is yuan / (kW) 2•h); When the unit efficiency is high (such as a gas turbine), take the smaller value; when the unit efficiency is low (such as a small diesel engine), take the larger value. It is the coefficient of the linear term, representing the linear cost component, such as fuel base costs, and its typical value range is within the range of... The unit is yuan / (kW·h); coal-fired units, gas turbines and other high-efficiency units have smaller primary coefficients, while distributed units such as diesel generators have larger primary coefficients. This is a constant term, representing fixed costs unrelated to output, such as routine maintenance and repair expenses. Its typical value range is within the range of... The unit is yuan / h; for large centralized generating units, the constant term is higher; for distributed small generating units, the constant term is lower; when When the output increases, the cost function rises in the form of a quadratic curve, indicating that the unit efficiency decreases and the marginal cost increases under high-output operation.

[0042] As a preferred implementation method, the energy storage operating cost at each moment is calculated based on the battery capacity and the change in state of charge of the energy storage battery; the specific process is as follows: Energy storage devices play a crucial role in regulating power balance in microgrids, but excessive use can shorten battery life. This embodiment uses changes in battery state of charge (SOC) to measure losses.

[0043] For each moment, the change in the state of charge (SOC) of the microgrid's energy storage batteries between the next moment and the current moment is multiplied sequentially by the battery capacity and a predefined SOC change influence coefficient to obtain the corresponding energy storage operating cost; the specific formula is as follows: ; in, and Let represent the state of charge of the energy storage battery in the energy storage system of the m-th microgrid at times t and t+1, respectively. This represents the magnitude of SOC change between adjacent time points, which is equivalent to the depth of charge / discharge. This represents the energy storage capacity of the m-th microgrid, which determines the actual energy magnitude corresponding to the change in SOC; This is the influence coefficient of the charge change amplitude, used to quantify the impact of SOC changes on economics and reflect the life loss caused by battery cycling.

[0044] In this embodiment, the greater the charge and discharge amplitude of the energy storage battery in adjacent moments, the more severe the life loss and the higher the corresponding cost. When optimizing the strategy, the agent will automatically learn to make a trade-off between electricity price fluctuations and battery life. For example, when the electricity price difference is small, it will reduce charging and discharging to extend the life; when the electricity price difference is significant, it will make reasonable use of energy storage to obtain additional benefits.

[0045] As a preferred implementation, the demand response cost at different times is modeled based on the load adjustment power required from users at each moment and the set unit power adjustment compensation coefficient; the specific process is as follows: Demand response, as a flexibility resource on the load side, can alleviate grid pressure by reducing or shifting electricity consumption. However, this behavior usually requires economic compensation to users, thus necessitating the modeling of these compensation costs.

[0046] For each time point, the load adjustment power of all users participating in demand response in the microgrid at that time is multiplied by the corresponding user's unit power adjustment compensation coefficient, and then summed to obtain the demand response cost for that time point; the specific formula is as follows: ; in, This represents the set of users or loads participating in demand response for the m-th microgrid. This represents the load adjustment power of the z-th user at time t. A positive value indicates that the user is reducing the load, while a negative value indicates that the user is increasing the load. It is the unit power adjustment compensation coefficient corresponding to the user, reflecting the incentive level required for the user to participate in the response; when the grid is under high load or the electricity price is high, activating demand response may be more economical; while when the price is low or energy storage is sufficient, the use of demand response should be reduced.

[0047] As a preferred implementation, by collecting the load adjustment amount and compensation amount of various users in past scheduling events, the average unit response compensation is calculated as the unit power adjustment compensation coefficient for the corresponding user: ; In the formula, This represents the load adjustment amount for the corresponding user at time t; This represents the compensation amount for the corresponding user at time t; this calculation process reflects the user's true economic response characteristics, ensuring the rationality and feasibility of the coefficients.

[0048] As a preferred implementation method, the real-time electricity price and peak-valley price difference in the electricity market can also be considered. Based on the principle that demand response incentives generally do not exceed the price difference, the following empirical formula is used to calculate the unit power adjustment compensation coefficient for the corresponding user: ; in, Let t be the peak and trough prices at time t; Let t be the real-time electricity price in the electricity market. The response sensitivity coefficient reflects the load adjustability of different user categories. In this embodiment, the response sensitivity coefficient for industrial users is used. Response sensitivity coefficient for business users Residential users' response sensitivity coefficient This calculation method ensures that the compensation level matches the market price system, avoiding excessive incentives or insufficient response.

[0049] As a preferred implementation method, the external electricity purchase and sale costs at different times are calculated based on the microgrid's purchased and sold power, line loss factor, and purchased and sold electricity prices; the specific formula is as follows: Energy exchange between microgrids and the distribution network is achieved through electricity purchase and sale. This purchase and sale is influenced not only by electricity prices but also by line losses. For any microgrid, 1 minus the line loss factor represents the corresponding microgrid's line transmission efficiency. The electricity purchase price at each moment is multiplied by the line transmission efficiency and the purchased power at that moment to obtain the microgrid's electricity purchase cost. The electricity sale price at each moment is multiplied by the sold power at that moment and then divided by the line transmission efficiency to obtain the microgrid's electricity sale revenue at that moment. Subtracting the electricity sale revenue from the electricity purchase cost yields the microgrid's external electricity purchase and sale cost at that moment. The specific formulas are as follows: ; In the formula, This represents the line transmission efficiency of the m-th microgrid.

[0050] The cost of purchasing electricity is amplified by losses, while the revenue from selling electricity is reduced due to losses, highlighting the asymmetry of bidirectional power transmission. This modeling necessitates that agents comprehensively consider both price and losses during scheduling, avoiding blind reliance on external power purchases or sales.

[0051] S3. Define the state spaces of the distribution network operator's intelligent agent and multiple microgrid intelligent agents respectively; for any intelligent agent, input the corresponding initial state space into the corresponding policy network, select actions, and update the corresponding state space.

[0052] In this embodiment, under a multi-microgrid energy dispatch scenario, each microgrid is treated as an independent intelligent agent, while the distribution network operator (DSO) is modeled as a global intelligent agent, thus constructing a multi-agent system architecture of "1 global intelligent agent + m local intelligent agents". The main reason for this design is that power systems possess both local optimization objectives and global dispatch objectives, which need to be unified through a reasonable division of intelligent agents. If only a single microgrid is modeled as an intelligent agent without considering the DSO, the system lacks global coordination, leading to conflicts between individual behaviors and overall operational goals. For example, a microgrid might purchase large amounts of electricity when prices are low to reduce its own purchase costs, but other microgrids might be performing similar operations simultaneously, causing a surge in distribution network load and triggering large-scale fluctuations. Therefore, this invention introduces the DSO intelligent agent to balance the contradictions between individual and global goals.

[0053] In this system, the microgrid agent (Agent-MG) is primarily responsible for local-level energy dispatch. Its observation information includes local load levels, renewable energy output, the state of charge (SOC) of energy storage systems, and local power purchase and sale price signals. Its actions include energy storage charging and discharging control, power purchase dispatch, power sale dispatch, and demand response level adjustment. The distribution network operator agent (Agent-DSO) is responsible for balancing power exchange from a global perspective. Its observation information includes estimates of power purchase and sale across all microgrids, the total power exchange in the system, the current electricity price curve, and the deviation from the desired power. Its actions include adjusting retail electricity prices, setting power purchase and sale strategies, and issuing demand-side response signals. Through this multi-agent structure, this implementation achieves a balance between distributed autonomy and global coordination, preserving the independence of microgrids while ensuring the overall stability of the power system.

[0054] For the m-th microgrid, the state space of the corresponding microgrid is defined as follows: ; in, Let m be the state space of the m-th microgrid at time t; This represents the local load demand of the m-th microgrid at time t, and is the direct driving force determining power purchase and discharge behavior; This parameter represents the output of renewable energy sources (such as photovoltaic and wind power) within a microgrid. It is typically random and uncertain, and the agent needs to adjust its energy storage or demand response behavior based on the volatility of this parameter. The state of charge of the energy storage system, with a range of values. This reflects the remaining regulation capacity of energy storage and is a key parameter to ensure the healthy operation of energy storage. , These are the purchase price and sales price of electricity for the microgrid, respectively, which are direct reflections of the electricity market signals. The smart device uses these signals to decide whether to purchase electricity from the grid or sell surplus electricity back to the grid. The action information from the previous moment is used to guide the continuity of decision-making and avoid action oscillations caused by a lack of historical information.

[0055] As a preferred implementation, at each time step, the energy flow of the m-th microgrid should satisfy the power balance constraint:

[0056] in, Let be the discharge power of the energy storage system of the m-th microgrid at time t; Let be the charging power of the energy storage system of the m-th microgrid at time t; In this embodiment, renewable energy output is typically provided by a prediction model (such as LSTM predicting solar irradiance or wind speed), but the reinforcement learning agent can adjust the ratio of energy storage to purchased power through action variables to achieve "prediction correction"; that is, if the predicted output is too low, the agent can increase the power storage ratio. Or reduce If the predicted output is too high, then increase... Or charging Therefore, the agent's optimal action strategy is automatically learned during training to utilize energy storage and the external electricity market to balance random fluctuations in renewable energy output.

[0057] This state-space design enables microgrid agents to make relatively reasonable scheduling decisions based solely on local observation information without relying on global communication, thereby reducing communication costs and system complexity.

[0058] For a distribution network operator's intelligent agent, its state space is defined as: ; in, Let t be the state space of the distribution network operator. The total power exchange of DSO is defined as the weighted sum of the power purchased and sold by all microgrids, and is used to reflect the global power balance. The expected switching power reference value set for operators is usually based on the grid's safe operation requirements or the predicted load curve, and is used to guide coordination among all microgrids; , As a market signal from operators, it will guide the electricity purchase and sale behavior of each microgrid, indirectly achieving overall coordination; This represents the deviation of the total power from the reference value, and is used to assess whether the system has excessive fluctuations. It is an important component of the operator's reward function.

[0059] The distribution network operator's intelligent agent inputs its current state space into the corresponding policy network, outputs the probability of each action in the action space, and selects actions based on these probabilities. The action space includes adjustments to the microgrid's electricity sales price, electricity purchase price, and the expected exchange power reference value, as detailed below: ; in, For the action space of the intelligent agent of the power distribution network operator; Let be the adjustment amount of the electricity purchase price of the microgrid at time t; The adjustment amount of the electricity sales price of the microgrid at time t; that is, adjusting the price signal according to the current power supply and demand status to guide the power purchase and sale behavior of the lower-level microgrid. The adjustment amount is the expected exchange power reference value at time t; This process provides a global scheduling constraint and price guidance mechanism, equivalent to a higher-level decision-maker in reinforcement learning. In this embodiment, the DSO agent does not directly control the behavior of the microgrid, but rather achieves a two-tiered coordination mechanism of economic incentives and physical constraints through dynamic adjustments of price signals and reference power. This price-oriented distributed scheduling mechanism effectively reduces communication overhead and enables autonomous operation and overall coordination of each microgrid.

[0060] When the operator broadcasts the new electricity purchase and sale prices, each microgrid agent inputs its corresponding state space into the corresponding policy network to select actions. The action space of the corresponding microgrid agent includes the adjustment amounts for energy storage charging and discharging power, electricity purchase power, electricity sale power, and load adjustment ratio, as detailed below: ; in, This represents the action space of the m-th microgrid. This is the adjustment amount for the charging power of the energy storage system, and it should be a non-negative value. This is the adjustment amount of the discharge power of the energy storage system, and it is taken as a non-negative value; This is an adjustment amount for the power purchased from the distribution network, used to address situations where renewable energy output is insufficient; The amount of electricity sold to the distribution network is adjusted to absorb surplus energy. The adjustment amount is the load adjustment ratio, used to update the adjustable load; The above five actions collectively determine the energy flow of the microgrid; the charging and discharging of energy storage affects the change of SOC, the power purchase and sale actions affect the total power exchange of the system, and the demand response actions participate in mitigating system fluctuations through flexible load regulation; to avoid physical conflicts, this embodiment introduces mutual exclusion constraints and upper and lower limit constraints when selecting actions: ; ; ; In the formula, the first condition is a mutual exclusion constraint, indicating that energy storage can only be in a charging or discharging state at any given time; the second and third conditions are the upper and lower limits of the energy storage charging power and discharging power, respectively. This is the upper limit of the charging power for energy storage. This represents the upper limit of the energy storage discharge power. As a preferred implementation, the state space at the current moment is determined based on the action selected by each microgrid agent: Based on the actions selected by each microgrid agent, the charging and discharging power of the energy storage system is updated; the charging efficiency of the energy storage is multiplied by the ratio of charging power to discharging efficiency (charging power minus discharging power to discharging efficiency) as the numerator, and the energy storage capacity is used as the denominator to calculate the degree of change in the state of charge (SOC) of the energy storage system at the previous moment; the degree of SOC change is multiplied by the scheduling time interval and then added to the current SOC to obtain the current SOC of the microgrid energy storage system; the specific calculation formula is as follows: ; in, and These represent the charging and discharging efficiencies of energy storage, respectively. For energy storage capacity; The time step is denoted by ; this formula reflects the energy conservation principle: charging energy storage increases the State of Charge (SOC), while discharging decreases the SOC; the efficiency term is used to account for energy loss; if the SOC exceeds the range, a penalty mechanism is triggered to prevent overcharging or over-discharging of the energy storage. Based on the actions selected by the microgrid agent, update the load adjustment ratio at the current moment; add 1 to the load adjustment ratio and multiply by the base power of the adjustable load to obtain the adjustable load of the corresponding microgrid; add the adjustable load power to the fixed load power to obtain the local load power of the corresponding microgrid at the current moment; expressed as: ; ; in, Fixed load power represents rigid demand from living and production equipment, etc. The load capacity is adjustable and regulated by the demand response mechanism. The reference power for adjustable loads is typically the normal power consumption of the user when not participating in demand response. This parameter represents the load adjustment ratio; a positive value indicates an increase in load (valley filling), while a negative value indicates a decrease in load (peak shaving). This parameter is typically used as part of the action output of a reinforcement learning agent to balance the peak-valley difference in energy scheduling. Based on the action choices of the distribution network operator's intelligent agent, the purchase price and sales price of the microgrid are updated.

[0061] S4. Based on the updated state space of each microgrid agent, calculate the microgrid operating cost objective function value at the current moment. Combined with the current state of charge of the corresponding microgrid energy storage system, determine the immediate reward function at the current moment and update the corresponding agent. Based on the updated state space of the distribution network operator agent, calculate the energy scheduling objective function value, determine the outer global reward function at the current moment, and update the corresponding agent. Weight the updated outer global reward function and immediate reward function of the agent to calculate the comprehensive revenue function and determine whether the predetermined convergence condition is met. If it is met, output the trained policy network and determine the optimal action of each agent.

[0062] Based on the current state space, calculate the corresponding microgrid operating cost objective function value; calculate the absolute value of the difference between the energy storage system's state of charge and the energy storage health reference value at the current moment to obtain the corresponding microgrid state of charge change; for each microgrid agent, weight the agent's state of charge change with the microgrid operating cost objective function value to calculate the corresponding instantaneous reward function at the current moment: ; in, This represents the instantaneous reward function at the current moment; For reference values ​​of energy storage health; and These are the weighting coefficients for the objective function value of microgrid operating cost and the change in state of charge, respectively, with values ​​all within the interval (0,1). This reward, which considers both economic efficiency and the health and operational stability of energy storage, serves as an important feedback signal for microgrids to learn local optimal strategies.

[0063] As a preferred implementation, the process of updating the state space based on the actions selected by the distribution network operator's intelligent agent is as follows: Based on the selected action, update the expected exchange power reference value, the microgrid's electricity purchase price and sales price, and the total power exchange volume of the distribution network operator; subtract the expected exchange power reference value from the updated total power exchange volume, and update the deviation between the total power and the reference value at the current moment.

[0064] Based on the state space of the distribution network operator's intelligent agent at the current moment, the energy scheduling objective function value is calculated; the absolute value of the difference between the updated total exchange power and the expected exchange power reference value is calculated to obtain the exchange power difference; the energy scheduling objective function value and the exchange power difference are weighted and the outer global reward function is calculated; the specific formula is as follows: ; in, For the outer global reward function; , The weights are normalized and their values ​​are all within the range (0,1). This reward reflects the operational balance and profit level of the entire system, providing a global learning signal for the outer agent.

[0065] In each scheduling cycle, after all agents complete their actions, the state spaces before and after the update, the current action, and the reward function value are stored in the operator's experience pool and the microgrid experience pool, respectively. Based on the priority experience replay mechanism, each experience sample is assigned a priority, and high-priority samples are sampled for training. After completing one round of inner and outer layer updates, the outer global reward function and the immediate reward are weighted to obtain the comprehensive reward function. ; In the formula, Let be the overall return function at the current moment; The iteration terminates when the difference in the comprehensive reward function value between adjacent time steps is less than a predetermined reward difference threshold or the difference in parameters of the policy network between adjacent iteration steps is less than a predetermined parameter difference threshold, thus obtaining a trained policy network and an evaluation network. The current state space is then input into the trained policy network, and the corresponding optimal action is output.

[0066] At this point, the policy parameters of all agents are stable, and the system enters its optimal operating state. The finally trained policy is embedded into the edge AI chip to achieve rapid inference and real-time scheduling based on local observations.

[0067] The AI ​​chip's high parallel computing capability ensures millisecond-level inference latency, enabling scheduling to respond in real time to changes in the electricity market and load. Simultaneously, combined with a priority experience replay mechanism, the chip can perform lightweight updates locally, thus maintaining policy adaptability under fluctuating scenarios.

[0068] This embodiment employs a priority experience replay mechanism, prioritizing the replay of high-value samples to improve the sample utilization, learning efficiency, and convergence speed of the reinforcement learning algorithm. Using the sampled experience data, temporal difference error is calculated, and a loss function is constructed based on this error to update the parameters of the evaluation network and the policy network. To ensure a more stable training process, a soft update method is used to slowly synchronize the parameters of the current network and the target network, preventing policy abrupt changes.

[0069] Example 2 Embodiment 2 of the present invention provides a microgrid energy dispatch system based on reinforcement learning, comprising: The scheduling objective construction module constructs an energy scheduling objective function that considers grid revenue and grid operation stability. The operating cost construction module constructs a microgrid operating cost objective function that includes unit power generation cost, energy storage operating cost, demand response cost, and external power purchase and sale cost; The multi-agent state space update module defines the state spaces of the distribution network operator agent and multiple microgrid agents respectively. For any agent, the corresponding initial state space is input into the corresponding policy network to select actions and update the corresponding state space. The multi-agent optimization module calculates the microgrid operating cost objective function value at the current moment based on the updated state space of each microgrid agent. Combining this with the current state of charge of the corresponding microgrid energy storage system, it determines the immediate reward function and updates the corresponding agent. Based on the updated state space of the distribution network operator agent, it calculates the energy scheduling objective function value, determines the current outer global reward function, and updates the corresponding agent. The updated outer global reward function and immediate reward function of each agent are weighted to calculate the comprehensive revenue function, and it is determined whether the predetermined convergence condition is met. If met, the trained policy network is output to determine the optimal action for each agent.

[0070] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0071] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0072] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0073] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A microgrid energy dispatching method based on reinforcement learning, characterized in that, include: S1. Construct an energy dispatch objective function that considers grid revenue and grid operation stability; S2. Construct a microgrid operating cost objective function that includes unit power generation cost, energy storage operation cost, demand response cost, and external power purchase and sale cost; S3. Define the state spaces of the distribution network operator's intelligent agent and multiple microgrid intelligent agents respectively; for any intelligent agent, input the corresponding initial state space into the corresponding policy network, select actions, and update the corresponding state space. S4. Based on the updated state space of each microgrid agent, calculate the target function value of the microgrid operating cost at the current moment, combine it with the state of charge of the corresponding microgrid energy storage system at the current moment, determine the instantaneous reward function at the current moment, and update the corresponding agent. Based on the updated state space of the distribution network operator's agent, the energy scheduling objective function value is calculated, the outer global reward function at the current moment is determined, and the corresponding agent is updated. The updated agent's outer global reward function and immediate reward function are weighted to calculate the comprehensive reward function, and it is determined whether the predetermined convergence condition is met. If the conditions are met, output the trained policy network and determine the optimal action for each agent.

2. The microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: In S1, the grid benchmark revenue is calculated based on the electricity sales and purchase prices at different times, and the electricity sales and purchase power of the microgrid; the power fluctuation benchmark value is calculated based on the electricity purchase and sales power of different microgrids at different times. For each time point, the grid revenue and power fluctuation metrics at the corresponding time point are normalized using the grid benchmark revenue and fluctuation benchmark values, respectively. By weighting the normalized grid revenue and power fluctuation metrics, the energy dispatch objective function is obtained.

3. The microgrid energy dispatching method based on reinforcement learning according to claim 2, characterized in that: The process of calculating the grid benchmark revenue and grid revenue is as follows: For each time moment, the electricity sales price of all microgrids at that time moment is multiplied by the electricity sales power and then the product of the electricity purchase price and the electricity purchase power is subtracted and summed up to obtain the grid revenue of all microgrids at that time moment; the grid revenue of all time moments is summed up and divided by the benchmark sampling period to obtain the grid benchmark revenue. Subtracting the line loss factor from 1 gives the line transmission efficiency of the corresponding microgrid; multiplying the line transmission efficiency of all microgrids by the purchased power and summing them up gives the total purchased power; subtracting the wholesale electricity price of the distribution network operator from the purchased electricity price of the microgrid and then multiplying it by the dispatch time interval and the total purchased power gives the grid revenue.

4. The microgrid energy dispatching method based on reinforcement learning according to claim 2, characterized in that: The calculation process for the fluctuation benchmark value and power fluctuation metric is as follows: For each moment, the total power exchange is obtained by multiplying the purchased power of all microgrids by the line transmission efficiency minus the ratio of the sold power to the line transmission efficiency, and summing the results. The total power exchange is then calculated by subtracting the total power exchange of adjacent moments within the reference sampling period and taking the average value. For each moment, if the difference between the total power exchange at the corresponding moment and the expected power exchange value is not less than a predetermined power exchange difference threshold, the power fluctuation measure at the corresponding moment is the power exchange difference; otherwise, the power fluctuation measure at the corresponding moment is 0.

5. The microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: In S2, for each microgrid connected to the distribution network operator, the power generation cost of different generator sets at each time moment is modeled by a quadratic function based on the output of the generator sets in the corresponding microgrid at each time moment. For each moment, the change in the state of charge of the microgrid's energy storage battery between the next moment and the current moment is multiplied by the battery capacity and a predefined influence coefficient of the change in charge amplitude to obtain the energy storage operating cost at the corresponding moment. The demand response cost at the corresponding moment is calculated by multiplying the load adjustment power of all users participating in demand response in the microgrid by the unit power adjustment compensation coefficient of the corresponding user and summing them up. The electricity purchase price of the microgrid at each time moment is multiplied by the line transmission efficiency and the purchased power to obtain the electricity purchase cost of the microgrid at the corresponding time moment; the electricity sales price of the microgrid at each time moment is multiplied by the sales power and then divided by the line transmission efficiency to obtain the electricity sales revenue of the microgrid at the corresponding time moment; the electricity purchase cost is subtracted from the electricity sales revenue to obtain the external electricity purchase and sales cost of the microgrid at the corresponding time moment. The objective function for microgrid operating costs is obtained by adding the energy storage operation cost, demand response cost, and external power purchase and sale cost to the generation cost of all units in the corresponding microgrid, and then multiplying the result by the dispatch time interval.

6. The microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: In S3, the state space of each microgrid includes the local load demand of the corresponding microgrid at the current moment, the output of renewable energy, the state of charge of the energy storage system, the purchase price of electricity, the sales price of electricity, and the actions of the corresponding microgrid at the previous moment. The state space of the distribution network operator's intelligent agent includes the total power exchange of the distribution network operator, the expected exchange power reference value, the purchase price and sales price of electricity in the microgrid, and the deviation between the total power exchange and the expected exchange power reference value.

7. The microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: The process of selecting actions in S3 is as follows: The distribution network operator's intelligent agent inputs its current state space into the corresponding policy network and selects actions within the action space; each microgrid intelligent agent inputs its corresponding state space into the corresponding policy network and selects actions; the action space of the distribution network operator's intelligent agent includes the adjustment amounts of the microgrid's electricity sales price, electricity purchase price, and expected exchange power reference value; the action space of the microgrid intelligent agent includes the adjustment amounts of energy storage charging and discharging power, electricity purchase power, electricity sales power, and load adjustment ratio.

8. The microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: The process of updating the state space of microgrid agents in S3: Based on the actions selected by each microgrid agent, update the energy storage charging and discharging power, power purchase power, power sales power, and load adjustment ratio; The numerator is the ratio of the charging efficiency of the updated energy storage system to the ratio of the charging power to the discharge efficiency, and the denominator is the energy storage capacity. The degree of change in the state of charge of the energy storage system at the previous moment is calculated. The degree of change in the state of charge is multiplied by the scheduling time interval and then added to the state of charge at the current moment to update the state of charge of the microgrid energy storage system at the current moment. Add 1 to the updated load adjustment ratio and multiply by the base power of the adjustable load to obtain the adjustable load of the corresponding microgrid; add the adjustable load power to the fixed load power to update the local load power of the corresponding microgrid at the current moment. Based on the action choices of the distribution network operator's intelligent agent, the purchase price and sales price of the microgrid are updated.

9. The microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: The process of determining the instantaneous reward function at the current moment in S4 is as follows: Based on the state space at the current moment, calculate the corresponding microgrid operating cost objective function value; calculate the absolute value of the difference between the state of charge of the energy storage system and the energy storage health reference value at the current moment to obtain the change in the state of charge of the corresponding microgrid. For each microgrid agent, the change in the state of charge of the corresponding agent is weighted with the objective function value of the microgrid operating cost, and the instantaneous reward function corresponding to the current moment is calculated.

10. A microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: The process of determining the outer global reward function at the current moment in S4 is as follows: Based on the state space of the distribution network operator's intelligent agent at the current moment, the energy scheduling objective function value is calculated; the absolute value of the difference between the updated total exchange power and the expected exchange power reference value is calculated to obtain the exchange power difference; the energy scheduling objective function value and the exchange power difference are weighted and the outer global reward function is calculated.

11. A microgrid energy dispatching method based on reinforcement learning according to claim 1, characterized in that: The process of determining the optimal action for each agent in S4 is as follows: In each scheduling cycle, after all agents complete their actions, the state space before and after the update, the action at the current moment, and the reward function value are stored in the operator's experience pool and the microgrid experience pool, respectively. According to the priority experience replay mechanism, each experience sample is assigned a priority, and high-priority samples are sampled for training. After completing a round of inner and outer layer updates, the outer global reward function and the immediate reward are weighted to obtain the comprehensive benefit function. The iteration terminates when the difference in the comprehensive reward function value between adjacent time steps is less than a predetermined reward difference threshold or the difference in parameters of the policy network between adjacent iteration steps is less than a predetermined parameter difference threshold, thus obtaining a trained policy network and an evaluation network. The current state space is then input into the trained policy network, and the corresponding optimal action is output.

12. A microgrid energy dispatch system based on reinforcement learning, using the method described in any one of claims 1-11, characterized in that, include: The scheduling objective construction module constructs an energy scheduling objective function that considers grid revenue and grid operation stability. The operating cost construction module constructs a microgrid operating cost objective function that includes unit power generation costs, energy storage operating costs, demand response costs, and external power purchase and sale costs. The multi-agent state space update module defines the state spaces of the distribution network operator agent and multiple microgrid agents respectively. For any agent, the corresponding initial state space is input into the corresponding policy network to select actions and update the corresponding state space. The multi-agent optimization module calculates the target function value of the microgrid operating cost at the current moment based on the updated state space of each microgrid agent, and determines the instantaneous reward function at the current moment by combining the current state of charge of the corresponding microgrid energy storage system, and updates the corresponding agent. Based on the updated state space of the distribution network operator's agent, the energy scheduling objective function value is calculated, the outer global reward function at the current moment is determined, and the corresponding agent is updated. The updated agent's outer global reward function and immediate reward function are weighted to calculate the comprehensive reward function, and it is determined whether the predetermined convergence condition is met. If the conditions are met, output the trained policy network and determine the optimal action for each agent.

13. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program performs the steps of the method according to any one of claims 1-11.

Images