A hierarchical cooperative scheduling method for micro-grid based on principal-agent game

Abstract

The application discloses a kind of based on master-slave game's microgrid layered cooperative scheduling method, this method constructs with distribution network operator as leading intelligent agent, each microgrid as following intelligent agent's master-slave cooperative decision framework, obtains the basic data of microgrid cooperative scheduling, and respectively constructs the global state information of leading intelligent agent and the local state information of each following intelligent agent;Price subsidy action is generated based on global state information, combined with local state information to generate joint response action, and obtain executable action by boundary clipping;Executable action is executed, and execution result and corresponding immediate reward are obtained, and centralized training-distributed execution mechanism is introduced to update master-slave cooperative decision framework, and the optimal cooperative scheduling scheme is obtained.The method can improve the cooperative scheduling ability of microgrid, and give consideration to operation safety and economy.

Description

Technical Field

[0001] This invention relates to the field of distribution microgrid collaborative scheduling technology, specifically to a hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory. Background Technology

[0002] With the widespread integration of diverse flexible resources such as distributed wind power, distributed photovoltaics, energy storage systems, electric vehicles, and interruptible loads into microgrids, the issue of coordinated scheduling between distribution networks and multiple microgrids is becoming increasingly prominent. Existing technologies employing a master-slave game theory approach for pricing and response optimization mostly focus on the game relationship between a single price signal and a single type of regulation object, or only consider microgrid-side operating cost optimization. They lack a unified coordinated scheduling mechanism that couples distribution network-side price subsidy guidance, joint response of diverse resources on the microgrid side, formation of boundary power purchase and sale results, and verification of distribution network constraints. In particular, the failure to further map boundary power purchase and sale results to the distribution network's power balance and flow constraints leads to insufficient linkage between upper-level price guidance and lower-level resource response, making it difficult to balance the safe operation of the distribution network with the economic efficiency of coordinated scheduling across multiple microgrids. Summary of the Invention

[0003] In order to overcome the above-mentioned defects of the prior art, the purpose of this invention is to provide a hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory, so as to solve the problem of insufficient upper and lower layer collaboration and difficulty in forming a unified collaborative mechanism in the existing collaborative scheduling of distribution microgrids.

[0004] To achieve the above objectives, this invention provides a hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory, comprising: Construct a master-slave collaborative decision-making framework with the distribution network operator as the leading intelligent agent and each microgrid as a following intelligent agent; Acquire basic data for the coordinated scheduling of distribution microgrids, and construct the global state information of the dominant intelligent agent and the local state information of each following intelligent agent respectively; Price subsidy actions are generated based on global state information, joint response actions are generated by combining local state information, and executable actions are obtained through boundary trimming. Execute executable actions, obtain execution results and corresponding immediate rewards, introduce a centralized training-distributed execution mechanism to update the master-slave collaborative decision-making framework, and obtain the optimal collaborative scheduling scheme.

[0005] Furthermore, the master-slave collaborative decision-making framework includes: Establish policy networks and value networks for the leading agent and each follower agent, and establish corresponding target policy networks, target value networks and experience replay pools. Establish a master-slave collaborative decision-making framework for centralized training and distributed execution to achieve joint optimization in the training phase and hierarchical decision-making in the execution phase.

[0006] Furthermore, basic data for the coordinated scheduling of distribution microgrids are acquired, and the global state information of the dominant agent and the local state information of each follower agent are constructed respectively; including: Acquire environmental prediction data, resource status data, operational boundary parameters, and historical operational feedback data, and construct a preset scheduling sequence; Based on the environmental prediction data and historical operation feedback data of the current scheduling period, construct the global state information of the dominant intelligent agent; Based on the environmental prediction data and resource status data of the corresponding microgrid, local status information of each following agent is constructed.

[0007] Furthermore, a price subsidy action is generated based on global state information, a joint response action is generated by combining local state information, and an executable action is obtained through boundary trimming; including: The global state information is input into the policy network corresponding to the dominant agent to generate price subsidy actions for each microgrid. The agent combines price subsidy actions with local state information to generate a joint response action; By performing boundary trimming on the price subsidy action and the joint response action, executable price subsidy action and joint response action are obtained.

[0008] Furthermore, execute executable actions to obtain execution results, including: The operating status of the microgrid's internal resources is updated based on the executable joint response action, and internal resource boundary constraint verification is performed. Power balance calculations are performed based on the updated internal resource operating status of the microgrid, and the boundary power purchase and sales results for each microgrid are determined by combining tie-line power constraints. The boundary power purchase and sales results are mapped to the power injection amount of the distribution network access node, the power balance relationship and power flow constraint relationship of the distribution network are established, the safety operation constraints of the distribution network are verified and the internal resource boundary constraints are reviewed, and the execution result of the current scheduling period is obtained.

[0009] Furthermore, the safety operation constraints of the distribution network are verified, and the internal resource boundary constraints are reviewed, including: Verify node voltage constraints, branch capacity constraints, and main grid switching power constraints; Simultaneously, the execution status of boundary constraints on resources within the microgrid is reviewed.

[0010] Furthermore, you will receive corresponding instant rewards, including: Based on the execution results of the current scheduling period, establish a distribution network revenue model and a microgrid cost model to determine the distribution network revenue and the operating cost of each microgrid during the current scheduling period. Based on the constraint verification results of the distribution network and microgrid, determine the constraint violation situation; The immediate reward for the dominant agent is constructed by combining the distribution network revenue with the constraint violation situation on the distribution network side. The operating cost of the microgrid is combined with the constraint violations on the microgrid side to construct the instantaneous reward for each following agent.

[0011] Furthermore, a centralized training-distributed execution mechanism is introduced to update the master-slave collaborative decision-making framework, including: The global state information, local state information, executable actions, and immediate rewards of the current scheduling period, as well as the global state information and local state information of the next scheduling period, are used to construct a state transition sample and stored in the experience replay pool. Sample from the experience replay pool, calculate the target value using the value network and the target value network, and update the value network; The updated policy network is then used to calculate the policy gradient and update the policy network. The target policy network and target value network are updated using a soft update method.

[0012] Furthermore, obtaining the optimal cooperative scheduling scheme includes: Training ends when the global reward stabilizes within a set number of consecutive training rounds, or when the maximum number of training rounds is reached. Output the optimal collaborative scheduling scheme.

[0013] Beneficial effects: This invention constructs a master-slave collaborative decision-making framework. The distribution network side generates price subsidy actions, which are then combined with local microgrid state information to generate joint response actions. Based on these joint response actions, boundary power purchase and sale results are formed, and further, distribution network constraint relationships are established. This achieves synergy between price guidance, resource response, boundary interaction, and distribution network security verification. This invention effectively connects distribution network-side scheduling demands with the microgrid-side multi-functional flexible resource adjustment process, improving the consistency between boundary power interaction results and distribution network operational constraints. While ensuring the safe operation of the distribution network, it also enhances the overall economy and coordination of distribution-microgrid collaborative scheduling. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart illustrating the overall steps of a hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory. Figure 2 This is a flowchart of the cooperative scheduling solution based on MADDPG in this embodiment; Figure 3 This is a schematic diagram of the centralized training-distributed execution architecture of the hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory. Detailed Implementation

[0016] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0017] This embodiment provides a hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory, applicable to a collaborative operation scenario consisting of a distribution network operator and multiple microgrids. The distribution network operator acts as the upper-level dominant decision-making entity, while each microgrid acts as a lower-level follower decision-making entity, and the two establish a coupling relationship through boundary power purchase and sale interactions.

[0018] Unlike existing methods that decentralized the economic dispatching of the distribution network, the resource optimization within the microgrid, and the network security verification, this embodiment constructs a unified hierarchical closed-loop collaborative dispatching process, which includes the generation of upper-level price subsidy actions, lower-level joint response actions, the formation of boundary power purchase and sale results, distribution network power flow calculation and security constraint verification, reward feedback, and policy updates. This enables the distribution network's operational objectives and the multi-dimensional flexible resource adjustment objectives of each microgrid to achieve coordinated optimization under the same dispatching framework.

[0019] Specifically, the upper-level distribution network operator generates purchase prices, sales prices, and subsidy parameters for each microgrid based on the current supply and demand environment, main grid settlement price, historical boundary interaction results, and network operation status. Each microgrid generates a joint response action for the current period based on the received price and subsidy actions and in conjunction with the operation status of local wind turbines, photovoltaics, loads, micro gas turbines, energy storage systems, electric vehicles, and interruptible air conditioners. After the joint response action is executed, boundary purchase and sales results are formed based on the internal power balance relationship of the microgrid and tie-line power constraints, and these boundary purchase and sales results are mapped to the distribution network power flow constraint and safe operation constraint verification process. Reward feedback is constructed based on revenue, cost, and constraint violation, and the upper and lower-level decision-making strategies are iteratively corrected until a distribution-microgrid collaborative scheduling scheme that meets economic and safety requirements is output.

[0020] like Figure 1 As shown, the cooperative scheduling method in this embodiment includes the following steps: A master-slave collaborative decision-making framework is constructed, with the distribution network operator as the leading intelligent agent and each microgrid as a following intelligent agent. Basic data for collaborative scheduling of distribution and microgrids are acquired, and global state information of the leading intelligent agent and local state information of each following intelligent agent are constructed respectively. Price subsidy actions are generated based on global state information, and joint response actions are generated by combining local state information. After boundary pruning, executable actions are obtained. Executable actions are executed, execution results are obtained, and corresponding immediate rewards are obtained. A centralized training-distributed execution mechanism is introduced to update the master-slave collaborative decision-making framework and obtain the optimal collaborative scheduling scheme.

[0021] To ensure consistency throughout the text, in this embodiment, the basic data for coordinated dispatch of distribution microgrids includes environmental prediction data, resource status data, operational boundary parameters, and historical operational feedback data; status information is used to characterize the input information of the dominant agent and each follower agent in the current time period; price subsidy actions are used to characterize the electricity purchase price, electricity sales price, and subsidy parameters issued by the upper layer to each microgrid; joint response actions are used to characterize the joint response actions generated by the lower-level microgrids in response to the upper-level price subsidy actions; and boundary purchase and sales results are used to characterize the boundary purchase power and boundary sales power results formed under the power balance relationship of the microgrid and tie-line constraints.

[0022] S1. Initialize the dominant agent and each follower agent, and establish a master-slave collaborative decision-making framework; wherein, the master-slave collaborative decision-making framework is a centralized training-distributed execution master-slave collaborative decision-making framework; like Figure 2As shown, the dominant agent, multiple follower agents, wind turbines, photovoltaics, micro gas turbines, energy storage systems, the main power grid, and the electricity price environment together constitute the distribution microgrid collaborative scheduling architecture. The experience replay pool is used to cache the state, action, reward, and next state samples formed during the training process to support policy updates under the centralized training and distributed execution architecture.

[0023] This step is used to initialize the policy network, value network, target network, and experience replay pool corresponding to the leading agent and each follower agent, and to establish a master-slave collaborative decision-making framework that matches the microgrid collaborative scheduling scenario.

[0024] S11. Determine the primary and secondary collaborative decision-making entities; The distribution network operator is modeled as the dominant intelligent agent in a master-slave collaborative decision-making structure, while each microgrid is modeled as a follower intelligent agent. The dominant intelligent agent acquires distribution network operation information, main grid electricity purchase and sale settlement prices, and the overall response status of the microgrid group, and generates price subsidy actions. The follower intelligent agents receive price subsidy actions from the upper-level dominant intelligent agent and, combined with local state information, generate joint response actions including micro-gas turbine output, energy storage system charging and discharging power, electric vehicle charging and discharging plans, and interruptible air conditioning load adjustments.

[0025] S12. Initialize the policy network, value network, target network, and experience replay pool; After determining the dominant agent and each follower agent in step S11, a policy network and a value network are established for the dominant agent, and a corresponding target policy network and a target value network are also established. At the same time, a policy network and a value network are established for each follower agent, and a corresponding target policy network and a target value network are also established for each follower agent.

[0026] During the initialization phase, the target policy network and target value network for each agent are set to be consistent with the corresponding policy network and value network, respectively.

[0027] At the same time, initialize the experience replay pool. D and the experience replay pool D Set as an empty pool; experience replay pool D Each sample record is saved in the order of the current time period joint state, the current time period joint action, the current time period joint reward, and the next time period joint state.

[0028] The experience playback pool D , is represented as:

[0029] In the formula, Indicates the current joint state; Indicates joint actions during the current time period; Indicates the joint reward for the current period; This indicates the joint state in the next time period after the action is performed; each sample record Stored in the experience replay pool D This is used for mini-batch sampling during subsequent training.

[0030] S13. Establish a master-slave collaborative decision-making framework of centralized training and distributed execution; This embodiment adopts a centralized training-distributed execution architecture. Specifically: During the training phase, each value network utilizes the global joint state. and joint actions A global value assessment is conducted to achieve coordination and stability in the training process. The joint state is composed of the global state information of the distribution network and the local state information of the microgrid, while the joint actions consist of price subsidy actions and joint response actions.

[0031] During the execution phase, each policy network generates actions based solely on the local state information observable by its own agent, without needing to acquire global information, thereby improving the real-time performance and engineering applicability of decision-making.

[0032] The above-described centralized training-distributed execution architecture ensures the stability of multi-agent collaborative decision-making while improving the execution flexibility of each agent in actual operation.

[0033] It should be noted that this embodiment models the microgrid collaborative scheduling process as a Markov decision process, and uses a multi-agent reinforcement learning mechanism to iteratively optimize the price subsidy actions of the upper-level dominant agent and the joint response actions of the lower-level follower agents. The Markov decision process and the multi-agent reinforcement learning mechanism are merely specific implementation methods of this embodiment and do not constitute a limitation on the technical solution of this invention.

[0034] S2. Obtain basic data for the coordinated scheduling of the distribution microgrid and construct the status information of the master and slave intelligent agents; This step is used to acquire environmental prediction data, resource status data, operational boundary parameters, and historical operational feedback data required for the coordinated dispatch of distribution microgrids. Based on this, the current state information of the leading agent and the local state information of each following agent are constructed, providing a state input basis for the subsequent generation of upper-level price subsidy actions and lower-level joint response actions. Among them, environmental prediction data, resource status data, and historical operational feedback data are used to form the decision input of each agent, and operational boundary parameters are used for subsequent action boundary setting, scheduling execution result constraint verification, and reward and penalty item construction.

[0035] S21. Obtain the environmental prediction data, resource status data, operation boundary parameters and historical operation feedback data required for the current time period's collaborative scheduling; During the current scheduling period t Obtain the following basic data: Environmental forecast data, used to characterize the supply and demand environment and external electricity price environment faced by the current period's coordinated dispatch; wherein, the supply and demand environment includes: the first i Predicted wind turbine power in microgrids Photovoltaic power forecast Forecasted load power ; The external electricity price environment includes: the electricity purchase settlement price from the main grid. Electricity price settled with the main grid .

[0036] Resource status data, which characterizes the adjustability of resources within each microgrid during the current time period; wherein, the resource status data includes: State of charge of each microgrid energy storage system electric vehicle state of charge And the indoor temperature corresponding to the interruptible air conditioning .

[0037] Operating boundary parameters include: minimum output power of the micro gas turbine. Maximum output power and maximum climbing gradient Upper limit of state of charge of energy storage system State of charge limit of energy storage system Maximum charging power and maximum discharge power Upper limit of state of charge for electric vehicles Electric vehicle state of charge limit Maximum charging power Maximum discharge power and minimum off-grid state of charge requirements Interruptible air conditioner rated power Lower limit of the comfortable temperature range Upper limit of the comfortable temperature range Maximum exchange power of the tie line between the microgrid and the distribution network ; and the upper limit of voltage at distribution network nodes Lower limit of voltage at distribution network nodes and branch capacity limit ; Historical operation feedback data includes the boundary power purchase, boundary power sales, distribution network node voltage amplitude, and branch load rate of each microgrid during the previous scheduling period.

[0038] It should be noted that the main grid purchase settlement price, the main grid sales settlement price, the upper and lower limits of distribution network node voltage, and the upper limit of branch capacity belong to the environmental prediction data and operating boundary parameters of the distribution network; the wind turbine predicted power, photovoltaic predicted power, load predicted power, energy storage system state of charge, electric vehicle state of charge, indoor temperature, and the operating boundary parameters of micro-turbines, energy storage systems, electric vehicles, and interruptible air conditioners belong to the environmental prediction data, resource status data, and operating boundary parameters of each microgrid; the maximum switching power of the tie line is used to characterize the boundary interaction capability between the microgrid and the distribution network.

[0039] S22. Construct global state information; After completing the acquisition of basic data, construct the dominant intelligent agent in the time period. t The global state information is used to characterize the input information referenced by upper-level distribution network operators when generating price subsidy actions in the current time period. Specifically, it is represented as:

[0040] In the formula, Indicates time period t Global state information; Indicates time period t Electricity purchase settlement price on the main grid; Indicates time period t Main grid electricity sales settlement price; Indicates time period t No. i Predicted wind turbine power of a microgrid; Indicates time period t No. i Predicted photovoltaic power output of a microgrid; Indicates time period t No. i Forecasted load power of a microgrid; Indicates the first time period of the previous period i The boundary power that a microgrid purchases from the distribution network; Indicates the first time period of the previous period i The boundary power of a microgrid selling electricity to the distribution network; Indicates the distribution network node in the previous time period n The voltage amplitude; Indicates the branch road in the previous time period l The load rate.

[0041] It should be noted that, in this embodiment, the global state information is composed of the current time period prediction data, the previous time period lower-level feedback information (power purchased and sold by each microgrid), and the distribution network operation feedback information (node ​​voltage, branch load rate). Among them, the previous time period lower-level feedback information and the distribution network operation feedback information are used to reflect the boundary interaction results and network operation status of the previous time period.

[0042] S23. Construct the local state information of each following agent; After constructing the upper-level state information, further construct the... i A following agent during the time period t The local state information of each following agent is used to characterize the local supply and demand environment and the operating status of internal flexible resources of the microgrid in the current time period, providing a local input basis for generating joint response actions after receiving price subsidies from the upper layer. Specifically, it is represented as follows:

[0043] In the formula, Indicates time period t No. i Local state information of each following agent; Indicates time period t No. i Predicted wind turbine power of a microgrid; Indicates time period t No. i Predicted photovoltaic power output of a microgrid; Indicates time period t No. i Forecasted load power of a microgrid; Indicates time period t No. i The state of charge of a microgrid energy storage system; Indicates time period t Access the i The first microgrid k The charged status of an electric vehicle; Indicates time period t No. k The indoor temperature corresponding to the interruptible air conditioning.

[0044] S24. Establish a preset scheduling sequence; Combination Figure 1 Discretize the scheduling period into T There are several equally spaced scheduling periods, each with a length of [length missing]. Furthermore, a training round and time-slot cycle environment corresponding to the scheduling time slots are established. In each round of training, the agent follows the time slot sequence. The process involves state observation, action generation, environment transition, and reward calculation in sequence. After completing one full scheduling cycle, the next round of training begins.

[0045] The total number of microgrids is denoted as The total number of distribution network nodes is denoted as The set of branches is denoted as , No. i The number of electric vehicles connected to each microgrid is denoted as , No. i The number of interruptible air conditioners connected to a microgrid is denoted as .

[0046] The aforementioned timing and parameter settings are applied throughout the entire training and execution process, serving to unify the decision-making step size and environmental interaction interface of each agent.

[0047] S3. Generate price subsidy actions and joint response actions based on the status information, and perform boundary trimming to obtain executable price subsidy actions and joint response actions; like Figure 1 As shown, after completing the initialization in step S1 and the state information construction in step S2, the system enters the time-segment iterative training process. This step is used to establish the time-segment linkage between the upper-level dominant decision and the lower-level follow-up response, so that there is a correspondence between the upper-level price subsidy action and the lower-level joint response action, and to ensure that all actions meet the system operation boundary constraints.

[0048] S31. Perform the training round and reset the environment; Let the training round number be The time period number is At the start of each training round, the environmental state is reset, that is, the basic data of microgrid collaborative scheduling obtained in step S2 is loaded as the initial state input for this round, and then... t =1. Specifically, the reset operation includes: Reset the state of charge of energy storage systems in each microgrid to their initial values. Reset the state of charge of each electric vehicle to its initial value. Reset the indoor temperature of each interruptible air conditioner to its initial value. ; Reset the voltage of each node and the load rate of each branch in the distribution network to the initial operating state; Clear the boundary interaction power records of the previous time period.

[0049] S32. Generate a price subsidy action based on the global status information; During the period t The dominant agent, based on its state information The original action vector for this time period is generated through its policy network, and is represented as follows:

[0050] In the formula, Indicates time period t The original action vector output by the dominant agent; The policy function representing the dominant agent; Indicates the network parameters of the dominant agent's policy; Indicates time period t Global state information.

[0051] The original action vector of the dominant agent The specific composition is as follows:

[0052] In the formula, Indicates time period t upper level to the first i The original electricity purchase price published by each microgrid; Indicates time period t upper level to the first i The original electricity sales price published by each microgrid; Indicates time period t upper level to the first i The original subsidy parameters were released by the microgrid.

[0053] It should be noted that action vectors Each component corresponds to a time period. t The electricity purchase price, electricity sales price, and subsidy parameters issued by the higher level to each microgrid, namely the original electricity purchase price, original electricity sales price, and original subsidy parameters, together constitute the price subsidy action for the current period.

[0054] S33. Generate a joint response action based on the local state information of each following agent and the price subsidy action of the upper layer; After obtaining the price subsidy action output in step S32, the first... i Each following agent combines the received price subsidy action with its local state information to form the decision input for the current time period. It should be noted that both the price subsidy action and the subsequently generated joint response action undergo boundary pruning in step S34 to obtain the final executable action. For clarity, this step illustrates the generation logic of the joint response action; the pruning process is uniformly performed in step S34.

[0055] No. i The decision input of a following agent is represented as:

[0056] In the formula, Indicates time period t No. iA follower agent's decision input; Indicates the first i Local state information of each following agent; , , They respectively represent the upper level to the first i The electricity purchase price, electricity sales price, and subsidy parameters published by each microgrid are used as executable price subsidy actions in subsequent execution after boundary trimming in step S34.

[0057] No. i A following agent based on its decision input The joint response action vector is generated through the policy network, and is represented as follows:

[0058] In the formula, Indicates time period t No. i A raw joint response action vector that follows the output of the agent; Indicates the first i A policy function for a following agent; Indicates the first i The network parameters follow the agent's policy.

[0059] No. i A primitive joint response action following the output of the agent. The specific components are as follows:

[0060] In the formula, Indicates the output power of the micro gas turbine; Indicates the energy storage charging power; Indicates the energy storage discharge power; Indicates the first k Charging power of electric vehicles; Indicates the first k Discharge power of an electric vehicle; Indicates the first k An air conditioner can be interrupted to reduce power.

[0061] It should be noted that the above-mentioned original joint response actions The components in the equation together constitute the joint response action of the microgrid in the current time period. Among them, the charging and discharging power of the energy storage system adopts independent output channels, and subsequent trimming and additional constraint processing in step S34 ensures that the two are not simultaneously non-zero; the charging and discharging power of electric vehicles is handled in the same way.

[0062] S34. Perform boundary clipping on the action vectors output by the policy network; Since the original action vectors output by the policy network may exceed the preset action boundaries, it is necessary to perform boundary pruning on the original price subsidy action vector of the dominant agent generated in step S32 and the original joint response action vectors of each follower agent generated in step S33.

[0063] Specifically, for each motion component in the motion vector, a dimension-by-dimensional correction is performed according to the preset motion boundaries; when a motion component is less than its lower boundary, the motion component is corrected to the corresponding lower boundary value; when a motion component is greater than its upper boundary, the motion component is corrected to the corresponding upper boundary value; when the motion component is between the corresponding upper and lower boundaries, its original output value remains unchanged.

[0064] S341, Boundaries of Upper-Level Price Subsidy Actions; The boundary settings for each component in the price subsidy action of the dominant intelligent agent are as follows: Electricity purchase price boundaries:

[0065] Electricity sales price boundaries: Subsidy parameter boundaries:

[0066] in,

[0067] In the above formula, and They represent the first i The lower and upper limits of electricity purchase prices for individual microgrids; and They represent the first i Lower and upper limits for electricity sales prices in individual microgrids; and They represent the first i The lower and upper limits of subsidy parameters for each microgrid.

[0068] The aforementioned boundary values ​​are pre-set according to electricity market rules and distribution network operation agreements.

[0069] After trimming the upper-level action vector, the executable price subsidy actions corresponding to the current time period's electricity purchase price, electricity sales price, and subsidy parameters issued to each microgrid are as follows:

[0070] S342, Lower-level joint response action boundary; No. i The boundary settings for each component in the joint response action of the following agents are as follows: Micro-turbine output power boundary:

[0071] Energy storage system power boundary: ,

[0072] Electric vehicle power boundary: ,

[0073] Interruptible air conditioning power reduction boundary:

[0074] In the above formula, and They represent the first i Minimum and maximum output power of a microgrid micro-turbine; and These represent the maximum charging power and maximum discharging power of the energy storage, respectively. and They represent the first k Maximum charging power and maximum discharging power of a single electric vehicle; Indicates the first k The rated power of an interruptible air conditioner.

[0075] After pruning the lower-level action vectors, the executable joint response actions corresponding to micro-turbines, energy storage systems, electric vehicles, and interruptible air conditioners in each microgrid during the current time period are as follows:

[0076] S35, Action Execution and State Transition; After completing the motion trimming, the executable price subsidy motions will be... The action is distributed to each following agent, and each following agent simultaneously executes the pruned joint response action. Based on the current state and the actions performed, the system transitions to the next time-period joint state according to the distribution network power flow model and the microgrid equipment dynamic model. And calculate the joint reward for the current period based on the execution results of the current period. Then ordered t = t +1, repeat steps S32 to S35 until all rounds in the current cycle are completed. T Scheduling decisions for each time period. When t > T When the current round ends, the next training round begins, the environment is reset, and the above process is repeated.

[0077] S4. Determine the boundary power purchase and sale results based on the joint response actions, and perform safety operation constraint verification of the distribution network; This step is used to input the executable joint response action generated in step S34 into the microgrid operating environment, and sequentially complete the updating of the internal resource status of the microgrid, the determination of the boundary power purchase and sale results, the calculation of the power flow of the distribution network, and the constraint verification of the distribution network and the microgrid, so as to provide scheduling execution results that conform to the physical operation law for subsequent reward function calculation and experience sample storage.

[0078] S41. Based on the joint response action, update the internal resource operation status of the microgrid and perform internal resource boundary verification; After obtaining the cropping result from step S34 i A group of agents jointly respond to actions. Subsequently, the operating status of the microgrid's internal resources is updated based on each action component, and internal resource boundary verification is performed. The operating status includes the state of charge of the energy storage system, the state of charge of the electric vehicle, the indoor temperature of the interruptible air conditioner, the output power of the micro gas turbine, and the actual power of the wind turbine and photovoltaic system.

[0079] S411. Update the state of charge of the energy storage system and verify the constraints of the energy storage system; According to step S34 i Energy storage and charging power output by the following intelligent agent and energy storage discharge power By combining the energy storage charging and discharging efficiency, the length of the discrete time period, and the rated energy capacity of the energy storage system, the state of charge of the energy storage system in the current time period is updated to characterize the change in the operating state of the energy storage system after executing the joint response action.

[0080] The formula for updating the state of charge of an energy storage system is expressed as:

[0081] In the formula, Indicates time period t No. i State of charge of microgrid energy storage systems; This indicates the state of charge of the energy storage system in the previous time period; Indicates energy storage charging efficiency; Indicates the energy storage discharge efficiency; Indicates time period t Energy storage charging power; Indicates time period t Energy storage discharge power; Indicates the length of a discrete time period; Indicates the first i Rated energy capacity of microgrid energy storage system.

[0082] After obtaining the current state of charge (SBC) of the energy storage system, the constraints of the energy storage system are verified based on the upper and lower limits of the SBC, the maximum charging power, the maximum discharging power, and the charging and discharging direction indicators to ensure that the energy storage system is within the allowable operating range during the current period and performs only a single-direction charging or discharging action within the same period. The constraints of the energy storage system include the energy storage SBC boundary constraints and the energy storage charging and discharging power constraints.

[0083] The energy storage charge state boundary constraints are expressed as follows:

[0084] In the formula, Indicates the lower limit of the energy storage's state of charge; This indicates the upper limit of the energy storage state of charge; the meanings of the other parameters follow the definitions above.

[0085] Energy storage charging and discharging power constraints are expressed as follows:

[0086] In the formula, Indicates the maximum charging power of the energy storage; Indicates the maximum discharge power of the energy storage; Indicates the direction of energy storage charging and discharging. When the charging direction is valid, The time indicates that the discharge direction is valid.

[0087] S412. Update the state of charge of the electric vehicle and verify the electric vehicle constraints; According to step S34 i Electric vehicle charging power output by a follow-up intelligent agent and electric vehicle discharge power By combining the rated capacity of the electric vehicle battery, the charging and discharging efficiency, and the length of the discrete time period, the state of charge of the electric vehicle connected in the current time period is updated to characterize the change in the operating state of the electric vehicle after executing the joint response action.

[0088] The formula for updating the state of charge of an electric vehicle is expressed as:

[0089] In the formula, Indicates time period t Access the i The first microgrid k The charged status of an electric vehicle; This indicates the state of charge of the electric vehicle in the previous time period; Indicates time period t No. k Charging power of electric vehicles; Indicates time periodt No. k Discharge power of an electric vehicle; Indicates the first k Rated capacity of electric vehicle battery; , and The meanings of these terms follow the definitions mentioned above.

[0090] After obtaining the electric vehicle's state of charge (SBC), the electric vehicle constraints are verified based on the upper and lower limits of the SBC, the access status flag, the maximum charging power, the maximum discharging power, and the minimum off-grid SBC requirement corresponding to the planned off-grid time. This ensures that the electric vehicle only participates in regulation when it is in the access state and meets the preset energy demand when it is off-grid. The electric vehicle constraints include the electric vehicle SBC boundary constraints, the electric vehicle charging and discharging power constraints, and the electric vehicle off-grid demand constraints.

[0091] The boundary constraints of the electric vehicle's charge state are expressed as follows:

[0092] In the formula, Indicates the first k The minimum charge level of an electric vehicle; Indicates the first k The upper limit of the state of charge of an electric vehicle.

[0093] The charging and discharging power constraints of electric vehicles are expressed as follows:

[0094] In the formula, Indicates the first k Maximum charging power of a vehicle; Indicates the first k Maximum discharge power of an electric vehicle; Indicates the first k Charging / discharging direction signs for electric vehicles; Indicates the access status flag, when This indicates that the vehicle is in the connected state. This indicates that the vehicle is not connected.

[0095] The off-grid demand constraint for electric vehicles is expressed as:

[0096] In the formula, Indicates the first k The planned offline time for 10 electric vehicles; Indicates the first k Minimum state of charge requirement for off-grid electric vehicles.

[0097] S413. Update the indoor temperature of the interruptible air conditioning system and verify the comfort constraints. According to step S34 i An interruptible air conditioner that follows the output of an intelligent agent to reduce power. By combining outdoor temperature, building heat exchange parameters, and air conditioning rated power, the indoor temperature for the current period is updated to characterize the thermal dynamic response of the interruptible air conditioning after it performs a reduction action.

[0098] The formula for updating the indoor temperature of an interruptible air conditioner is expressed as follows:

[0099] In the formula, Indicates time period t No. i The first microgrid k The indoor temperature corresponding to each air conditioner; This indicates the indoor temperature corresponding to the previous time period; Indicates time period t Outdoor temperature; Indicates the building's heat exchange coefficient; This represents the coefficient indicating the impact of changes in air conditioning load on indoor temperature. Indicates the first k The rated power of each air conditioner; Indicates time period t No. k The air conditioner reduced its power.

[0100] After obtaining the indoor temperature, the indoor temperature is verified according to the preset comfort temperature range to ensure that the interruptible air conditioner does not exceed the user's acceptable comfort boundary while participating in the scheduling.

[0101] Comfort constraints are expressed as:

[0102] In the formula, Indicates the first k The lower limit of the comfortable temperature range for air conditioning; Indicates the first k The upper limit of the air conditioning comfort temperature range; the meanings of the other parameters follow the aforementioned definitions.

[0103] S414. Determine the actual implementation results of renewable energy and micro-turbines; Based on the current photovoltaic power forecast The photovoltaic (PV) power is allocated between the predicted PV power and the curtailed PV power to determine the actual performance of PV resources in the current time period. In this embodiment, the actual PV power is the minimum of the predicted PV power and the microgrid's absorption capacity, and the curtailed PV power is the difference between the predicted power and the actual power.

[0104] The actual power relationship of photovoltaic power is expressed as follows:

[0105] In the formula, Indicates time period t No. i Actual photovoltaic power of a microgrid; Indicates time period t No. i Curtailed solar power in microgrids; Indicates time period t No. i Predicted photovoltaic power output for microgrids.

[0106] Similarly, based on the predicted power output of wind turbines for the current period... The predicted wind turbine power is allocated between the actual wind turbine power and the curtailed wind power to determine the actual execution result of wind power resources in the current period.

[0107] The actual power relationship of the wind turbine is expressed as follows:

[0108] In the formula, Indicates time period t No. i Actual power of each microgrid wind turbine; Indicates time period t No. i wind curtailment power of individual microgrids; Indicates time period t No. i Predicted power output of wind turbines in a microgrid.

[0109] According to step S34 i The output power of the micro-engine following the intelligent agent's output. By combining the minimum and maximum output power of the micro gas turbine with the ramp boundary of adjacent time periods, the output constraint of the micro gas turbine in the current time period is verified to determine the actual execution result of the micro gas turbine in the current time period.

[0110] The output boundary constraints of the micro gas turbine are expressed as follows:

[0111] In the formula, Indicates time period t No. i Output power of a microgrid micro-gas turbine; This indicates the minimum output power of the micro gas turbine; This indicates the maximum output power of the micro gas turbine.

[0112] The ramp constraint for a micro gas turbine is expressed as:

[0113] In the formula, This indicates the output power of the micro-turbine in the previous period; This indicates the maximum gradient of the micro-engine; the meanings of the other parameters follow the definitions above.

[0114] S42. Determine the boundary power purchase and sale results based on the microgrid power balance relationship and tie-line power constraints; After completing the update of the internal resource operation status, the results of the current time period boundary power purchase and sale are further determined based on the internal power balance relationship of the microgrid.

[0115] Specifically, the actual power of photovoltaic Actual power of the fan Micro-turbine output power Energy storage and discharge power and electric vehicle discharge power The actual execution results, and the microgrid load power Energy storage charging power Electric vehicle charging power And can interrupt air conditioning to reduce power Substitute these values ​​into the internal power balance relationship of the microgrid to obtain the boundary power purchase and boundary power sales results of the microgrid for the current period.

[0116] The formula for the power balance relationship within a microgrid is expressed as follows:

[0117] In the formula, Indicates the actual power output of the photovoltaic system; Indicates the actual power of the fan; Indicates the output power of the micro gas turbine; Indicates the energy storage discharge power; Indicates the first k Discharge power of an electric vehicle; Indicates the boundary power purchase capacity; Indicates load power; Indicates the energy storage charging power; Indicates the first k Charging power of electric vehicles; Indicates the boundary power sales capacity; Indicates the first k An interruptible air conditioner can reduce its power output; Indicates the first i The number of electric vehicles connected to each microgrid; Indicates the first i Number of microgrid-connected interruptible air conditioners.

[0118] It should be noted that the boundary power purchase capacity With boundary power sales The complementary relationship must be satisfied, meaning that at least one of the boundary power purchase and the boundary power sales is zero within the same time period, to characterize the uniqueness of the boundary interaction direction.

[0119] Obtain the boundary power purchase capacity and boundary power sales Subsequently, the tie-line power constraints are verified to ensure that the boundary power exchange between each microgrid and the distribution network does not exceed the allowable range, and that the boundary interaction direction is unique within the same time period.

[0120] Tie line power constraints are expressed as follows:

[0121]

[0122] In the formula, Indicates the first i Maximum switching power of microgrid interconnects; Indicates the direction of boundary power exchange, when When the time indicates that the electricity purchase direction is valid, This indicates that the electricity sales direction is valid.

[0123] S43. Based on the results of boundary power purchase and sale, establish the power balance relationship and power flow constraint relationship of the distribution network, and verify the safety operation constraints of the distribution network. S431. Establish the power balance relationship and power flow constraint relationship of the distribution network; After obtaining the power purchase and sale results at the boundaries of each microgrid, these results are mapped to the power injection amount of the corresponding access nodes. This further establishes the power balance relationship and power flow constraint relationship of the distribution network, so as to characterize the network operation characteristics and power transmission constraints of the distribution network in the current period.

[0124] The power balance relationship of the distribution network is expressed as follows:

[0125]

[0126] In the formula, and They represent time periods respectively. t Reactive power purchased and reactive power sold between the distribution network and the main grid; and They represent time periods respectively. t No. i The reactive power output of a microgrid to the distribution network and the reactive power absorbed from the distribution network; and These represent the total active load and total reactive load of the distribution network, respectively. Indicates time period t Reactive power loss in the distribution network.

[0127] The power flow constraints of a distribution network branch are expressed as follows:

[0128]

[0129]

[0130] In the formula, Indicates a side road; and They represent time periods respectively. t branch road The active and reactive currents; and They represent time periods respectively. t node j Active and reactive loads; and Representing access nodes j microgrids during time periods t Active and reactive power injected into nodes; and Representing nodes respectively i and nodes j The voltage amplitude; and Representing branch roads Resistance and reactance; This represents the reference voltage.

[0131] S432. Verify safe operation constraints based on power balance and power flow constraints in the distribution network. After establishing the power balance relationship and power flow constraint relationship of the distribution network, the safety operation constraints of the distribution network in the current period are further verified, and the execution of resource constraints within the microgrid is reviewed simultaneously, so as to ensure that the scheduling execution results in the current period meet the physical operation requirements of the distribution network side and the microgrid side.

[0132] Among them, the constraints for safe operation include: node voltage constraints, branch capacity constraints, and main network switching power constraints.

[0133] The node voltage constraint is expressed as:

[0134] In the formula, Indicates time period t node n The voltage amplitude; Represents a noden Lower voltage limit; Represents a node n Voltage limit.

[0135] Branch capacity constraints are expressed as:

[0136] In the formula, Indicates time period t branch road The meritorious trend; Indicates time period t branch road The unproductive current; Indicates safety parameters; Indicates a branch The rated apparent capacity limit.

[0137] The main network switching power constraint is expressed as:

[0138]

[0139] In the formula, Indicates time period t The power that the distribution network purchases from the main grid; Indicates time period t The amount of electricity sold from the distribution network to the main grid; This indicates the maximum exchange power between the distribution network and the main grid; This indicates the direction of the main network switching.

[0140] In addition, the execution status of the operational boundary constraints of the energy storage system, electric vehicle, interruptible air conditioner, and micro gas turbine in step S41 needs to be reviewed synchronously. If any constraint is not met, a corresponding penalty term is applied to the reward function in step S5, and the scheduling result for that period is recorded as infeasible, but the state transition process of the environment is not interrupted to ensure the continuity of the training process.

[0141] Through the above verification process, it can be ensured that the current time period boundary power purchase and sale results are consistent with the lower-level internal resource joint response process and meet the requirements of safe operation of the distribution network, providing executable scheduling results for subsequent reward feedback construction and network parameter updates.

[0142] S44. Generate the joint state for the next time period; After completing the generation of boundary power purchase and sale results, the establishment of power balance relationship of distribution network, the establishment of power flow constraint relationship and the verification of security constraints, the joint state of the next time period is generated based on the updated microgrid resource status, boundary power purchase and sale results and distribution network operation status, and is used as the input basis for writing experience samples and generating actions for the next time period.

[0143] The joint state for the next time period is represented as follows:

[0144] In the formula, Indicates the global status information for the next time period; This represents the local state information of the i-th following agent in the next time period.

[0145] Subsequently, t = t +1, return to step S3, and proceed to the decision-making process for the next time period, until all training rounds are completed. T The process continues until each time period is completed.

[0146] S5. Calculate the reward and update the policy network based on the revenue, cost and constraint violation, and output the microgrid collaborative scheduling scheme. This step is used to convert the boundary power purchase and sale results, distribution network power balance relationship, power flow constraint relationship and security constraint verification results formed in step S4 into the corresponding benefit items, cost items and penalty items of the upper-level dominant agent and each lower-level follower agent, further generate real-time rewards, and complete the network parameter update under the centralized training-distributed execution architecture; after the training rounds meet the convergence conditions or reach the maximum training rounds, the globally optimal collaborative scheduling scheme of the distribution microgrid is output.

[0147] It should be noted that the training and update mechanism in this step is only used as an optimization method for the aforementioned closed loop of price subsidy action - joint response action - boundary power purchase and sale result - network constraint verification technology. Its role is to continuously correct the upper-level price subsidy action and the lower-level joint response action, so that they gradually approach the global optimal collaborative scheduling result, without changing the core technical solution of this invention.

[0148] S51. Establish a single-period revenue model for distribution network operators; Distribution network operators during time periods t The single-period revenue is used to characterize the comprehensive economic effect obtained by the distribution network side under the combined effect of upper-level price subsidy actions and lower-level boundary interactions in the current period. The comprehensive economic effect includes: the revenue from power purchase and sale between the distribution network and each microgrid, the revenue from power purchase and sale settlement between the distribution network and the main grid, network loss costs, subsidy expenditures, and local renewable energy consumption incentives.

[0149] Distribution network operators during time periods t The single-period return is expressed as:

[0150] In the formula, Indicates time period t Revenue of power distribution network operators in a single time period; Indicates time period t The electricity price offered by distribution network operators to microgrids; Indicates time period t The electricity purchase price for microgrids by distribution network operators; Indicates time period t The subsidy parameters released by distribution network operators for microgrids Indicates the first i Power purchase capacity at the boundary of each microgrid; Indicates the first i Electricity sales capacity at the boundary of each microgrid; This indicates the electricity price settled through the main grid purchase. This indicates the electricity price settled through the main grid. This indicates the power that the distribution network purchases from the main grid; This indicates the power output of the distribution network sold to the main grid; This represents the network loss cost coefficient; This indicates the active power loss in the distribution network; This represents the local renewable energy consumption incentive coefficient; Indicates the first i Actual photovoltaic power of a microgrid; Indicates the first i Actual power of each microgrid wind turbine; Indicates the total number of microgrids; This indicates the incentive parameters for local renewable energy consumption.

[0151] S52. Establish a single-period operation cost model for each microgrid; No. i microgrids during time period t The single-period operating cost is used to characterize the comprehensive operating costs incurred by the microgrid in completing the coordinated response actions of its internal resources within the current period. The comprehensive operating costs include: boundary power purchase and sale costs, micro-gas turbine operating costs, energy storage regulation costs, electric vehicle regulation costs, interruptible air conditioning regulation costs, wind and solar curtailment penalty costs, and carbon emission costs.

[0152] No. i microgrids during time period t The single-period operating cost is expressed as:

[0153] In the formula, Indicates the first i microgrids during time period t The overall operating cost; Indicates the upper level to the first i The electricity purchase price published by each microgrid; Indicates the upper level to the first iElectricity prices published by individual microgrids; Indicates time period t Subsidy parameters released by distribution network operators for microgrids; Indicates the first i Power purchase capacity at the boundary of each microgrid; Indicates the first i Electricity sales capacity at the boundary of each microgrid; This indicates the operating cost of the micro gas turbine; Indicates the cost of energy storage regulation; Indicates the adjustment cost of electric vehicles; This indicates the cost of interruptible air conditioning adjustments; This indicates the penalty cost for abandoning wind and solar power. This indicates the cost of carbon emissions.

[0154] Specifically, the operating cost of the micro gas turbine is expressed as:

[0155] In the formula, , and They represent the first i Weighting of operating costs for microgrids and micro gas turbines; This indicates the output power of the micro-engine.

[0156] The energy storage regulation cost is expressed as:

[0157] In the formula, Indicates the first i Weight of energy storage regulation costs for individual microgrids; Indicates the energy storage charging power; This indicates the energy storage discharge power.

[0158] The adjustment cost of electric vehicles is expressed as:

[0159] In the formula, Indicates the first k Weighting of adjustment costs for electric vehicles; Indicates the first k Charging power of electric vehicles; Indicates the first k Discharge power of an electric vehicle; Indicates the first i Number of electric vehicles connected to a microgrid.

[0160] Air conditioning adjustment costs are expressed as follows:

[0161] In the formula, Indicates the first k Weighting of air conditioning adjustment costs; Indicates the first k The air conditioner reduced its power; Indicates the first i Number of microgrid-connected interruptible air conditioners.

[0162] The penalty cost of curtailing wind and solar power is expressed as follows:

[0163] In the formula, Indicates the first i The penalty weight for wind and solar power curtailment in individual microgrids; Indicates the power of light discarded; This indicates the amount of wind power that has been curtailed.

[0164] The cost of carbon emissions is expressed as:

[0165] In the formula, Indicates the first i Carbon cost weighting for each microgrid; This indicates the carbon emission weight per unit power of a micro gas turbine; This indicates the output power of the micro-engine.

[0166] S53. Convert revenue, costs, and constraint violations into immediate rewards; After obtaining the revenue from the upper-level distribution network operator and the operating costs of each lower-level microgrid, an immediate reward system is constructed based on the current time-limited constraint violation situation. The upper-level immediate reward is used to guide the leading agent to optimize price subsidy actions; the lower-level immediate reward is used to guide the following agents to optimize their joint response actions regarding internal resources.

[0167] S531, Construct immediate rewards and penalties for the dominant intelligent agent; The dominant agent in time period t The reward is represented as:

[0168] In the formula, Indicates time period t Instant rewards for the leading intelligent agent; This indicates the revenue of the distribution network operator; This indicates a penalty for violating constraints on the distribution network side.

[0169] The penalty for violating distribution network side constraints is represented as follows:

[0170] In the formula, , and These represent the penalty weights for node voltage exceeding limits, branch capacity exceeding limits, and main network switching exceeding limits, respectively.

[0171] S532, Construct instant rewards and penalties for the following agent; No. i A following agent during the time period t The instant reward is represented as:

[0172] In the formula, Indicates time period t No. i Instant rewards for each following agent; Indicates the first i Comprehensive operating cost of a microgrid; This indicates a penalty for violating microgrid-side constraints.

[0173] The penalty for violating microgrid-side constraints is represented as follows:

[0174] In the formula, , , , and These represent the penalty weights for exceeding the energy storage state of charge limit, the penalty weight for not meeting the off-grid demand of electric vehicles, the penalty weight for exceeding the indoor temperature comfort limit, the penalty weight for exceeding the tie line switching power limit, and the penalty weight for exceeding the micro-turbine hill-climbing limit, respectively.

[0175] It should be noted that the above-mentioned penalties The penalties include those for exceeding the charge state limits of energy storage, failing to meet the off-grid demand of electric vehicles, exceeding the limits for indoor temperature comfort, exceeding the limits for tie line switching power, and exceeding the limits for micro-turbine hill climbing.

[0176] S54. Write the samples into the experience replay pool and update the network parameters; After writing the current joint state, joint action, joint reward, and next state into the experience replay pool, the parameters of the value network, policy network, and target network are updated based on the sampled samples in the experience replay pool.

[0177] The joint actions during the current time period are represented as follows:

[0178] In the formula, Indicates time period t Joint actions; Indicates time period t The price subsidy action that drives the output of the leading intelligent agent; Indicates time period t No. i A joint response action that follows the output of an agent.

[0179] The combined reward for the current period is represented as follows:

[0180] In the formula, Indicates time period t Joint rewards; Indicates time period t Instant rewards for the dominant agent; Indicates time period t No. i An instant reward for following an intelligent agent.

[0181] From the experience replay pool D Random collection M The sample is used to construct the first sample. j The first sample corresponding to the i The target value corresponding to each intelligent agent.

[0182] No. j The first sample corresponding to the i The target value of an agent is represented as:

[0183] In the formula, Indicates the first j The first sample corresponding to the sampling sample i The target value of an intelligent agent; Indicates the first j The first sample in the sample i Instant rewards for each agent; Indicates the discount weight; Indicates the first i A network of target values ​​for individual agents; Indicates the first i Target value network parameters for each agent; Indicates the first j The joint state of the next time period for each sampled item; Indicates the first j The next time period of each sampled sample is generated by the joint action of each target policy network.

[0184] Among them, the next phase of joint actions Represented as:

[0185] In the formula, Indicates the first jEach sample corresponds to a price subsidy action generated by the distribution network operator's target strategy network in the next time period; Indicates the first j The sampled item corresponds to the next time period from the first sample. i The joint response action generated by the target strategy network of each microgrid.

[0186] Based on the aforementioned target values, the first i The value network loss function for each agent is expressed as:

[0187] In the formula, Indicates the first i The loss function of the value network for each agent; Indicates the first i Value network parameters of each agent; M This represents the number of mini-batch samples drawn in a single training session. j Indicates the sample number; Indicates the first i The agent in the th... j Current value network output on each sample; Indicates the first j The joint state of the samples; Indicates the first j The combined action of the samples.

[0188] After completing the value network update, the first i The policy network parameters of each agent are updated according to the policy gradient, as follows:

[0189] In the formula, Indicates the first i Policy gradient of an agent's policy network; Indicates the first i Individual agent policy network parameters; Indicates the first i A policy network for individual agents; Indicates the first j The first sample in the sample i The current decision input corresponding to each agent.

[0190] It should be noted that the first i In a master-slave collaborative decision-making framework, any decision-making agent includes the upper-level master agent. and each of the lower-level following intelligent agents When the first i When an agent is the dominant agent, the current decision input Corresponding to the global state information constructed in step S22 When the first i When an agent is a follower agent, the current decision input Corresponding to the microgrid decision input information constructed in step S33 .

[0191] To improve training stability, soft updates need to be performed on the target network.

[0192] The target network soft update is represented as:

[0193] In the formula, Indicates the first i Network parameters for the target policy of each agent; Indicates the first i Network parameters for the target value of each agent; Indicates the soft update parameter, satisfying .

[0194] It should be noted that the update mechanisms of the experience replay pool, value network, policy network, and target network in this embodiment are only used to iteratively optimize the aforementioned price subsidy actions and joint response actions. Their role is to continuously correct the upper and lower layer action outputs based on the boundary power purchase and sale results, distribution network power balance relationship, power flow constraint relationship, and security constraint verification results formed in step S4, thereby gradually approaching the optimal collaborative scheduling scheme.

[0195] S55. Iterate cyclically according to time periods and training rounds, and output a collaborative scheduling scheme; After completing the network parameter update in step S54, determine whether the current training round has reached the preset time period termination condition.

[0196] If the current time period satisfies Then let Then return to step S3 to proceed to the next time period's price subsidy action generation and joint response action generation process; if all time periods of the current training round have been completed, then let Reset the environment and proceed to the next training round.

[0197] When the global reward stabilizes within a preset number of consecutive training rounds, or when the maximum number of training rounds is reached... Upon completion, the training process ends, and a collaborative scheduling scheme is output. The collaborative scheduling scheme must include at least the optimal electricity purchase price offered by the distribution network operator to each microgrid. Best electricity price and optimal subsidy parameters Optimal output power of micro-turbines within each microgrid Optimal charging power of energy storage system Optimal discharge power of energy storage system Optimal charging power for electric vehicles Optimal discharge power of electric vehicles and the optimal power reduction of interruptible air conditioning Boundary power purchase results for each microgrid and border electricity sales results The coordinated scheduling results satisfy the distribution network node voltage constraints, branch capacity constraints, and main grid switching power constraints, including node voltage in each time period. Branch roads have contributed to the trend and branch road unproductive current .

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory, characterized in that, include: Construct a master-slave collaborative decision-making framework with the distribution network operator as the leading intelligent agent and each microgrid as a following intelligent agent; Acquire basic data for the coordinated scheduling of distribution microgrids, and construct the global state information of the dominant intelligent agent and the local state information of each following intelligent agent respectively; Price subsidy actions are generated based on global state information, joint response actions are generated by combining local state information, and executable actions are obtained through boundary trimming. Execute executable actions, obtain execution results and corresponding immediate rewards, introduce a centralized training-distributed execution mechanism to update the master-slave collaborative decision-making framework, and obtain the optimal collaborative scheduling scheme.

2. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory as described in claim 1, characterized in that, The master-slave collaborative decision-making framework includes: Establish policy networks and value networks for the leading agent and each follower agent, and establish corresponding target policy networks, target value networks and experience replay pools. Establish a master-slave collaborative decision-making framework for centralized training and distributed execution to achieve joint optimization in the training phase and hierarchical decision-making in the execution phase.

3. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory as described in claim 1, characterized in that, Acquire basic data for the coordinated scheduling of distribution microgrids, and construct the global state information of the dominant agent and the local state information of each follower agent; including: Acquire environmental prediction data, resource status data, operational boundary parameters, and historical operational feedback data, and construct a preset scheduling sequence; Based on the environmental prediction data and historical operation feedback data of the current scheduling period, construct the global state information of the dominant intelligent agent; Based on the environmental prediction data and resource status data of the corresponding microgrid, local status information of each following agent is constructed.

4. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory as described in claim 1, characterized in that, A price subsidy action is generated based on global state information, a joint response action is generated by combining local state information, and an executable action is obtained through boundary trimming; including: The global state information is input into the policy network corresponding to the dominant agent to generate price subsidy actions for each microgrid. The agent combines price subsidy actions with local state information to generate a joint response action; By performing boundary trimming on the price subsidy action and the joint response action, executable price subsidy action and joint response action are obtained.

5. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory according to claim 1, characterized in that, Execute executable actions and obtain execution results, including: The operating status of the microgrid's internal resources is updated based on the executable joint response action, and internal resource boundary constraint verification is performed. Power balance calculations are performed based on the updated internal resource operating status of the microgrid, and the boundary power purchase and sales results for each microgrid are determined by combining tie-line power constraints. The boundary power purchase and sales results are mapped to the power injection amount of the distribution network access node, the power balance relationship and power flow constraint relationship of the distribution network are established, the safety operation constraints of the distribution network are verified and the internal resource boundary constraints are reviewed, and the execution result of the current scheduling period is obtained.

6. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory according to claim 5, characterized in that, Verification of distribution network safety operation constraints and review of internal resource boundary constraints include: Verify node voltage constraints, branch capacity constraints, and main grid switching power constraints; Simultaneously, the execution status of boundary constraints on resources within the microgrid is reviewed.

7. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory as described in claim 5, characterized in that, Receive corresponding instant rewards, including: Based on the execution results of the current scheduling period, establish a distribution network revenue model and a microgrid cost model to determine the distribution network revenue and the operating cost of each microgrid during the current scheduling period. Based on the constraint verification results of the distribution network and microgrid, determine the constraint violation situation; The immediate reward for the dominant agent is constructed by combining the distribution network revenue with the constraint violation situation on the distribution network side. The operating cost of the microgrid is combined with the constraint violations on the microgrid side to construct the instantaneous reward for each following agent.

8. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory according to claim 2, characterized in that, The master-slave collaborative decision-making framework is updated by introducing a centralized training-distributed execution mechanism, including: The global state information, local state information, executable actions, and immediate rewards of the current scheduling period, as well as the global state information and local state information of the next scheduling period, are used to construct a state transition sample and stored in the experience replay pool. Sample from the experience replay pool, calculate the target value using the value network and the target value network, and update the value network; The updated policy network is then used to calculate the policy gradient and update the policy network. The target policy network and target value network are updated using a soft update method.

9. The hierarchical collaborative scheduling method for distribution microgrids based on master-slave game theory according to claim 1, characterized in that, Obtaining the optimal cooperative scheduling scheme includes: Training ends when the global reward stabilizes within a set number of consecutive training rounds, or when the maximum number of training rounds is reached. Output the optimal collaborative scheduling scheme.

Images