Multi-agent reinforcement learning regional energy collaborative scheduling method and system

Abstract

The application provides a multi-agent reinforcement learning regional energy collaborative scheduling method and system, belongs to the field of regional energy system scheduling, and constructs the coupling degree and the coupling degree matrix among intelligent agents; an observation vector is input into a policy network to obtain a decision action; individual basic rewards and system economic rewards are calculated to construct constraint benchmark rewards; individual differentiated basic rewards and distribution rewards are calculated; the decision action is input into a physical quantity prediction network to calculate a physical consistency reward, obtain a final reward and a global reward, and calculate a target return; observation vectors and decision actions of all intelligent agents are spliced, a global joint observation vector and a joint action vector are spliced and then input into a value network for training; a local state set is input into the trained policy network to output a scheduling instruction, and the scheduling instruction is input into the trained value network to output an evaluation result; and the problems of rigid depiction of the coupling relationship of intelligent agents, lack of a collaborative benefit distribution mechanism, and insufficient decision physical consistency are solved.

Description

Technical Field

[0001] This invention relates to the field of regional energy system scheduling technology, and in particular to a multi-agent reinforcement learning method and system for regional energy collaborative scheduling. Background Technology

[0002] With the continuous increase in the penetration rate of distributed resources such as photovoltaics, energy storage, and adjustable loads in regional energy systems, the energy system is evolving towards a form of "multi-energy coupling and distributed coordination," providing important support for the low-carbon transformation of the energy system. However, the randomness and volatility of distributed resources, as well as the complex interaction between multiple energy entities in the region, have brought significant challenges to traditional energy dispatching technologies: (1) Traditional centralized dispatching methods rely on global modeling and optimization. When faced with large-scale distributed resources, the computational complexity increases exponentially, and the real-time response capability is insufficient; (2) Although distributed dispatching methods reduce computational pressure, they lack a global coordination mechanism, are prone to local optima, and are difficult to guarantee the overall economic efficiency and security of the system.

[0003] Multi-Agent Deep Reinforcement Learning (MADRL), with its advantages of "distributed decision-making + autonomous learning," is gradually becoming a core technology for regional energy collaborative scheduling. Chinese invention patent application CN118627779A, entitled "Optimal Scheduling Method for Multi-Energy Coupled Microgrids Based on Multi-Agent Reinforcement Learning," uses a multi-agent reinforcement learning model trained with a near-end policy optimization algorithm. This model, combined with the current state variables, determines the action space for each agent and optimizes the scheduling of the multi-energy coupled microgrid based on this action space. This approach adapts to different source-load conditions and improves the scheduling capability of the multi-energy coupled microgrid.

[0004] However, existing scheduling methods based on multi-agent deep reinforcement learning still have key defects: (1) Rigid characterization of agent coupling relationship: Existing methods mostly assume that the relationship between agents is fixed, which cannot adapt to the dynamic changes in resource association strength under scenarios such as photovoltaic power output fluctuation and load change, which can easily lead to scheduling decision deviation; (2) Lack of collaborative benefit allocation mechanism: The additional benefits generated by multi-agent cooperation are not quantified and fairly allocated, the incentive for agent cooperation is insufficient, and it is easy to fall into the dilemma of local optimum; (3) Low efficiency of constraint learning: Existing technologies mostly adopt the method of "introducing all constraints at once", and agents frequently violate hard constraints in the early stage of training, resulting in policy oscillation and slow convergence speed; (4) Insufficient physical consistency of decision: The scheduling decision has low consistency with the physical laws of power system Kirchhoff's law and power balance, and the safety risk is high in engineering applications.

[0005] Meanwhile, the existing multi-agent centralized training architecture has obvious shortcomings: (1) global information fusion only relies on the simple summary of observation and action, without combining the real-time power grid status to dynamically characterize the coupling relationship of the agents; (2) the loss function design only focuses on policy stability and does not incorporate coupling and coordination constraints, resulting in insufficient policy coordination and physical system adaptability, making it difficult to support the efficient and safe scheduling of regional energy systems.

[0006] In summary, existing technologies have not yet solved the integrated scheduling requirements of "dynamic coupling relationship characterization, efficient collaborative training, and physical consistency guarantee," and there is an urgent need for a multi-agent reinforcement learning scheduling method and training architecture that can adapt to the dynamic characteristics of regional energy systems. Summary of the Invention

[0007] The technical problem to be solved by this invention is: how to solve the problems of rigid characterization of agent coupling relationship, lack of collaborative benefit allocation mechanism and insufficient physical consistency of decision in the existing scheduling methods based on multi-agent deep reinforcement learning.

[0008] This invention solves the above-mentioned technical problems through the following technical solution: a multi-agent reinforcement learning regional energy cooperative scheduling method, comprising:

[0009] S1. Obtain the local state set of all agents, construct the coupling degree between agents based on energy interaction, power correlation and spatiotemporal distance, and construct the coupling degree matrix based on the coupling degree.

[0010] S2. Based on the agent's local state set and coupling degree matrix, construct the agent's observation vector, input the agent's observation vector into the policy network, and combine it with the coupling degree matrix to obtain the agent's decision action.

[0011] S3. Calculate the individual basic reward of the agent based on the agent's local state set and the agent's decision action, calculate the system economic reward based on the individual basic reward, and construct the constraint benchmark reward in stages based on the system economic reward.

[0012] S4. Calculate the individual differentiated basic reward based on the constraint benchmark reward, and integrate the collaborative benefit share of each agent into the individual differentiated basic reward to obtain the agent's allocation reward;

[0013] S5. Input the agent's decision action into the physical quantity prediction network to obtain the physical quantity prediction value. Calculate the physical consistency reward based on the physical quantity prediction value and the physical quantity calculation value. Integrate the physical consistency reward into the allocation reward to obtain the final reward. Add the final rewards of all agents to obtain the global reward. Calculate the target reward based on the global reward.

[0014] S6. Concatenate the observation vectors of all agents to obtain the global joint observation vector. Concatenate the decision actions of all agents to obtain the joint action vector. Concatenate the global joint observation vector and the joint action vector and input them into the value network to obtain the global state value estimate. Based on the policy pruning loss, the value loss of the global state value estimate and the target reward, and the coupling regularization term, jointly train the policy network and the value network to obtain the trained policy network and value network.

[0015] S7. Input the local state set of the agent to be scheduled into the trained policy network, output the scheduling instruction, input the scheduling instruction into the trained value network, and output the scheduling evaluation result.

[0016] This invention constructs an adaptive coupling matrix that can dynamically sense and quantify energy interaction, power fluctuations, and spatiotemporal correlations. This adaptive coupling matrix can be dynamically adjusted according to real-time system operation data to capture the time-varying coupling strength between agents, thereby accurately characterizing the time-varying cooperative relationship between agents and overcoming the rigidity problem of fixed coupling relationships in existing methods. By introducing a fair allocation mechanism based on cooperative benefits, the additional benefits generated by global cooperation are reasonably fed back to each agent, thereby effectively incentivizing them to adopt globally optimal cooperative strategies. By incorporating the physical consistency index as an additional reward into the final reward function, the invention guides agents to generate decisions that conform to physical laws, thereby solving the problem of insufficient physical consistency in decision-making in existing methods.

[0017] The preferred method for constructing the coupling degree between agents based on energy interaction, power correlation, and spatiotemporal distance is as follows:

[0018]

[0019] in, Let be the coupling degree between agent i and agent j at time t. Let be the energy interaction power between agent i and agent j at time t. This represents the maximum value of the energy interaction power. For the power change of agent i With the power change of agent j covariance, , The power changes of agent i are respectively Power changes of agent j standard deviation Let i be the spatiotemporal distance between agent i and agent j. , , These are the weighting coefficients. This is the distance attenuation coefficient;

[0020] The coupling degree matrix is:

[0021]

[0022] in, This is the coupling degree matrix corresponding to time t. , For the number of agents, The degree of coupling at time t. For electrical connection adjacency matrix, Represents intelligent agents , An electrical connection exists; otherwise, the value is 0. For Hadama accumulation, This represents the normalization function.

[0023] Preferably, the process of obtaining the agent's decision action by combining the coupling degree matrix is ​​as follows:

[0024]

[0025] in, Let i be the decision action of agent i at time t. Let be the observation vector of agent i at time t. This indicates that the agent's observation vector Input Policy Network The basic decision to output This is the coupling effect coefficient. Coupling degree matrix At time t OK, Let i be the decision action of other agents besides agent i at time t-1. For mean averaging; the observation vector of agent i at time t. for:

[0026]

[0027] in, Let be the set of local states of agent i at time t. Let be the set of local states of all agents other than agent i at time t. This is a weighted pooling operation.

[0028] Preferably, the basic individual reward for an agent is:

[0029]

[0030] in, This represents the basic reward for agent i. Let i be the local reward function for agent i. Let i be the set of local states of agent i. For agent i, the decision-making action;

[0031] System economic rewards for:

[0032]

[0033] in, N represents the economic benefits generated by the interaction between the regional energy system and the upper-level power grid, and N is the number of intelligent agents.

[0034] Preferably, the process of constructing the constraint benchmark reward in stages based on the system economic reward is as follows: the entire training phase is divided into a first stage, a second stage, and a third stage according to the amount of data of the agent's observation vector and the agent's decision actions. In the first stage, a first global constraint benchmark reward is calculated based on the system economic reward and the equipment operation constraints, and the first global constraint benchmark reward is used as the constraint benchmark reward. In the second stage, a second global constraint benchmark reward is calculated based on the first global constraint benchmark reward and the local security constraints, and the second global constraint benchmark reward is used as the constraint benchmark reward. In the third stage, a third global constraint benchmark reward is calculated based on the second global constraint benchmark reward and the global constraints, and the third global constraint benchmark reward is used as the constraint benchmark reward.

[0035] This invention guides intelligent agents to master and satisfy various safety constraints from the device level to the system level in a phased and progressive manner by designing a constraint learning strategy that progresses from easy to difficult.

[0036] The preferred individualized basic reward is:

[0037]

[0038] in, As a basic reward for individual differences, To constrain the baseline reward, the proportion of the baseline reward of agent i to the total is... for:

[0039]

[0040] , Let i and j represent the basic individual rewards of agents i and j, respectively.

[0041] Agent reward allocation for:

[0042]

[0043] in, The number of agents; Let i be the share of the collaborative benefits of agent i at time t. To allocate benefit weights.

[0044] Preferred, physical consistency reward for:

[0045]

[0046] The final reward is:

[0047]

[0048] in, For physical quantity prediction networks, The input to the physical quantity prediction network, To enable intelligent agents to make decisions and actions Calculated values ​​of physical quantities obtained from inputting physical equations. Let i be the final reward for agent i at time t. Assign rewards to agent i at time t. Let i be the decision action of agent i at time t. Consistency reward weight.

[0049] Preferably, the global state value estimation and the value loss of the target return. for:

[0050]

[0051] in, This is the time series expectation, i.e., the average over all times t. For global state value estimation, target return for:

[0052]

[0053] in, The global reward for time t. It is a discount factor. This is the target value network, and its parameters are updated following the overall value network. Let be the agent's observation vector at time t+1. Represents the observation vector based on time t+1 The estimated global state value.

[0054] This invention improves the value network's ability to assess the value of the global state by minimizing the mean square error between the value network's predicted value and the target return. By accurately assessing the long-term value of the state, the value network can guide the policy network to update towards a better decision direction.

[0055] Preferably, the coupling regularization term for:

[0056]

[0057] in, Let be the mean action of agent i output by the policy network. M is the number of samples. Adaptive Coupling Degree Matrix The i-th row vector, This represents the mean vector of actions of all agents except agent i. This represents the output of a weighted average strategy based on coupling degree.

[0058] This invention introduces a coupling regularization term to train the network. The coupling regularization term is used to guide the cooperation between agents. In a regional energy system, there are complex coupling relationships between various distributed resources. The coupling regularization term measures the difference between the policy output of each agent and the weighted average of the policy outputs of its neighbors through an adaptive coupling degree matrix, and minimizes this difference. This makes the agents consider not only their own rewards when updating their policies, but also the consistency of cooperation with their neighbors. This helps to achieve global cooperative scheduling, avoid local optima, and improve the economy and security of the overall system.

[0059] This invention also provides a multi-agent reinforcement learning regional energy cooperative scheduling system, comprising:

[0060] The adaptive coupling degree matrix construction module is used to obtain the local state set of all agents, construct the coupling degree between agents based on energy interaction, power correlation, and spatiotemporal distance, and construct the coupling degree matrix based on the coupling degree.

[0061] The decision-making action calculation module is used to construct the agent's observation vector based on the agent's local state set and coupling degree matrix, input the agent's observation vector into the policy network, and obtain the agent's decision-making action by combining it with the coupling degree matrix.

[0062] The constrained baseline reward phased construction module is used to calculate the individual basic reward of the agent based on the agent's local state set and the agent's decision actions, calculate the system economic reward based on the individual basic reward, and construct the constrained baseline reward in phases based on the system economic reward.

[0063] The reward allocation calculation module is used to calculate the individual differentiated basic reward based on the constraint benchmark reward, and integrate the collaborative benefit share of each agent into the individual differentiated basic reward to obtain the agent's allocation reward;

[0064] The target reward calculation module is used to input the agent's decision action into the physical quantity prediction network to obtain the physical quantity prediction value. Based on the physical quantity prediction value and the physical quantity calculation value, the physical consistency reward is calculated. The physical consistency reward is integrated into the allocation reward to obtain the final reward. The final rewards of all agents are added together to obtain the global reward. The target reward is calculated based on the global reward.

[0065] The learning module is used to concatenate the observation vectors of all agents to obtain the global joint observation vector, and to concatenate the decision actions of all agents to obtain the joint action vector. The global joint observation vector and the joint action vector are concatenated and input into the value network to obtain the global state value estimate. Based on the policy pruning loss, the value loss of the global state value estimate and the target reward, and the coupling regularization term, the policy network and the value network are jointly trained to obtain the trained policy network and value network.

[0066] The inference module is used to input the local state set of the agent to be scheduled into the trained policy network, output scheduling instructions, input the scheduling instructions into the trained value network, and output the scheduling evaluation results. Attached Figure Description

[0067] Figure 1 The flowchart is a multi-agent reinforcement learning regional energy collaborative scheduling method provided in Embodiment 1 of the present invention;

[0068] Figure 2 This is a schematic diagram of the multi-agent reinforcement learning regional energy collaborative scheduling method provided in Embodiment 1 of the present invention;

[0069] Figure 3 This is a flowchart illustrating the construction of the coupling degree matrix in the multi-agent reinforcement learning regional energy collaborative scheduling method provided in Embodiment 1 of the present invention;

[0070] Figure 4 This is a flowchart illustrating the calculation of agent decision-making actions in the multi-agent reinforcement learning regional energy collaborative scheduling method provided in Embodiment 1 of the present invention.

[0071] Figure 5 This is a flowchart illustrating the calculation of global reward in the multi-agent reinforcement learning regional energy collaborative scheduling method provided in Embodiment 1 of the present invention;

[0072] Figure 6 This is a schematic diagram of a multi-agent reinforcement learning regional energy collaborative scheduling system provided in Embodiment 2 of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0074] Example 1

[0075] like Figure 1 and Figure 2 As shown, this embodiment provides a multi-agent reinforcement learning regional energy cooperative scheduling method, including the following steps:

[0076] Step 1: First, conduct full-dimensional data collection on the diverse resources and operational status of the regional energy system to obtain local status data of all intelligent entities, including distributed photovoltaic units, energy storage systems, loads, and grid-side entities. Local status data includes output data of distributed photovoltaic units, SOC and charging / discharging power of energy storage systems, demand curves of adjustable loads and base loads on the user side, and operational parameters such as node voltage, line transmission power, and geographical and electrical distances between resources on the grid side, forming a complete dataset covering "source-storage-load-grid".

[0077] The dataset undergoes preprocessing, including outlier cleaning in the first stage using physical logic verification and the 3σ criterion to remove invalid data. The second stage involves missing value imputation. For missing data in a single time period, linear interpolation across four adjacent time periods is used. For missing data across multiple consecutive time periods, an LSTM time-series prediction model is invoked to generate imputed values ​​based on historical data of similar operating conditions. The preprocessed dataset then becomes the local state set for all agents. In practice, the preprocessed dataset can be divided into training and validation sets according to a predetermined ratio, such as 7:3. The training set is used for iterative learning of agent policies, while the validation set is used to evaluate the model's generalization ability in real time, laying a solid data foundation for subsequent multi-agent decision-making processes.

[0078] See Figure 3 After obtaining the local state set of all agents, the coupling degree between agents is constructed based on energy interaction, power correlation, and spatiotemporal distance. Based on coupling degree Constructing the coupling matrix The coupling degree between agent i and agent j at time t. The calculation formula is:

[0079]

[0080] in, Let be the energy interaction power between agent i and agent j at time t. This represents the maximum value of the energy interaction power. For the power change of agent i With the power change of agent j covariance, , The power changes of agent i are respectively Power changes of agent j standard deviation Let be the spatiotemporal distance between agent i and agent j. It combines electrical distance and geographic distance, with the weights for electrical distance and geographic distance set to 0.6 and 0.4, respectively. , , All are weighting coefficients, satisfying , This is the distance attenuation coefficient.

[0081] Based on coupling degree Constructing the coupling matrix Coupling matrix The calculation formula is:

[0082]

[0083] in, This is the coupling degree matrix corresponding to time t. , For the number of agents, The degree of coupling at time t. For electrical connection adjacency matrix, Represents intelligent agents , An electrical connection exists; otherwise, the value is 0. For Hadama accumulation, This represents the normalization function, used for matrix normalization to ensure that the sum of the elements in each row is 1, thus guaranteeing numerical stability.

[0084] The adaptive coupling matrix can be dynamically adjusted according to real-time system operation data, capturing the time-varying coupling strength between agents and overcoming the rigidity problem of fixed coupling relationships in traditional methods.

[0085] Step 2, see Figure 4 Based on the local state set of the agent and coupling matrix Construct intelligent agent observation vectors , to transform the agent's observation vector Input Policy Network Combined with the coupling matrix Obtain the agent's decision action It should be noted that in this invention, all agents share a set of policy network parameters, adopting a centralized training and distributed execution architecture. Each agent is a distributed execution branch of the policy network, ensuring the synergy of multi-agent policies and training efficiency. "The observation vector of agent i at time t..." for:

[0086]

[0087] in, Let be the set of local states of agent i at time t. Let be the set of local states of all agents other than agent i at time t. This is a weighted pooling operation.

[0088] Combining the coupling matrix Obtain the agent's decision action The process is as follows:

[0089]

[0090] in, Let i be the decision action of agent i at time t. Let be the observation vector of agent i at time t. This indicates that the agent's observation vector Input Policy Network The basic decision to output This is the coupling effect coefficient. Coupling degree matrix At time t OK, Let i be the decision action of other agents besides agent i at time t-1. This is a mean-averaging operation used to balance the correlation and autonomy of multi-agent decision-making.

[0091] Step 3, see Figure 5 Based on the agent's local state set and agent decision-making actions Calculate the individual basic reward of the agent. Individual basic rewards are the most basic unit of rewards, which are based entirely on the agent's own state and actions, without considering system-level constraints and cooperation.

[0092] Based on individual basic reward Calculate the system's economic reward And based on system economic rewards Phased construction of constraint benchmark rewards .

[0093] Individual basic reward of the agent for:

[0094]

[0095] in, Let be the local reward function of agent i. The local reward function is calculated by subtracting the cost of the agent from the agent's own revenue. Let i be the set of local states of agent i. For agent i, the decision-making action;

[0096] The local reward function for distributed photovoltaic intelligent agents is:

[0097]

[0098] in, The electricity price for time period t. It is the unit output operation and maintenance cost of the photovoltaic unit. Scheduling time interval.

[0099] The local reward function for the energy storage system agent is:

[0100]

[0101] in, It refers to the decision-making action of agent i, that is, the charging and discharging power command of the energy storage unit in time period t. It is the absolute value of the charging / discharging power. The electricity purchase price for time period t. The electricity price for time period t. It is the converted loss cost per unit power charge and discharge of energy storage. It is the operation and maintenance cost of charging and discharging energy per unit power.

[0102] The local reward function for an adjustable load agent is:

[0103]

[0104] in, It is the decision-making action of agent i, that is, the power adjustment command of the adjustable load in time period t. It is the absolute value of the load adjustment. It is a demand response subsidy per unit power. The discount is the cost / comfort loss caused by the unit power adjustment of the load. The electricity purchase price for time period t. It takes a non-negative value of the adjustment amount and only takes effect when electricity consumption increases.

[0105] The sum of the individual basic rewards of all agents, plus the interaction benefits between the regional energy system and the upper-level power grid, yields the system-level pure economic benefit, i.e., the system economic reward. This serves as the benchmark for subsequent adjustments to constraints and penalties. System economic rewards. for:

[0106]

[0107] in, N represents the economic benefits generated by the decision-making actions of the regional energy system and the upper-level grid intelligent agent i, i.e., the interaction of charging and discharging power commands of the energy storage unit in time period t. N is the number of intelligent agents.

[0108] The process of constructing a constraint benchmark reward based on system economic rewards in stages is as follows: The entire training phase is divided into three stages based on the amount of data from the agent's observation vectors and decision actions. In the first stage, the first global constraint benchmark reward is calculated based on the system economic rewards and equipment operation constraints. and the first global constraint benchmark reward As a constraint benchmark reward; in the second phase, based on the first global constraint benchmark reward. The second global constraint baseline reward is calculated based on the local security constraints. and the second global constraint benchmark reward As a constraint benchmark reward; in the third phase, based on the second global constraint benchmark reward. The third global constraint baseline reward is calculated based on the global constraints. And the third global constraint benchmark reward Rewards serve as a constraint benchmark.

[0109] In this embodiment, the data volume of the agent's observation vectors and decision actions is divided into three equal parts. The data volume corresponding to the first stage of training is 0-30% of the total data volume, the data volume corresponding to the second stage of training is 31%-70% of the total data volume, and the data volume corresponding to the third stage of training is 71%-100% of the total data volume. In the first stage, only equipment operation constraints (energy storage charging and discharging power limit, photovoltaic output limit) are introduced, based on the system's economic reward. and equipment operation constraints Calculate the first global constraint baseline reward :

[0110]

[0111] in, This represents the global weighting coefficient for the first stage; in this embodiment, the global weighting coefficient for the first stage is 0.1. Equipment operation constraints. for:

[0112]

[0113] in, Let be the device penalty coefficient for the i-th agent. For the actual operating parameters of the device constraints of the i-th intelligent agent, Let max(0, ...) be the device operation threshold for the i-th agent. ) is the truncation function.

[0114] In the second phase, rewards are based on the first global constraint benchmark. and local security constraints Calculate the second global constraint benchmark reward :

[0115]

[0116] in, This is the global weighting coefficient for the second stage; in this embodiment, the global weighting coefficient for the second stage is 0.8. Local security constraints. for:

[0117]

[0118] in, Let be the penalty coefficient for the m-th local subsystem. These are the actual operating parameters of the m-th local subsystem. Let be the security threshold for the m-th local subsystem.

[0119] In the third phase, rewards are based on the second global constraint benchmark. and global constraints Calculate the third global constraint benchmark reward :

[0120]

[0121] in, This refers to the global weight coefficient for the third stage, which in this embodiment increases linearly from 0.8 to 1. Global Constraints for:

[0122]

[0123] in, Let be the penalty coefficient for the p-th type of global constraint. For the actual running parameters of the p-th type of global constraint, is the safety threshold for the p-th type of global constraint.

[0124] In multi-agent reinforcement learning, the constraint benchmark reward is a reward function that integrates the economic goals and safety constraints of the entire system and is jointly determined by the actions of all agents. Its core role is to guide the training to converge to a strategy that is "globally optimal and compliant with constraints", thus avoiding local optima and safety risks.

[0125] Step 4: Reward based on constraint benchmarks Calculate individual-differentiated basic rewards The share of collaborative benefits for each agent Incorporating individualized basic rewards Receive the reward allocated by the intelligent agent .

[0126] The individualized basic reward is as follows:

[0127]

[0128] in, As a basic reward for individual differences, To constrain the baseline reward, the proportion of the baseline reward of agent i to the total is... for:

[0129]

[0130] , Let i and j represent the basic individual rewards of agents i and j, respectively.

[0131] Agent reward allocation for:

[0132]

[0133] in, The number of agents; Let i be the share of the collaborative benefits of agent i at time t. To allocate benefit weights.

[0134] This invention uses the Shapley value method to calculate the intelligent agent. The allocation of collaborative benefits ensures fair distribution based on contribution, with agent i's share of collaborative benefits at time t. The calculation formula is:

[0135]

[0136] in, For those that do not include intelligent agents Any subset of the alliance, For the alliance Synergistic benefits For the alliance The number of intelligent agents, The total number of agents. The core advantage of the Shapley value method is "fair distribution based on contribution". It calculates the share of collaborative benefits that each agent should receive by quantifying the marginal contribution of each agent to all possible alliances. It is a classic fairness method for solving the problem of distributing benefits in multi-agent cooperation.

[0137] This invention relates to the allocation of rewards by computational agents. At the same time, it balances individual and collaborative benefits, allowing the agent to consider its own interests while actively pursuing the maximization of global collaborative benefits during the reinforcement learning process. The allocated amount... Incorporating a reward function is a key bridge to transform global collaborative benefits into local reward signals for the agent, directly driving the reinforcement learning strategy to converge toward the global optimum.

[0138] Step 5: Input the agent's decision action into the physical quantity prediction network to obtain the physical quantity prediction value. Calculate the physical consistency reward based on the physical quantity prediction value and the physical quantity calculation value. Integrate the physical consistency reward into the allocation reward to obtain the final reward. Add the final rewards of all agents to obtain the global reward. Calculate the target reward based on the global reward.

[0139] Physical consistency requires models to adhere to fundamental physical rules such as conservation laws, thermodynamic laws, and electromagnetic laws. For example, in fluid simulations, neglecting mass conservation might lead to absurd results such as fluids appearing or disappearing out of thin air; by incorporating physical consistency constraints, experimental results will closely match real physical phenomena. Physical consistency metrics are integrated as an additional reward into the final reward function, guiding the agent to generate decisions that conform to physical laws. First, a key physical equation model is established, and a physical consistency metric is defined to measure the degree to which decisions conform to physical laws.

[0140] Physical consistency reward for:

[0141]

[0142] in, Let be a physical quantity prediction network, and let x represent the input set of the physical quantity prediction network. When the physical quantity to be predicted is node power, the input set x of the physical quantity prediction network includes the scheduling decision vectors of all agents, the node-branch correlation matrix, and the node reference voltage. and branch impedance parameters When the physical quantity to be predicted is the node voltage amplitude, the input set x of the physical quantity prediction network includes the scheduling decision vectors of all agents, the node-branch admittance matrix, and the node reference voltage. and system reference power When the physical quantity to be predicted is the active power transmitted by the line, the input set x of the physical quantity prediction network includes the scheduling decision vectors of all agents, the node-branch correlation matrix, and the branch impedance parameters. And node voltage amplitude. In this embodiment, the physical quantity prediction network uses an MLP network, which is an existing network. To enable intelligent agents to make decisions and actions The calculated values ​​of physical quantities are obtained by inputting the physical equations. Taking nodal power as an example, the calculated values ​​of the physical quantities are... The calculation formula is:

[0143]

[0144] in, It is the net active power injected into node m in the regional energy system at time step t. It is the total decision output of all photovoltaics at node m. The total decision power for all energy storage at node m. The total decision power of all adjustable loads on node m; these three parameters are the core parameters of the scheduling action instructions output by the agent. Let m be the fixed load power of node m.

[0145] Physical consistency index This is used to quantify the discrepancy between neural network predictions and physical equation calculations in intelligent agent decision-making. This discrepancy represents the difference between the physical results predicted by the model and the theoretical results determined by physical laws. The core principle is to measure this error using the L2 norm. The closer to 1, the better the consistency. If If it is too small, it indicates that the physical quantity predicted by the neural network based on the agent's decision 'a' is too small. Compared with the benchmark physical quantities calculated strictly according to the laws of physics If the deviation is extremely large, then the decision will have a serious physical and logical conflict and will not satisfy the underlying physical laws.

[0146] The final reward is:

[0147]

[0148] in, Let i be the final reward for agent i at time t. Assign rewards to agent i at time t. Let i be the decision action of agent i at time t. The consistency reward weight is 0.2 in this embodiment.

[0149] Calculate the target return based on the global reward. The method is as follows:

[0150]

[0151] in, The global reward for time t. It is a discount factor. For the target value network, This represents the agent's observation vector at time t+1. The target value network provides a relatively stable and slowly changing benchmark for calculating the training target (TD target), thus avoiding learning instability caused by drastic fluctuations in the target value during training. The input to the target value network is the global joint observation vector for the next time step. The output of the target value network is the global joint observation vector based on time t+1. The global state value estimate. Parameters of the target value network. _target updates with the value network; it represents the parameters of the target value network. The update method for _target is as follows: after each training step, the parameters of the value network (training network) are updated. With the current parameters of the target value network The target is a weighted average of the training network parameters. The weight is Target network parameters The weight of _target is That is, through the formula _target← + _target updates the target network parameters, which is about to + The target value network is fed as a new parameter. This approach allows the target network parameters to slowly and smoothly follow changes in the training network parameters, avoiding drastic changes in the target value due to frequent fluctuations in the training network parameters, thus ensuring the stability of the entire training process. The target value network has the exact same network structure as the value network, and its parameters are completely homologous. The target value network is a stable copy of the value network, does not participate in active learning, and only slowly follows the updates of the value network.

[0152] Step 6: At the current decision-making moment, the observation vectors of all agents are concatenated in a preset order to form a global joint observation. Simultaneously, the action instructions output by each agent based on the current policy network are concatenated into a joint action vector. Then the global joint observation vector will be... With joint action vector The features are concatenated along the feature dimension to form the input feature vector of the centralized value network. , input feature vector Input Value Network , The value network can be trained by using the forward propagation to obtain the global state value estimate. The global state value estimate represents the expected cumulative discounted return that the system can obtain in the future after performing joint actions under joint observation. It is the core indicator for measuring the long-term performance of multi-agent cooperative strategies.

[0153] Based on strategy-based pruning loss Global state value estimation and the value loss of target return Coupling regularization term The policy network and value network are trained together to obtain the trained policy network and value network. Joint training refers to training the policy network and value network based on a single total loss function, which is the policy pruning loss. Value loss Coupling regularization term The weighted sum, expressed as the total loss function using a formula. for:

[0154]

[0155] in, , All are weights.

[0156] Total loss function The sum of policy loss, value loss, and coupling regularization term balances policy optimization, value estimation accuracy, and multi-agent collaborative consistency. Optimal decisions are made by optimizing and updating the agent policy network parameters through gradient descent, thereby improving economy, security, and physical consistency. In this invention, both the policy network and the value network adopt existing network architectures. For example, the policy network uses a multilayer perceptron with two fully connected neural network layers. The input dimension is the agent's observation vector dimension, with 128 neurons in the first layer and 64 in the second. The centralized value network also uses a multilayer perceptron with two fully connected neural network layers. The input dimension is the concatenation dimension of the global joint observation and joint action vectors, with 128 neurons in the first layer and 64 in the second.

[0157] Strategy pruning loss The calculation formula is:

[0158]

[0159] in, The probability ratio of the strategies. The parameter is The current policy network, observed in the environment at time step t. Next, output and select the decision action. The probability of. The parameter is The old policy network, observed in the environment at time step t. Next, output and select the decision action. The probability of. As a pruning parameter, in this embodiment the pruning parameter is set to 0.2 to limit the range of change of the probability ratio and ensure the stability of the strategy update. This is the time series expectation, which is the average over all time steps t. It is a truncation function whose purpose is to truncate the data. Limited to Within the range; Probability Ratio Truncate the segment. If Cut to ;like Cut to .like ,Keep Unchanged. By truncation, the policy update steps are prevented from being too large, thus stabilizing the training process. The generalized advantage estimate (GAE) is calculated using the following formula:

[0160] ,

[0161] is the GAE weighting coefficient, and k is the future step size index. , The global reward for time t. It is a return discount factor. , It is an observation of the output of the value network. The corresponding state value, It is the observation of the target value network output. The corresponding state value. Single-step TD error at time step t+k. TD error of generalized advantage estimation (GAE) In the calculation, the value of the next state Adopting a target value network The output, and the value loss. Mid-target return The next state value source remains consistent, ensuring that policy updates and value updates are based on the same value benchmark, thus avoiding conflicts in training logic.

[0162] Strategy pruning loss The policy network is used to update the policy algorithm, aiming to maximize expected reward while ensuring policy update stability. By pruning the probability ratio, excessively large policy update steps are avoided, thus stabilizing the training process. In regional energy scheduling, the policy network is responsible for generating scheduling actions based on current observations. This is achieved by optimizing the policy pruning loss. The policy network gradually learns scheduling policies that can yield higher returns.

[0163] Global state value estimation and the value loss of the target return for:

[0164]

[0165] in, This is the time series expectation, i.e., the average over all times t. For global state value estimation, To achieve the desired return.

[0166] Value loss Loss is achieved by minimizing the value network prediction. With target return The mean squared error between states is used to improve the value network's ability to assess the value of the global state. By accurately evaluating the long-term value of states, the value network can guide the policy network to update towards better decision-making directions.

[0167] Coupling regularization terms for:

[0168]

[0169] in, Let be the mean action of agent i output by the policy network. M is the number of samples. Adaptive Coupling Degree Matrix The i-th row vector, This represents the mean vector of actions of all agents except agent i. This represents the output of a weighted average strategy based on coupling degree.

[0170] Coupling regularization terms This regularization term guides cooperation among agents. In regional energy systems, complex coupling relationships exist between various distributed resources. It uses an adaptive coupling matrix to measure and minimize the difference between the policy output of each agent and the weighted average of the policy outputs of its neighbors. This ensures that agents consider not only their own rewards but also the consistency with their neighbors when updating their policies, facilitating global coordinated scheduling, avoiding local optima, and improving the overall system's economy and security.

[0171] Step 7: Input the local state set of the agent to be scheduled into the trained policy network, output the scheduling instruction, input the scheduling instruction into the trained value network, and output the scheduling evaluation result.

[0172] Based on the optimized scheduling strategy, targeted control commands are first output to achieve dynamic optimization of photovoltaic output, charging and discharging regulation of energy storage units, flexible response of adjustable loads, and power purchase and sale interaction with the upstream power grid. Then, the scheduling effect is evaluated from multiple dimensions: economic efficiency is measured by the average daily scheduling cost, safety is assessed by the percentage of time periods when the node voltage is within the acceptable range, and robustness is verified by the system recovery speed and constraint violation under extreme conditions. At the same time, the PyTorch framework is used to support model training, and the MQTT protocol is used to ensure reliable data transmission, so as to ensure the efficiency and stability of the entire scheduling process.

[0173] This invention constructs an adaptive coupling degree matrix that can dynamically sense and quantify energy interaction, power fluctuations, and spatiotemporal correlations to accurately characterize the time-varying collaborative relationships among agents. By introducing a fair distribution mechanism for collaborative benefits based on Shapley values, the additional benefits generated by global cooperation are reasonably fed back to each agent, effectively incentivizing them to adopt globally optimal collaborative strategies. Through a phased, progressive constraint learning strategy, agents are guided to master and satisfy various safety constraints from the device level to the system level efficiently and reliably, progressing from easy to difficult. Physical consistency indicators are incorporated as additional rewards into the final reward function, guiding agents to generate decisions that conform to physical laws. The centralized training process and loss function design of the multi-agent system are optimized to improve the collaborative adaptability and training stability of the policy network, enabling agents to make optimal scheduling decisions based on the current situation, thereby comprehensively improving the economy, security, and physical consistency of regional energy system scheduling.

[0174] Example 2

[0175] See Figure 6 This embodiment provides a multi-agent reinforcement learning regional energy cooperative scheduling system, including:

[0176] The adaptive coupling degree matrix construction module is used to obtain the local state set of all agents, construct the coupling degree between agents based on energy interaction, power correlation, and spatiotemporal distance, and construct the coupling degree matrix based on the coupling degree.

[0177] The method for constructing the coupling degree between intelligent agents based on energy interaction, power correlation, and spatiotemporal distance is as follows:

[0178]

[0179] in, Let be the coupling degree between agent i and agent j at time t. Let be the energy interaction power between agent i and agent j at time t. This represents the maximum value of the energy interaction power. For the power change of agent i With the power change of agent j covariance, , The power changes of agent i are respectively Power changes of agent j standard deviation Let i be the spatiotemporal distance between agent i and agent j. , , These are the weighting coefficients. This is the distance attenuation coefficient;

[0180] The coupling degree matrix is:

[0181]

[0182] in, This is the coupling degree matrix corresponding to time t. , For the number of agents, The degree of coupling at time t. For electrical connection adjacency matrix, Represents intelligent agents , An electrical connection exists; otherwise, the value is 0. For Hadama accumulation, This represents the normalization function.

[0183] The decision-making action calculation module is used to construct the agent's observation vector based on the agent's local state set and coupling degree matrix, input the agent's observation vector into the policy network, and obtain the agent's decision-making action by combining it with the coupling degree matrix.

[0184] The process of obtaining the agent's decision-making action by combining the coupling degree matrix is ​​as follows:

[0185]

[0186] in, Let i be the decision action of agent i at time t. Let be the observation vector of agent i at time t. This indicates that the agent's observation vector Input Policy Network The basic decision to output This is the coupling effect coefficient. Coupling degree matrix At time t OK, Let i be the decision action of other agents besides agent i at time t-1. For mean averaging; the observation vector of agent i at time t. for:

[0187]

[0188] in, Let be the set of local states of agent i at time t. Let be the set of local states of all agents other than agent i at time t. This is a weighted pooling operation.

[0189] The constraint benchmark reward phased construction module is used to calculate the individual basic reward of the agent based on the agent's local state set and the agent's decision actions, calculate the system economic reward based on the individual basic reward, and construct the constraint benchmark reward in phases based on the system economic reward.

[0190] The basic individual reward for an agent is:

[0191]

[0192] in, This represents the basic reward for agent i. Let i be the local reward function for agent i. Let i be the set of local states of agent i. For agent i, the decision-making action;

[0193] System economic rewards for:

[0194]

[0195] in, N represents the economic benefits generated by the interaction between the regional energy system and the upper-level power grid, and N is the number of intelligent agents.

[0196] The process of constructing a constraint benchmark reward in stages based on system economic rewards is as follows: The entire training phase is divided into three stages based on the amount of data of agent observation vectors and agent decision actions. In the first stage, a first global constraint benchmark reward is calculated based on system economic rewards and equipment operation constraints, and this first global constraint benchmark reward is used as the constraint benchmark reward. In the second stage, a second global constraint benchmark reward is calculated based on the first global constraint benchmark reward and local security constraints, and this second global constraint benchmark reward is used as the constraint benchmark reward. In the third stage, a third global constraint benchmark reward is calculated based on the second global constraint benchmark reward and global constraints, and this third global constraint benchmark reward is used as the constraint benchmark reward.

[0197] The reward allocation calculation module is used to calculate the individual differentiated basic reward based on the constraint benchmark reward, and integrate the collaborative benefit share of each agent into the individual differentiated basic reward to obtain the agent's allocated reward.

[0198] Individualized basic rewards are:

[0199]

[0200] in, As a basic reward for individual differences, To constrain the baseline reward, the proportion of the baseline reward of agent i to the total is... for:

[0201]

[0202] , Let i and j represent the basic individual rewards of agents i and j, respectively.

[0203] Agent reward allocation for:

[0204]

[0205] in, The number of agents; Let i be the share of the collaborative benefits of agent i at time t. To allocate benefit weights.

[0206] The target reward calculation module is used to input the agent's decision actions into the physical quantity prediction network to obtain the physical quantity prediction value. Based on the physical quantity prediction value and the physical quantity calculation value, the physical consistency reward is calculated. The physical consistency reward is integrated into the allocation reward to obtain the final reward. The final rewards of all agents are added together to obtain the global reward. The target reward is calculated based on the global reward.

[0207] Physical consistency reward for:

[0208]

[0209] The final reward is:

[0210]

[0211] in, For physical quantity prediction networks, The input to the physical quantity prediction network, To enable intelligent agents to make decisions and actions Calculated values ​​of physical quantities obtained from inputting physical equations. Let i be the final reward for agent i at time t. Assign rewards to agent i at time t. Let i be the decision action of agent i at time t. Consistency reward weight.

[0212] Global state value estimation and the value loss of the target return for:

[0213]

[0214] in, This is the time series expectation, i.e., the average over all times t. For global state value estimation, target return for:

[0215]

[0216] in, The global reward for time t. It is a discount factor. This is the target value network, and its parameters are updated following the overall value network. Let be the agent's observation vector at time t+1. Represents the observation vector based on time t+1 The estimated global state value.

[0217] The learning module is used to concatenate the observation vectors of all agents to obtain the global joint observation vector, and to concatenate the decision actions of all agents to obtain the joint action vector. The global joint observation vector and the joint action vector are then concatenated and input into the value network to obtain the global state value estimate. Based on the policy pruning loss, the value loss of the global state value estimate and the target reward, and the coupling regularization term, the policy network and the value network are jointly trained to obtain the trained policy network and value network.

[0218] Coupling regularization terms for:

[0219]

[0220] in, Let be the mean action of agent i output by the policy network. M is the number of samples. Adaptive Coupling Degree Matrix The i-th row vector, This represents the mean vector of actions of all agents except agent i. This represents the output of a weighted average strategy based on coupling degree.

[0221] The inference module is used to input the local state set of the agent to be scheduled into the trained policy network, output scheduling instructions, input the scheduling instructions into the trained value network, and output the scheduling evaluation results.

[0222] Example 3

[0223] This embodiment uses the IEEE 33-node regional energy system with distributed energy as the simulation object to introduce the multi-agent reinforcement learning regional energy cooperative scheduling method of Embodiment 1. The IEEE 33-node regional energy system has 10 photovoltaic units with a single unit capacity of 500kW and a total installed capacity of 5MW, and 5 energy storage units with a single unit capacity of 400kWh / 200kW and a total capacity of 2MWh / 1MW. The initial SOC value is 50%, and the charge and discharge efficiency is 92%. It has 20 adjustable load units, each with a capacity of 400kW, and a total capacity of 8MW. The demand data uses actual measured values ​​from the Belgian power grid. The dispatch cycle and step size are 24 hours / dispatch day, with a time step of 15 minutes and 96 dispatch periods per day. The economic parameters are peak-valley time-of-use electricity prices: RMB 1.2 / kWh during peak hours (08:00-22:00) and RMB 0.5 / kWh during off-peak hours (22:00-08:00 the next day). The grid purchase default cost is RMB 2 / kWh. In terms of safety constraints, the node voltage deviation is ±5%, the photovoltaic output fluctuation is ≤20% / 15 minutes, and the energy storage SOC operating range is 20%-80%. It uses an Intel Xeon Gold 6330 processor, 64GB DDR4 memory, and an NVIDIA A10 GPU; deep learning training uses PyTorch 2.0, power flow calculation uses MatPower 7.1, and data transmission uses MQTT 3.1.1; the network parameters are a data sampling frequency of 1Hz, a modified MQTT-SN transmission protocol, and an edge-to-cloud data interaction latency of ≤20ms.

[0224] Six months of measured photovoltaic (PV) output and load demand data were collected from the Belgian power grid. After removing outliers, the data was divided into a training set of 24,528 records and a validation set of 10,512 records, in a 7:3 ratio. The data cleaning phase focused on removing three types of outliers: first, physical and logical outliers, where PV output at night (7:00 PM - 6:00 AM the next day) was >0 and load demand was <0; second, fluctuation outliers, where data showing PV output fluctuations >50% within 15 minutes and load demand fluctuations >30% within 15 minutes were removed using the 3σ criterion; and third, missing value completion, where missing data after cleaning were completed using linear interpolation, ultimately yielding 34,862 valid data records.

[0225] Rounds 0-6000 (30% of total rounds): Reward function used Only ensure equipment safety; save model parameters every 100 rounds; from rounds 6001 to 14000 (31% to 70% of total rounds): switch to reward function. Increase local voltage constraints. Linearly increase from 0.1 to 0.8; Rounds 14001-20000 (71%-100% of total rounds): Switch to reward function A global power balance constraint is applied, and the validation set performance is evaluated every 50 rounds. Training stops if there is no performance improvement for three consecutive rounds. After each round, the improved MAPPO total loss is calculated, and the policy network parameters are updated using gradient descent until the validation set MAPE is ≤3.5% for five consecutive rounds, at which point the model converges. Gradient descent optimization uses the Adam optimizer to update the policy network parameters, and the learning rate decays according to the CosineAnnealing policy. The initial learning rate of the policy network is set. Value network learning rate The weight decay coefficient is 0.01.

[0226] 1. Routine performance testing consists of the following three steps:

[0227] (1) Test data preparation: Select 2,880 time periods of actual test data for one month that do not overlap with the training to simulate the actual operation scenario.

[0228] (2) Scheduling execution: The proposed method was deployed in comparison with three other methods: traditional centralized particle swarm optimization (PSO) scheduling method, single agent proximal policy optimization (PPO) reinforcement learning scheduling method, and traditional multi-agent proximal policy optimization (MAPPO) reinforcement learning scheduling method. Scheduling instructions were output in real time for photovoltaic power output control, energy storage charging and discharging power and load regulation. Power flow distribution was calculated using MatPower to verify the satisfaction of constraints such as node voltage and power balance.

[0229] (3) Indicator statistics: Daily average scheduling cost, photovoltaic absorption rate and voltage qualification rate are calculated. The number of training convergence rounds is calculated every 5 days. Finally, the average value of 30 days is taken as the conventional performance indicator.

[0230] In terms of economic advantages, the daily dispatch cost of this invention is reduced by RMB 0.19 million compared to the traditional MAPPO, a decrease of 4.5%. This is mainly because the adaptive coupling matrix optimizes the cooperative relationship between photovoltaics, energy storage, and load, reducing peak-hour high-price electricity purchases and redundant energy storage adjustments. Regarding the improvement of renewable energy absorption, the photovoltaic absorption rate reaches 97.3%, an increase of 1.2 percentage points compared to the traditional MAPPO. This is due to the Shapley value allocation mechanism incentivizing photovoltaic intelligent agents to prioritize power output, while the energy storage intelligent agents actively charge when photovoltaic output is excessive. In terms of security... The voltage compliance rate was 99.8%, with only 0.2% of the time periods experiencing slight voltage exceedances, significantly better than the 0.8% exceedance rate of traditional MAPPO, demonstrating the precise control of node voltages by progressive constraint learning; the convergence rounds were only 2500, 1300 fewer than the 3800 rounds of traditional MAPPO, due to the reduction of policy update oscillations caused by the coupling regularization term, and the centralized value network improved the accuracy of global state evaluation; the 60-day accuracy decay rate was only 3.5%, far lower than the 8.2% of traditional MAPPO, due to the lifelong learning mechanism enabling the model to continuously adapt to environmental changes.

[0231] 2. Extreme operating condition verification

[0232] The robustness of the testing method was assessed by selecting five common extreme operating conditions in regional energy systems:

[0233] (1) Condition 1 is a sudden drop in photovoltaic output: cloud cover caused the total photovoltaic output to drop from 4.2MW to 1.8MW, lasting for 5 minutes;

[0234] (2) Condition 2 is an abnormal SOC of energy storage: the SOC of a certain energy storage unit drops sharply from 65% to 20% for 10 minutes;

[0235] (3) Condition 3 is a sudden load increase: the industrial load suddenly increases by 3MW and lasts for 15 minutes;

[0236] (4) Operating condition 4 is grid voltage fluctuation: the upstream grid voltage drops from 10kV to 9.2kV (deviation 8%), lasting for 20 minutes;

[0237] (5) The working condition is 5 multiple fault superposition: the photovoltaic output drops suddenly and the load increases suddenly at the same time, lasting for 8 minutes; for each working condition, record the response delay, fault identification accuracy and constraint violation number of the method to evaluate the adaptability under extreme scenarios.

[0238] In three extreme operating conditions—sudden drop in photovoltaic output, abnormal energy storage SOC, and multiple fault superposition—this invention demonstrates superior response speed, collaborative control capabilities, and risk avoidance effects. Compared to traditional MAPPO, it not only achieves a 48.6%-58.6% reduction in response delay and a 58.6% acceleration in system recovery time, but also avoids voltage overruns, energy storage capacity degradation, and high constraint violation costs, fully verifying its robustness and safety in extreme scenarios.

[0239] Photovoltaic output drop conditions: This invention identifies the output drop within 120ms, triggering emergency discharge of 2 energy storage units and temporary reduction of 3 adjustable loads, quickly filling the 0.7MW power deficit, and keeping the voltage stable within the range of 9.6-10.4kV; while the traditional MAPPO response delay is 350ms, causing the voltage of 2 nodes to drop to 9.1kV, exceeding the limit by 4%, triggering low voltage protection.

[0240] Abnormal operating conditions of energy storage SOC: This invention identifies SOC abnormalities through the physical consistency index Iphy and schedules two adjacent energy storage units to increase discharge power within 500ms to avoid load power outage; the traditional MAPPO does not identify the abnormality and continues to issue discharge commands, resulting in deep discharge of the energy storage unit and permanent capacity decay of 5%.

[0241] Multiple fault superposition conditions: This invention rapidly enhances the correlation strength between photovoltaic-energy storage-load through the coupling degree matrix, and coordinates the execution of the strategy of "photovoltaic priority output + energy storage full power discharge + orderly load reduction", restoring system stability within 240 seconds; the traditional MAPPO, due to its fixed coupling relationship, cannot coordinate quickly, and the recovery time is as long as 580 seconds, resulting in a constraint violation cost of 1560 yuan.

[0242] Based on 30 days of routine operating data and extreme operating condition loss data, the economic benefits are calculated from three aspects: "direct benefits - cost savings - reduction in failure losses".

[0243] In terms of direct benefits, this invention promotes the improvement of photovoltaic (PV) grid integration rate, enabling a 5MW PV power plant to generate an average of 240kWh more per day, with an annual direct power generation revenue of 52,560 yuan, creating an additional revenue advantage compared to the industry average PV grid integration rate. In terms of cost savings, peak-valley arbitrage optimization achieves an annual electricity purchase cost saving of 693,500 yuan, while predictive maintenance replaces regular inspections, reducing the need for maintenance personnel and saving 273,750 yuan in maintenance costs annually. The two cost savings combined amount to 967,250 yuan annually. In terms of reducing fault losses, rapid response under extreme operating conditions reduces annual power generation loss by 1,920 yuan, and decisions that conform to physical laws avoid abnormal equipment wear and tear, extending inverter life and saving 2,170 yuan in equipment replacement costs annually. Fault-related losses are reduced by 4,090 yuan annually. In summary, the total annual economic benefit of this method for the 5MW regional energy system is approximately RMB 1.0239 million, which is significantly higher than that of similar dispatching technologies in the industry. The initial deployment cost of RMB 1.8 million corresponds to an investment payback period of only 1.76 years, which is far lower than the industry average, demonstrating outstanding value for engineering promotion.

[0244] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent reinforcement learning regional energy cooperative scheduling method, characterized by: include: S1. Obtain the local state set of all agents, including distributed photovoltaic units, energy storage systems, loads and grid-side systems. Construct the coupling degree between agents based on energy interaction, power correlation and spatiotemporal distance, and construct the coupling degree matrix based on the coupling degree. S2. Based on the agent's local state set and coupling degree matrix, construct the agent's observation vector, input the agent's observation vector into the policy network, and combine it with the coupling degree matrix to obtain the agent's decision action. S3. Calculate the individual basic reward of the agent based on the agent's local state set and the agent's decision action, calculate the system economic reward based on the individual basic reward, and construct the constraint benchmark reward in stages based on the system economic reward. The process of constructing the constraint benchmark reward in stages based on the system economic reward is as follows: the entire training stage is divided into the first stage, the second stage and the third stage according to the data volume of the agent's observation vector and the agent's decision action. In the first stage, the first global constraint benchmark reward is calculated based on the system economic reward and the equipment operation constraints, and the first global constraint benchmark reward is used as the constraint benchmark reward. The equipment operation constraints include the energy storage charging and discharging power limit and the photovoltaic power output limit. In the second phase, the second global constraint benchmark reward is calculated based on the first global constraint benchmark reward and the local security constraints, and the second global constraint benchmark reward is used as the constraint benchmark reward. In the third stage, the third global constraint benchmark reward is calculated based on the second global constraint benchmark reward and the global constraints, and the third global constraint benchmark reward is used as the constraint benchmark reward. S4. Calculate the individual differentiated basic reward based on the constraint benchmark reward, and integrate the collaborative benefit share of each agent into the individual differentiated basic reward to obtain the agent's allocation reward; S5. Input the agent's decision action into the physical quantity prediction network to obtain the physical quantity prediction value. Calculate the physical consistency reward based on the physical quantity prediction value and the physical quantity calculation value. Integrate the physical consistency reward into the allocation reward to obtain the final reward. Add the final rewards of all agents to obtain the global reward. Calculate the target reward based on the global reward. S6. Concatenate the observation vectors of all agents to obtain the global joint observation vector. Concatenate the decision actions of all agents to obtain the joint action vector. Concatenate the global joint observation vector and the joint action vector and input them into the value network to obtain the global state value estimate. Based on the policy pruning loss, the value loss of the global state value estimate and the target reward, and the coupling regularization term, jointly train the policy network and the value network to obtain the trained policy network and value network. S7. Input the local state set of the agent to be scheduled into the trained policy network, output the scheduling instruction, input the scheduling instruction into the trained value network, and output the scheduling evaluation result.

2. The multi-agent reinforcement learning regional energy cooperative scheduling method according to claim 1, characterized in that: The method for constructing the coupling degree between intelligent agents based on energy interaction, power correlation, and spatiotemporal distance is as follows: in, Let be the coupling degree between agent i and agent j at time t. Let be the energy interaction power between agent i and agent j at time t. This represents the maximum value of the energy interaction power. For the power change of agent i With the power change of agent j covariance, , The power changes of agent i are respectively Power changes of agent j standard deviation Let i be the spatiotemporal distance between agent i and agent j. , , These are the weighting coefficients. This is the distance attenuation coefficient; The coupling degree matrix is: in, This is the coupling degree matrix corresponding to time t. , For the number of agents, The degree of coupling at time t. For electrical connection adjacency matrix, Represents intelligent agents , An electrical connection exists; otherwise, the value is 0. For Hadama accumulation, This represents the normalization function.

3. The multi-agent reinforcement learning regional energy cooperative scheduling method according to claim 1, characterized in that: The process of obtaining the agent's decision-making action by combining the coupling degree matrix is ​​as follows: in, Let i be the decision action of agent i at time t. Let be the observation vector of agent i at time t. This indicates that the agent's observation vector Input Policy Network The basic decision to output This is the coupling effect coefficient. Coupling degree matrix At time t OK, Let i be the decision action of other agents besides agent i at time t-1. For mean averaging; the observation vector of agent i at time t. for: in, Let be the set of local states of agent i at time t. Let be the set of local states of all agents other than agent i at time t. This is a weighted pooling operation.

4. The multi-agent reinforcement learning regional energy cooperative scheduling method according to claim 1, characterized in that: The basic individual reward for an agent is: in, This represents the basic reward for agent i. Let i be the local reward function for agent i. Let i be the set of local states of agent i. For agent i, the decision-making action; System economic rewards for: in, N represents the economic benefits generated by the interaction between the regional energy system and the upper-level power grid, and N is the number of intelligent agents.

5. The multi-agent reinforcement learning regional energy cooperative scheduling method according to claim 1, characterized in that: Individualized basic rewards are: in, As a basic reward for individual differences, To constrain the baseline reward, the proportion of the baseline reward of agent i to the total is... for: , Let i and j represent the basic individual rewards of agents i and j, respectively. Agent reward allocation for: in, The number of agents; Let i be the share of the collaborative benefits of agent i at time t. To allocate benefit weights.

6. The multi-agent reinforcement learning regional energy cooperative scheduling method according to claim 1, characterized in that: Physical consistency reward for: The final reward is: in, For physical quantity prediction networks, The input to the physical quantity prediction network, To enable intelligent agents to make decisions and actions Calculated values ​​of physical quantities obtained from inputting physical equations. Let i be the final reward for agent i at time t. Assign rewards to agent i at time t. Let i be the decision action of agent i at time t. Consistency reward weight.

7. The multi-agent reinforcement learning regional energy cooperative scheduling method according to claim 1, characterized in that: Global state value estimation and the value loss of the target return for: in, This is the time series expectation, i.e., the average over all times t. For global state value estimation, target return for: in, The global reward for time t. It is a discount factor. This is the target value network, and its parameters are updated following the overall value network. Let be the agent's observation vector at time t+1. Represents the observation vector based on time t+1 The estimated global state value.

8. The multi-agent reinforcement learning regional energy cooperative scheduling method according to claim 1, characterized in that: Coupling regularization terms for: in, Let be the mean action of agent i output by the policy network. M is the number of samples. Adaptive Coupling Degree Matrix The i-th row vector, This represents the mean vector of actions of all agents except agent i. This represents the output of a weighted average strategy based on coupling degree.

9. A multi-agent reinforcement learning regional energy collaborative scheduling system, characterized in that: include: The adaptive coupling degree matrix construction module is used to obtain the local state set of all agents, including distributed photovoltaic units, energy storage systems, loads and grid-side systems. The coupling degree between agents is constructed based on energy interaction, power correlation and spatiotemporal distance, and the coupling degree matrix is ​​constructed based on the coupling degree. The decision-making action calculation module is used to construct the agent's observation vector based on the agent's local state set and coupling degree matrix, input the agent's observation vector into the policy network, and obtain the agent's decision-making action by combining it with the coupling degree matrix. The constrained baseline reward phased construction module is used to calculate the individual basic reward of the agent based on the agent's local state set and the agent's decision actions, calculate the system economic reward based on the individual basic reward, and construct the constrained baseline reward in phases based on the system economic reward. The process of constructing the constraint benchmark reward in stages based on the system economic reward is as follows: the entire training stage is divided into the first stage, the second stage and the third stage according to the data volume of the agent's observation vector and the agent's decision action. In the first stage, the first global constraint benchmark reward is calculated based on the system economic reward and the equipment operation constraints, and the first global constraint benchmark reward is used as the constraint benchmark reward. The equipment operation constraints include the energy storage charging and discharging power limit and the photovoltaic power output limit. In the second phase, the second global constraint benchmark reward is calculated based on the first global constraint benchmark reward and the local security constraints, and the second global constraint benchmark reward is used as the constraint benchmark reward. In the third stage, the third global constraint benchmark reward is calculated based on the second global constraint benchmark reward and the global constraints, and the third global constraint benchmark reward is used as the constraint benchmark reward. The reward allocation calculation module is used to calculate the individual differentiated basic reward based on the constraint benchmark reward, and integrate the collaborative benefit share of each agent into the individual differentiated basic reward to obtain the agent's allocation reward; The target reward calculation module is used to input the agent's decision action into the physical quantity prediction network to obtain the physical quantity prediction value. Based on the physical quantity prediction value and the physical quantity calculation value, the physical consistency reward is calculated. The physical consistency reward is integrated into the allocation reward to obtain the final reward. The final rewards of all agents are added together to obtain the global reward. The target reward is calculated based on the global reward. The learning module is used to concatenate the observation vectors of all agents to obtain the global joint observation vector, and to concatenate the decision actions of all agents to obtain the joint action vector. The global joint observation vector and the joint action vector are concatenated and input into the value network to obtain the global state value estimate. Based on the policy pruning loss, the value loss of the global state value estimate and the target reward, and the coupling regularization term, the policy network and the value network are jointly trained to obtain the trained policy network and value network. The inference module is used to input the local state set of the agent to be scheduled into the trained policy network, output scheduling instructions, input the scheduling instructions into the trained value network, and output the scheduling evaluation results.

10. The multi-agent reinforcement learning regional energy cooperative scheduling system according to claim 9, characterized in that: The method for constructing the coupling degree between intelligent agents based on energy interaction, power correlation, and spatiotemporal distance is as follows: in, Let be the coupling degree between agent i and agent j at time t. Let be the energy interaction power between agent i and agent j at time t. This represents the maximum value of the energy interaction power. For the power change of agent i With the power change of agent j covariance, , The power changes of agent i are respectively Power changes of agent j standard deviation Let i be the spatiotemporal distance between agent i and agent j. , , These are the weighting coefficients. This is the distance attenuation coefficient; The coupling degree matrix is: in, This is the coupling degree matrix corresponding to time t. , For the number of agents, The degree of coupling at time t. For electrical connection adjacency matrix, Represents intelligent agents , An electrical connection exists; otherwise, the value is 0. For Hadama accumulation, This represents the normalization function.

Images