Multi-type micro-grid cross-layer collaborative scheduling method based on multi-agent reinforcement learning

Abstract

The invention provides a multi-type micro-grid cross-layer collaborative scheduling method based on multi-agent reinforcement learning, and relates to the technical field of power system and micro-grid scheduling, and the method comprises the steps: firstly constructing a multi-type micro-grid layered collaborative architecture, a multi-agent reinforcement learning scheduling model, and a centralized coordination layer where global agents are deployed in a platform layer; and the local agents are deployed at each distributed micro-grid node of the platform layer, a cross-layer cooperative training mode is adopted to train the model, and finally, cooperative decision results of the global agents and the local agents are integrated to generate a global-local cooperative scheduling scheme. According to the method, global optimization and local flexibility can be considered, and the hierarchical multi-agent architecture adopts a multi-type micro-grid hierarchical collaborative architecture, so that the method can be suitable for multi-type micro-grid operation scheduling of a novel power system, and multi-type micro-grid collaborative scheduling considering global optimization and local flexibility is realized.

Description

Technical Field

[0001] This invention relates to the field of power system and microgrid dispatching technology, and in particular to a multi-agent reinforcement learning-based cross-layer collaborative dispatching method for multi-type microgrids. Background Technology

[0002] In recent years, with the accelerated global energy transition and the continuous rise in the penetration rate of new energy sources, large-scale centralized grid connection of new energy sources has brought challenges to the safe operation of the power grid and the balance of power supply. Microgrids, a new technological form integrating source, grid, load, and storage resources on the distribution side, have been widely applied in various scenarios as a core carrier integrating photovoltaic, wind power, energy storage, and diverse loads. However, microgrid application scenarios cover a variety of differentiated scenarios, including renewable energy scenarios, industrial scenarios, and commercial scenarios. These differences in application scenarios pose a severe challenge to the global optimization and local response coordination capabilities of the power system. The distribution network needs to coordinate the power interaction of multiple microgrids to avoid line overload and voltage exceeding limits. Different types of microgrids have different characteristics and focuses, requiring flexible adjustment of internal source, load, and storage resources to maximize their own benefits. Coordinated optimization of multi-type microgrids can improve the renewable energy absorption rate and ensure the reliability of power supply on the user side by breaking down the operational islands between microgrids. Currently, multi-microgrid dispatching technologies are mainly divided into three categories: centralized dispatching, distributed dispatching, and preliminary coordinated dispatching.

[0003] Centralized scheduling methods achieve global optimization by centralizing global data and solving a unified model; distributed scheduling methods optimize each microgrid independently and interact through boundary parameters; preliminary collaborative scheduling methods assume homogeneous microgrid characteristics and perform simple power interaction, boundary parameter exchange, and static power allocation to coordinate scheduling of multiple types of microgrids.

[0004] The disadvantages of centralized scheduling methods are that, although they can achieve global optimization, they rely heavily on accurate forecast data when facing fluctuations in the proportion of new energy sources. At the same time, their computational complexity increases exponentially with the number of microgrids, making them unable to meet real-time scheduling requirements. The disadvantages of distributed scheduling methods are that, since each microgrid optimizes independently, it is easy to fall into the trap of local optima. The disadvantages of preliminary collaborative scheduling methods are that they often assume that the characteristics of microgrids are homogeneous and do not design scheduling strategies for the differentiated needs of different types of microgrids. Summary of the Invention

[0005] To address the technical problems of existing microgrid scheduling methods being unable to simultaneously achieve global optimization and flexible local response, and being unable to adapt to the characteristics of different types of microgrids, the purpose of this invention is to provide a multi-agent reinforcement learning-based cross-layer collaborative scheduling method for multiple types of microgrids, realizing collaborative scheduling of global optimization and flexible local response for multiple types of microgrids.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a multi-agent reinforcement learning-based method for cross-layer collaborative scheduling of various types of microgrids, comprising: S1. acquiring historical and real-time operating status data of various types of microgrids and preprocessing them to obtain preprocessed historical operating status data and real-time operating status data; the operating status data includes photovoltaic output, load demand, energy storage SOC, and tie-line power; S2. based on the preprocessed historical operating status data and microgrid type, constructing a multi-type microgrid layered collaborative architecture including a device layer, a platform layer, and an application layer; the device layer is the local execution layer of the microgrid; the platform layer is a hybrid collaborative decision layer, including a centralized coordination layer and multiple distributed microgrid nodes; the application layer is a scenario-based adaptation layer, used to determine the scheduling objective function and weights according to different application scenarios; S 3. Based on a multi-type microgrid hierarchical collaborative architecture, a multi-agent reinforcement learning scheduling model is constructed, including a global agent deployed on the centralized coordination layer and local agents deployed on each distributed microgrid node; S4. Based on preprocessed historical operating status data, a centralized training-distributed execution mode is used to perform cross-layer collaborative training on the multi-agent reinforcement learning scheduling model; S5. The preprocessed real-time operating status data is input into the trained multi-agent reinforcement learning scheduling model. The global agent outputs the real-time tie-line power plan as the global decision, and each local agent outputs the real-time local device operation instructions as the local decision. Based on the real-time deviation correction and fusion mechanism, the global decision and the corresponding local decision are fused to generate a global-local collaborative scheduling scheme.

[0007] The present invention provides a preferred embodiment in which the preprocessing includes: data cleaning, fixed sampling interval, removal of outliers, timestamp alignment, and polynomial interpolation to complete the data.

[0008] This invention provides a preferred embodiment in which the centralized coordination layer of the platform layer receives device layer data from all microgrids and is responsible for global optimization, and each distributed microgrid node is independently deployed in its respective microgrid, receives boundary constraints from the centralized coordination layer and is responsible for local device optimization.

[0009] This invention provides a preferred embodiment in which the global agent is constructed based on a deep deterministic policy gradient algorithm and a microgrid type attention mechanism. The deep deterministic policy gradient algorithm includes the current Actor network, the current Critic network, the target Actor network, and the target Critic network. It uses an experience replay mechanism to collect interaction experience, updates the target network parameters through a soft update strategy, updates the Critic network to minimize value error, and updates the Actor network to maximize action value.

[0010] This invention provides a preferred embodiment in which the local agent is constructed based on a deep Q-network. The deep Q-network fits the Q-value function through a deep neural network, generates actions using a greedy exploration strategy, samples batches of samples through an experience replay mechanism, introduces a target Q-network to calculate the Temporal Difference target, and uses a hard update strategy to synchronize the parameters of the target network and the current Q-network.

[0011] Further preferably, the implementation process of the microgrid type attention mechanism is as follows: during the state input stage of the global agent, a trainable type embedding vector is assigned to each microgrid type, and the dimension of the type embedding vector is aligned with the original operating state vector; the original operating state vector under each microgrid type is added element-wise to the type embedding vector to obtain the fusion feature; through the attention mechanism, the differentiated weights of the fusion features of different microgrid types are dynamically determined, and the final input state of the global agent is obtained after performing attention-weighted summation on the fusion features of all microgrid types.

[0012] This invention provides a preferred embodiment in which a multi-agent reinforcement learning scheduling model is trained across layers using a centralized training-distributed execution mode based on preprocessed historical operating state data. This includes: determining the environmental state and actions of each local agent and the global agent, where the environmental state of each local agent includes the power generation, load power, energy storage SOC, and tie-line power plan of each microgrid; and the actions of the local agents are local equipment control quantities that satisfy power balance constraints, including energy storage charging and discharging power, controllable load adjustment, and distributed power output. The environmental state of the global agent includes the environmental state of each local agent, tie-line power plan, local execution deviation, and scenario weight; and the actions of the global agent are the tie-line power of each microgrid that satisfy power conservation constraints. A cross-layer reward function with feedback linkage is designed, with local rewards fed back to the global agent, and the global reward fused with local reward feedback and the global objective. Global and local policies are collaboratively updated through cross-layer parameter synchronization and experience sharing until convergence to a Nash equilibrium state, at which point the training of the multi-agent reinforcement learning scheduling model is complete.

[0013] More preferably, the cross-layer parameter synchronization update specifically involves updating the global agent policy parameters and local agent policy parameters through hard synchronization and gradient fusion strategies; the experience sharing specifically involves injecting the experience samples of local agents into the global experience pool according to their weights.

[0014] Further preferably, when the changes in global policy parameters, the maximum changes in all local agent policy parameters, and the changes in global reward within two consecutive time steps do not exceed the first preset convergence threshold, the second preset convergence threshold, and the third preset convergence threshold, respectively, it is determined that the convergence has reached the Nash equilibrium state.

[0015] This invention provides a preferred embodiment in which the global-local collaborative scheduling scheme is generated by fusing global decisions and corresponding local decisions based on a real-time deviation correction and fusion mechanism. This includes: when there is a deviation between the real-time tie-line power plan and the corresponding microgrid's actual local execution value according to the real-time local equipment operation command, dynamically adjusting the weights of the global decisions and the corresponding local decisions, and generating a tie-line power plan fusion result and a local equipment operation command fusion result as the global-local collaborative scheduling scheme.

[0016] Compared with the prior art, the present invention has the following advantages: This invention first constructs a hierarchical collaborative architecture for multiple types of microgrids, then builds a multi-agent reinforcement learning scheduling model. The global agent is deployed in the centralized coordination layer of the platform layer, while local agents are deployed in each distributed microgrid node of the platform layer. Subsequently, a cross-layer collaborative training method is used to train the model. Finally, the collaborative decision-making results of the global agent and local agents are integrated to generate a global-local collaborative scheduling scheme as the final scheduling scheme. This invention mainly achieves a balance between global optimization and local flexibility by deploying global and local agents in a hierarchical multi-agent architecture and performing collaborative optimization scheduling based on global and local decisions. Furthermore, the hierarchical multi-agent architecture adopts a multi-type microgrid hierarchical collaborative architecture, making the scheduling method of this invention applicable to the operation and scheduling of multiple types of microgrids in new power systems. It has diverse application prospects in various microgrid system application scenarios, realizing collaborative scheduling of multiple types of microgrids that balances global optimization and local flexibility. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the steps of a multi-agent reinforcement learning-based cross-layer cooperative scheduling method for multi-type microgrids provided in a specific embodiment of the present invention. Figure 2 This diagram illustrates the implementation process of a multi-agent reinforcement learning-based cross-layer collaborative scheduling method for multi-type microgrids, as provided in a specific embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Considering that existing multi-type microgrid scheduling technologies struggle to balance global optimization with flexible local response, cannot adapt to the differences in microgrid characteristics, and exhibit decision-making lag, repetitive modeling requirements, and insufficient robustness when facing scenario fluctuations, leading to low overall operating efficiency, limited renewable energy absorption, and high operating costs, this invention provides a multi-agent reinforcement learning-based cross-layer collaborative scheduling method for multi-type microgrids. Please refer to [reference needed]. Figure 1 This is mainly achieved through steps S1 to S5: S1. Acquire historical and real-time operating status data of various types of microgrids and preprocess them to obtain preprocessed historical operating status data and real-time operating status data; the operating status data includes photovoltaic output, load demand, energy storage SOC and tie-line power.

[0021] In S1, firstly, regarding data acquisition, in one optional implementation, source-load-storage data of different types of microgrids such as wind-solar-storage microgrids and industrial microgrids are acquired based on terminal sensors, edge computing gateways, and cloud coordination centers. The data includes photovoltaic output, load demand, energy storage SOC, and tie-line power.

[0022] More specifically, taking two types of microgrid data and global interactive data as examples, the two types of microgrid data include wind-solar-storage microgrid data and industrial microgrid data: Wind-solar-storage microgrid data includes real-time photovoltaic output (0-5MW), real-time wind power output (0-3MW), energy storage SOC (0-100%), energy storage charging and discharging power (-2MW to +2MW, negative for charging and positive for discharging), and local residential load (1-6MW). This type of microgrid data exhibits strong time-varying characteristics and varies depending on time and weather.

[0023] Industrial microgrid data: gas turbine output (0-3MW), controllable motor load (3-8MW), load regulation response speed (0.5MW / min), production line operating status (running / shutting down); the load data of industrial microgrids are strongly correlated with production plans, reflecting the characteristics of strong load rigidity but short-term adjustment.

[0024] Global interactive data: power of interconnecting lines between microgrids (-3MW to +3MW), distribution network time-of-use tariff (0.5-1.2 yuan / kWh), current power of lines (0-20MW) and rated capacity (20MW); global interactive data reflects the collaborative relationship between microgrids, including the complementarity of different types of microgrids at different times and in different weather conditions, as well as the difference in electricity prices during peak and off-peak periods. Meanwhile, the load of the backbone network is closely related to security.

[0025] Subsequently, for data preprocessing, in a preferred embodiment, the preprocessing mainly involves: performing data cleaning, fixing the sampling interval, removing outliers, aligning timestamps, and using polynomial interpolation to complete the data, thereby forming a standardized multi-type microgrid time series dataset for subsequent multi-agent model training and collaborative decision optimization.

[0026] S2. Based on preprocessed historical operating status data and microgrid types, a multi-type microgrid hierarchical collaborative architecture is constructed, including a device layer, a platform layer, and an application layer. The device layer is the local execution layer of the microgrid; the platform layer is a hybrid collaborative decision-making layer, including a centralized coordination layer and multiple distributed microgrid nodes; the application layer is a scenario-based adaptation layer, used to determine the scheduling objective function and weights according to different application scenarios.

[0027] Understandably, in S2, a multi-layered collaborative architecture for microgrids is constructed to address the aforementioned application scenarios and data, comprising a device layer, a platform layer, and an application layer; please refer to [reference needed]. Figure 2 The system comprises three layers: the device layer, which is the local execution layer of the microgrid, deploying corresponding sensing and execution units based on the differences in microgrid types; and the platform layer, a hybrid collaborative decision-making layer, which includes a centralized coordination layer and distributed microgrid nodes. The centralized coordination layer receives device layer data from all microgrids and is responsible for global optimization, while the distributed microgrid nodes are deployed independently in each microgrid, receiving boundary constraints from the centralized coordination center and being responsible for local device optimization. The application layer is a scenario-based adaptation layer, adjusting the objective function and weights according to different scenarios.

[0028] S3. Based on a multi-type microgrid hierarchical collaborative architecture, a multi-agent reinforcement learning scheduling model is constructed, including a global agent deployed on a centralized coordination layer and local agents deployed on each distributed microgrid node. In a preferred embodiment, the centralized coordination layer of the platform layer receives device layer data from all microgrids and is responsible for global optimization. Each distributed microgrid node is independently deployed in its respective microgrid, receives boundary constraints from the centralized coordination layer, and is responsible for local device optimization. More preferably, the global agent is constructed based on a deep deterministic policy gradient algorithm and a microgrid type attention mechanism. The deep deterministic policy gradient algorithm includes the current Actor network, the current Critic network, the target Actor network, and the target Critic network. It uses an experience replay mechanism to collect interaction experience, updates the target network parameters through a soft update strategy to minimize the value error in updating the Critic network, and maximizes the action value in updating the Actor network.

[0029] Understandably, the global agent is deployed in the centralized coordination layer of the platform layer and is responsible for global decision-making. The global agent is built around the Deep Deterministic Policy Gradient (DDPG) algorithm and incorporates a microgrid type attention mechanism. DDPG is a deep reinforcement learning-based algorithm suitable for solving problems in continuous action spaces. Combining deterministic policies and deep neural networks, DDPG is a model-free reinforcement learning algorithm belonging to the Actor-Critic framework and possessing the advantages of both DQN and policy gradient. Its environment state, actions, and reward functions are the global state, actions, and multi-objective reward functions of each microgrid. The principle of DDPG is as follows: The core of DDPG includes two current networks and two target networks, as well as experience replay and policy update rules; the current network includes the current Actor and the current Critic, and the target network includes the target Actor and the target Critic.

[0030] The Actor network outputs deterministic policies, determining deterministic actions given a system state; the target Actor network calculates the target action for the next state; the current Critic network outputs the corresponding value based on the input state and action; and the target Critic network calculates the Temporal Difference (TD) objective, which serves as the Critic's learning objective.

[0031] More specifically, we will first break down and explain the steps involved in constructing the global agent in S3: Step 1: Interact with the environment and collect empirical data. At this stage, the output of the current Actor network is a deterministic action.

[0032]

[0033] in, The action output by the Actor network at time t; The state of the environment at time t; (*) represents the parameter. The current Actor neural network is usually a fully connected layer, which outputs a definite value in a continuous action space; These are the parameters of the neural network; and These are the weights and biases in the fully connected layer, respectively. For the activation function, this application chooses the Sigmoid function.

[0034] After interaction, experience tuples are obtained and stored in the experience replay pool. Once the replay pool has accumulated enough data, a batch of samples is randomly sampled for subsequent network updates.

[0035] The second step is to calculate the TD objective. The Critic network performs action value evaluation, which evaluates the value of the action output by the Actor in the corresponding state, i.e., the Q value.

[0036] Current Critic network:

[0037] Target Critic Network:

[0038] The Critic needs to learn the true value of actions to more accurately evaluate the value of the Actor's output actions. The true value, based on Bellman optimization theory, is determined by the sum of the immediate reward and the target value for the next state. First, the optimal action for the next state is generated using the target Actor network. Then, the target Critic network evaluates the value of this action, and finally, it is summed with the immediate reward.

[0039]

[0040] In the above formula, (*) represents the parameter. The current Critic neural network is usually composed of fully connected layers. The formula for the fully connected layer neural network has been given in the above section and will not be repeated here. The action to calculate the next state for the target Actor; These are the parameters of the current Critic network; The Q-value represents the value of the current action in the corresponding state, as output by the current Critic network. (*) represents the parameter. The target Critic neural network is usually composed of fully connected layers. The formula for a fully connected neural network has been given in the above section and will not be repeated here. The parameters of the target Critic network; The value of the future action output by the target Critic network in the corresponding state; For parameters The target Actor neural network; The next state; For instant rewards; (*) represents the evaluation of the value of the action in the next state; The TD objective is the current learning objective of the Critic. This is the discount factor.

[0041] The parameters of the target Actor network do not directly participate in gradient updates, but are tracked through soft updates to keep track of the current Actor's parameters.

[0042] Step 3: Update the current Critic network to minimize the value error. The Critic's task is to make the current Q-value approximate the TD target. This is achieved by updating the parameters by minimizing the mean squared error. The loss function during the current Critic training process is:

[0043] in, is the Critic loss function; N is the total number of time steps; the other parameters have been given in the above sections and will not be repeated here.

[0044] The network parameters are optimized using gradient descent. The network parameters for the target Critic are also updated using soft updates.

[0045] Step 4: Update the Actor network to maximize action value. The Actor's goal is to maximize the Q-value evaluated by the Critic; the Actor network updates its parameters through deterministic policy gradients, the direction of which is determined by the gradient of the Critic with respect to the action and the gradient of the Actor with respect to its own parameters. The Actor's policy gradient is:

[0046] Optimization through gradient ascent :

[0047] in, The learning rate of the Actor; For the Actor's parameters; For gradient operators; Let be the objective function of the policy, which measures the performance of the current Actor network. Starting from the initial state, follow the policy. The expected cumulative discount reward; N is the total number of time steps; The state at time t; The gradient of the Critic with respect to the action is the partial derivative of the Q-value output by the Critic network with respect to the action. It represents how the Q-value changes when the action a undergoes a small change, and can provide the Actor with the direction of adjustment to improve the value of the action. The gradient of the Actor with respect to its own parameters, i.e., the action output by the Actor network. Regarding its own parameters The partial derivative of represents how the output action changes when the Actor's parameters change slightly.

[0048] Step 5: Soft update the target network. To avoid fluctuations in the target network parameters affecting training stability, soft updates are used to allow the target network to slowly track the current network parameters, thus stabilizing the training. Target Critic parameter update:

[0049] Target Actor parameter update:

[0050] In the above formula, The soft update coefficient is 0.001 in this application; the other parameters have been given in the above sections and will not be repeated here.

[0051] During training, the above process is repeated in each iteration until the Actor policy converges and can output the optimal action in the continuous action space.

[0052] In a preferred embodiment, a microgrid type attention mechanism is also incorporated into the global agent. This mechanism dynamically determines differentiated weights for different states to address the differences between various microgrid types, thereby enhancing the adaptability of global decision-making to different microgrid types. The implementation process of the microgrid type attention mechanism is as follows: During the state input phase of the global agent, a trainable type embedding vector is assigned to each microgrid type, with the dimension of the type embedding vector aligned with the original operating state vector; the original operating state vector and the type embedding vector under each microgrid type are element-wise added to obtain fusion features; the differentiated weights of the fusion features for different microgrid types are dynamically determined through the attention mechanism; and the final input state of the global agent is obtained by performing attention-weighted summation on the fusion features of all microgrid types.

[0053] More specifically, during the state input phase of the global agent, microgrid type information is fused with the original state data; a trainable type embedding vector (also known as a one-hot encoded vector) is assigned to each microgrid type, with its dimension aligned with the original operating state vector; and an attention mechanism is used to weight the state vectors of different microgrid types.

[0054]

[0055]

[0056] In the above formula, For microgrid indexing; This represents the total number of microgrids in the system. The original importance score for each microgrid; Let be the original operating state vector of the m-th microgrid; and These are the trainable weight vector and the bias term, respectively; These are trainable, independent-hot encoded vectors used to explicitly embed microgrid type information; This represents element-wise addition, combining the original state of the microgrid with the microgrid type information; For the first Attention weights for individual microgrids; The global state vector, after type embedding and attention weighting, is the final input state of the global agent.

[0057] In a preferred embodiment, the local agent is constructed based on a deep Q-Net. The deep Q-Net fits the Q-value function using a deep neural network, generates actions using a greedy exploration strategy, samples batches of data through an experience replay mechanism, introduces a target Q-Net to calculate the Temporal Difference objective, and uses a hard update strategy to synchronize the parameters of the target network with the current Q-Net. The local agent is deployed in each distributed microgrid node within the platform layer, responsible for optimizing the operation of local devices. The local agent is built around a deep Q-Net (DQN). DQN is an algorithm combining deep learning and reinforcement learning. By using a deep neural network to approximate the Q-value function, DQN can handle high-dimensional state spaces. The core concepts of DQN include experience replay and a target network, which effectively improve learning stability and efficiency.

[0058] More specifically, the steps for constructing local agents in S3 will be broken down and explained in detail: Step 1: DQN uses a deep neural network to fit the value function (Q-value function) of the state and action, representing the expected cumulative discounted reward obtained after choosing the current action in the current state. The formula is:

[0059] Step 2: Exploration Strategy. To balance exploring new actions with utilizing known optimal actions, DQN employs a greedy strategy to generate interactive actions, with the following formula:

[0060] After interaction, the experience tuple is obtained and stored in the experience replay pool.

[0061] In the above formula, The output of DQN is an estimate of the value of states and actions; The immediate reward obtained after performing an action; The state at time t; For the action at time t; These are the parameters of the current DQN network; The parameters of the target Q-network; (*) indicates the statistical expectation of the outcome of the interaction with the environment; The next state is to perform an action. The new state that was subsequently transferred to; For the next action; This represents finding the maximum value of 'a' among all possible actions, representing a greedy strategy. As a discount factor, it weighs the importance of current and future rewards; To find the action that maximizes the Q value for action a, i.e., the optimal action selection; For exploration rate; These are random actions, that is, exploratory actions randomly selected from a discrete action space; Step 3: To address the potential training instability caused by temporal correlation of samples in reinforcement learning, DQN randomly samples a batch of samples from the experience replay pool for network updates, instead of using the latest samples. Let the batch size be N, and the sampling results are:

[0062] Step 4: TD Target Calculation. DQN introduces a target Q-network to calculate a stable target Q-value, avoiding target oscillations caused by fluctuations in current network parameters. The TD target formula is:

[0063] In the above formula, A batch sample is a set of empirical data sampled from the empirical replay pool. This is a termination flag, which can be either 0 or 1, with 1 indicating that the current state is a termination state; Let be the TD target at time t; the other parameters have been given in the above section and will not be repeated here.

[0064] Step 5: Network update to minimize Q-value error. The current optimization objective of the Q-network is to minimize the mean squared error between the predicted Q-value and the TD target. The loss function formula is:

[0065] Step 6: The target network is synchronized via hard updates. To maintain the stability of the target network, DQN employs a hard update strategy, which means that at fixed intervals, the parameters of the current Q-network are directly copied to the target Q-network.

[0066] In the above formula, This represents the total number of time steps. The loss function of the Q network is the mean square error between the predicted Q value and the TD target, which is optimized using the gradient descent algorithm; the other parameters have been given in the above sections and will not be repeated here.

[0067] S4. Based on preprocessed historical operating status data, a centralized training-distributed execution mode is adopted to conduct cross-layer collaborative training of the multi-agent reinforcement learning scheduling model. Cross-layer collaborative training is a hierarchical linkage training mechanism between the global agent in the centralized coordination layer and the local agents in the distributed microgrid layer: the global agent collects various types of microgrid data preprocessed by the equipment layer, learns the global policy through the global agent, and outputs the tie-line power plan as the constraint boundary of the local agents; the local agents optimize the local equipment operation policy under the global constraints and feed back the local rewards and execution deviations to the global agent; through parameter synchronization and experience sharing, a collaborative update that takes into account both global policy and local adaptation is achieved, and finally, the global and local policies converge to a Nash equilibrium state that takes into account both global optimality and local flexibility.

[0068] More specifically, S4 is implemented through the following sub-steps: Step 1: Determine the environmental state and actions of each local agent and the global agent respectively: The global agent's environment state is:

[0069] The global agent's environmental state includes the environmental state of each local agent (also known as the local state of each microgrid), tie-line power plan, local execution deviation, and scenario weight. The global agent's action is the tie-line power of each microgrid that satisfies the power conservation constraint.

[0070] The global agent's action is to output the power of each microgrid tie line:

[0071] Satisfy the power conservation constraint:

[0072] in, The power of each microgrid interconnect line.

[0073] The local agent's environment state is:

[0074] The environmental state of each local agent includes the power generation (also known as wind and solar output), load power, energy storage SOC and tie-line power plan of each microgrid. The actions of the local agents are local equipment control quantities that satisfy power balance constraints, including energy storage charging and discharging power, controllable load adjustment and distributed power output.

[0075] This includes constraints issued by the global agent (power balance constraints), which local actions must satisfy:

[0076] The actions of a local agent are local device control variables:

[0077] In the above formula, m is the microgrid number. Let m be the set of states of the m-th local agent at time t; Let be the power generation equipment power of the m-th local intelligent agent at time t; Let be the load power of the m-th local agent at time t; Let SOC be the energy storage device of the m-th local agent at time t; The power exchanged on the communication lines of the m-th local agent at time t; and The distribution represents the power flowing outward and flowing inward from the outside at time t for the m-th local agent; Let m be the set of local device control variables of the m-th local agent at time t; Let m be the energy storage charging and discharging power of the m-th local intelligent agent at time t; Let be the controllable load adjustment amount of the m-th local agent at time t; The distributed power output of the m-th local agent at time t. This refers to the local execution deviations fed back by the local agent.

[0078] Step 2: Design a cross-layer reward function with feedback linkage, where local rewards are fed back to the global agent, and the global reward is integrated with the local reward feedback and the global goal. The local reward is:

[0079] The feedback is then sent to the global intelligent agent.

[0080] Global rewards integrate local feedback and global goals:

[0081] In the above formula, For indexing local intelligent agents or microgrids; This represents the total number of local intelligent agents or microgrids. Let be the local reward of the m-th local agent at time t; and The distribution represents the weighting coefficients of local rewards, used to weigh the importance of different items in the local rewards; Let be the local cost of the m-th local agent at time t; Let be the power change of the m-th local agent at time t; The global reward at time t; Let be the reward weight of the m-th local agent at time t, used to weigh the importance of rewards for different local agents; Let be the global total cost at time t; Step 3: Through cross-layer parameter synchronization and experience sharing, global and local policies are updated collaboratively until convergence to the Nash equilibrium state, and the training of the multi-agent reinforcement learning scheduling model is completed.

[0082] Cross-layer parameter synchronization update is as follows: Update the global agent policy parameters and local agent policy parameters using a hard synchronization and gradient fusion strategy:

[0083] Experience sharing is as follows: The experience samples of local agents are injected into the global experience pool according to certain weights:

[0084] In the above formula, The policy parameters are updated for the m-th local agent at time t+1; Let be the policy parameters of the m-th local agent at time t; This is the parameter synchronization coefficient, used to balance the ratio of local original parameters to global synchronization parameters; Let be the parameter mapping matrix for the m-th local agent, used for mapping global parameters to local parameters; This is a projection function, ensuring that the parameters are within a valid range; and They are time points and time The global experience pool; The weight coefficients for the experience of the m-th local agent; Let t be the local experience pool of the m-th local agent.

[0085] The changes in global policy parameters, the maximum changes in policy parameters of all local agents, and the changes in global reward within two consecutive time steps do not exceed the first preset convergence threshold. Second preset convergence threshold The third preset convergence threshold When the convergence reaches the Nash equilibrium state, it is determined that the convergence has reached the state described above.

[0086] More specifically, a convergence to a Nash equilibrium is considered to occur when the global and local policies satisfy the following conditions: Global policy stability: The change in global policy parameters within two consecutive time steps does not exceed the first preset convergence threshold. :

[0087] Local policy consistency: The maximum change in the policy parameters of all local agents within two consecutive time steps exceeds the second preset convergence threshold. :

[0088] Global reward convergence: The change in global reward within two consecutive time steps does not exceed the third preset convergence threshold. :

[0089] in, Let be the policy parameters updated by the global agent at time t+1. These are the policy parameters of the global agent at time t; The global reward at time t+1. The global reward at time t; , and These are the convergence thresholds.

[0090] S5. The preprocessed real-time operating status data is input into the trained multi-agent reinforcement learning scheduling model. The global agent outputs the real-time tie-line power plan as the global decision, and each local agent outputs its own real-time local device operation instructions as its own local decisions. Based on the real-time deviation correction and fusion mechanism, the global decision and the corresponding local decisions are fused to generate a global-local collaborative scheduling scheme. In a preferred embodiment, when there is a deviation between the real-time tie-line power plan and the corresponding microgrid's actual local execution value according to the real-time local device operation instructions, the weights of the global decision and the corresponding local decisions are dynamically adjusted. After weighted fusion, the tie-line power plan fusion result and the local device operation instruction fusion result are generated as the global-local collaborative scheduling scheme.

[0091] Integrated collaborative decision-making results refer to the fusion and optimization of the tie-line power plan output by the global agent in the centralized coordination layer and the local device operation instructions output by the local agents in the distributed microgrid layer to form the final execution plan. Based on a real-time deviation correction mechanism, when there is a deviation between the global plan and the local execution, the decision weights of both parties are dynamically adjusted through weight allocation. This ensures both global power balance and system security, while retaining the flexibility of the local microgrid to cope with new energy fluctuations and load changes. The final output is a collaborative scheduling action that takes into account both global optimization objectives and local characteristics, achieving efficient collaborative operation of multiple types of microgrids.

[0092] More specifically, the tie-line power execution value (local execution value) output by the local agent is derived from local decision-making:

[0093] Then, dynamic weight allocation is performed based on the deviation between the global plan and local execution: Absolute deviation:

[0094] Relative deviation:

[0095] The weights of global and local decisions are adjusted based on the relative deviation. A smaller deviation means more local flexibility is retained, while a larger deviation means stronger global constraints.

[0096]

[0097] Generate the final execution plan (generate the tie-line power plan fusion result and the local equipment operation command fusion result as the global-local collaborative scheduling plan): Tether power fusion results:

[0098] Local device execution command fusion results: ) In the above formula, For time steps; This represents the absolute deviation between the planned tie-line power and the actual local value. This refers to the relative deviation between the planned tie-line power and the locally executed value. The output of the tie-line power is the value of the local agent. The power plan for the interconnection line output to the global agent; This represents the maximum planned power for the tie line. Global weight; Weights for local decision-making; Let be the tie-line power finally executed by the m-th microgrid at time t; Local device energy storage charging and discharging plan; For the eventual integration of local device energy storage charging and discharging plans; Let be the local distributed power output of the m-th local agent at time t; Let m be the base load power of the m-th local agent at time t; Let be the power generation equipment power of the m-th local intelligent agent at time t; Let be the load change of the m-th local agent at time t; k is the deviation influence coefficient. Based on global weights.

[0099] Other device instructions are adjusted according to the same logic.

[0100] Through the above specific embodiments, the present invention can achieve the following beneficial technical effects: This invention proposes a multi-agent reinforcement learning-based cross-layer collaborative scheduling method for various types of microgrids. It constructs a hierarchical collaborative architecture for various types of microgrids by building a device layer, a platform layer, and an application layer. Through differentiated perception at the device layer, hybrid decision-making at the platform layer, and scenario-based adaptation at the application layer, it achieves accurate and efficient linkage among various types of microgrids. This allows the needs of various types of microgrids to be met collaboratively in the hierarchical collaboration, solving the problems of data silos, scheduling target conflicts, and weak scenario adaptability caused by the differences in characteristics of various types of microgrids.

[0101] The proposed method for cross-layer collaborative scheduling of multi-type microgrids based on multi-agent reinforcement learning constructs a global agent with DDPG as the core. DDPG can learn the dynamic interaction rules of multiple microgrids in a continuous action space and output smooth tie-line power plans through a policy network, taking into account both global objectives and local flexibility. This makes global decisions more in line with the actual operation of complex and ever-changing microgrids, solving the problems of traditional optimization algorithms that rely on accurate physical models, have weak dynamic adaptability to new energy fluctuations and load changes, and are difficult to handle multi-objective coupled optimization.

[0102] The proposed method for cross-layer collaborative scheduling of multi-type microgrids based on multi-agent reinforcement learning incorporates a microgrid type attention mechanism into the global agent constructed with DDPG as the core. During the state processing of the global agent in the centralized coordination layer, the attention mechanism dynamically determines the differentiated weights of different states based on the differences between different types of microgrids, thereby improving the pertinence and collaborative accuracy of the global strategy in mixed scenarios of multi-type microgrids. This solves the problems of lack of adaptability of global decision-making to different types of microgrids and the decision bias caused by homogenizing the processing of different types of microgrids.

[0103] The proposed method for cross-layer collaborative scheduling of multi-type microgrids based on multi-agent reinforcement learning constructs local agents with DQN as the core. DQN can quickly learn the operating characteristics of local devices and output local control actions adapted to the microgrid type under global tie-line power constraints. Furthermore, it improves sample utilization efficiency through an experience replay mechanism, solving the problems of slow response speed of local devices, difficulty in adapting to the differences in characteristics of different microgrids, and decision accuracy being greatly affected by local data noise.

[0104] The proposed method for cross-layer collaborative scheduling of multi-type microgrids based on multi-agent reinforcement learning enables hierarchical collaborative training of global and local agents. By synchronizing parameters, sharing experiences, and adjusting scenario-based target weights between global and local agents, the method achieves collaborative convergence of global and local policies, enabling the system to maintain high efficiency in dynamic scenarios. This solves the problems of disconnect between global optimization and local execution, policy inconsistency caused by isolated training data, and difficulty in achieving Nash equilibrium that balances global optimization and local flexibility in multi-type microgrids.

[0105] The proposed method for cross-layer collaborative scheduling of multi-type microgrids based on multi-agent reinforcement learning integrates and optimizes the tie-line power plan output by the global agent in the centralized coordination layer with the local equipment operation instructions output by each local agent in the distributed microgrid layer. This integration and collaboration forms the final decision result, ensuring both global power balance and local flexibility. It achieves efficient collaborative operation of multi-type microgrids and solves the problems of large deviations between global planning and local execution, limited flexibility due to rigid constraints, and difficulty in achieving both global and local optima in multi-type microgrid collaboration.

[0106] In summary, this invention features strong practicality, high flexibility in coordinated scheduling, strong anti-interference capability, and strong robustness. It can be applied to the operation of multi-microgrid clusters in novel power systems and has diverse application prospects in various microgrid system application scenarios. This method can achieve coordinated scheduling of multiple types of microgrids with power collaborative optimization and reliable dynamic scenario response. This solves the technical problems of existing multi-type microgrid scheduling technologies, such as difficulty in balancing global optimization and flexible local response, inability to adapt to the differences in characteristics of different microgrids, and issues with decision-making lag, need for repeated modeling, and insufficient robustness when facing scenario fluctuations, leading to low overall operating efficiency, limited renewable energy consumption, and high operating costs.

[0107] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0108] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. Furthermore, the above embodiments only illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning, characterized in that, The application relates to a multi-type micro-grid hierarchical collaborative scheduling method. S1. Obtain historical and real-time operation state data of a multi-type micro-grid and perform preprocessing to obtain preprocessed historical operation state data and real-time operation state data; the operation state data includes photovoltaic output, load demand, energy storage SOC and tie-line power; S2. Construct a multi-type micro-grid hierarchical collaborative architecture including a device layer, a platform layer and an application layer based on the preprocessed historical operation state data and the micro-grid type; The device layer is a micro-grid local execution layer; the platform layer is a hybrid collaborative decision layer, including a centralized coordination layer and a plurality of distributed micro-grid nodes; the application layer is a scenario-adaptive layer, used for determining a scheduling objective function and a weight according to different application scenarios; S3. Construct a multi-agent reinforcement learning scheduling model based on the multi-type micro-grid hierarchical collaborative architecture, including a global agent deployed on the centralized coordination layer and local agents deployed on the distributed micro-grid nodes; S4. Perform cross-layer collaborative training on the multi-agent reinforcement learning scheduling model in a centralized training-distributed execution mode based on the preprocessed historical operation state data; S5. Input the preprocessed real-time operation state data into the trained multi-agent reinforcement learning scheduling model, and output a real-time tie-line power plan as a global decision by the global agent, and output real-time local device operation instructions as local decisions by the local agents; based on a real-time deviation correction and fusion mechanism, the global decision and the corresponding local decisions are fused to generate a global-local collaborative scheduling scheme.

2. The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The preprocessing includes data cleaning, fixed sampling interval, removal of abnormal values, timestamp alignment and polynomial interpolation data completion.

3. The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The centralized coordination layer of the platform layer receives device layer data of all micro-grids and is responsible for global optimization, and each distributed micro-grid node is independently deployed on each micro-grid and receives boundary constraints from the centralized coordination layer and is responsible for local device optimization.

4. The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The global agent is constructed based on a deep deterministic policy gradient algorithm and a micro-grid type attention mechanism, the deep deterministic policy gradient algorithm includes a current Actor network, a current Critic network, a target Actor network and a target Critic network, an experience replay mechanism is used to collect interaction experience, a soft update strategy is used to update target network parameters, a Critic network is updated to minimize value error, and an Actor network is updated to maximize action value.

5. The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The local agent is constructed based on a deep Q network, the deep Q network fits a Q value function through a deep neural network, a greedy exploration strategy is used to generate actions, a batch of samples is sampled through an experience replay mechanism, a target Q network is introduced to calculate a Temporal Difference target, and a hard update strategy is used to synchronize target network and current Q network parameters.

6. The multi-agent reinforcement learning-based multi-type microgrid cross-layer collaborative scheduling method according to claim 4, characterized in that, The implementation process of the micro-grid type attention mechanism is as follows: in the state input stage of the global agent, a trainable type embedding vector is allocated to each micro-grid type, the dimension of the type embedding vector is aligned with that of an original operation state vector, and the original operation state vector under each micro-grid type is added to the type embedding vector element by element to obtain fused features. By means of the attention mechanism, the differentiated weights of the fusion features of different micro-grid types are dynamically determined, and the final input state of the global agent is obtained by performing attention weighted summation on the fusion features of all micro-grid types.

7. The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The pre-processed historical operation state data is used to train the multi-agent reinforcement learning scheduling model in a centralized training-distributed execution mode for cross-layer collaborative training, including: The environment states and actions of each local agent and global agent are determined respectively, wherein the environment state of each local agent includes the generation power, load power, energy storage SOC and tie-line power plan of each micro-grid, and the action of the local agent is the local device control amount including the energy storage charging and discharging power, controllable load adjustment amount and distributed power output amount that meets the power balance constraint; the environment state of the global agent includes the environment state of each local agent, tie-line power plan, local execution deviation and scene weight, and the action of the global agent is the tie-line power of each micro-grid that meets the power conservation constraint; A cross-layer reward function with feedback linkage is designed, and the local reward is fed back to the global agent, and the global reward is fused with the local reward feedback and global target; The global strategy and local strategy are updated cooperatively through cross-layer parameter synchronous updating and experience sharing until the Nash equilibrium state is converged, and the multi-agent reinforcement learning scheduling model training is completed.

8. The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning according to claim 7, characterized in that, The cross-layer parameter synchronous updating specifically updates the global agent strategy parameters and local agent strategy parameters through hard synchronization and gradient fusion strategy; and the experience sharing specifically injects the experience samples of the local agent into the global experience pool according to the weight. 9.The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning of claim 7, wherein, When the global strategy parameter variation amplitude in two consecutive time steps, the maximum variation amplitude of all local agent strategy parameters and the global reward variation amplitude are less than the first preset convergence threshold, the second preset convergence threshold and the third preset convergence threshold respectively, it is determined that the Nash equilibrium state is converged.

10. The multi-type microgrid cross-layer collaborative scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The global-local collaborative scheduling scheme is generated by fusing the global decision and the corresponding local decision based on the real-time deviation correction and fusion mechanism, including: when there is a deviation between the real-time tie-line power plan and the local execution value actually executed by the corresponding micro-grid according to the real-time local device operation instruction, the weight of the global decision and the weight of the corresponding local decision are dynamically adjusted, and the tie-line power plan fusion result and the local device operation instruction fusion result are generated as the global-local collaborative scheduling scheme after weighted fusion.

Images