Optical storage direct-flexible building cluster optimization operation method, system and equipment based on multi-agent deep reinforcement learning, and medium

Abstract

The invention discloses an optical storage direct-flexible building cluster optimization operation method, system and device based on multi-agent deep reinforcement learning, and a medium, and belongs to the technical field of optical storage direct-flexible building cluster optimization operation, and the method comprises the steps: constructing a building cluster optimization operation model comprising a plurality of building units; a building cluster optimization problem of the building cluster optimization operation model is expressed as a partially observable Markov decision process, and an intelligent agent of each building unit makes a decision according to local observation information; a multi-agent reinforcement learning algorithm is used for solving a decision of a partially observable Markov decision process through a strategy optimization agent of centralized training and decentralized execution. According to the POMDP provided by the invention, on the premise of protecting the data privacy of each building, the designed reward function considers the situation that the flexible load centralized transfer may cause the building operation pressure, and the situation that the load is excessively concentrated at a certain moment is effectively relieved.

Description

Technical Field

[0001] This invention relates to the field of optimized operation technology for photovoltaic-storage-direct-flexible building clusters, specifically to a method, system, equipment, and medium for optimized operation of photovoltaic-storage-direct-flexible building clusters based on multi-agent deep reinforcement learning. Background Technology

[0002] With the rapid development of distributed energy systems, photovoltaic (PV) and other new energy power generation technologies are widely used in distributed energy networks. Buildings, as a crucial sector for achieving my country's "dual-carbon" goals, are becoming a key battleground in my country's energy transition. Against this backdrop, PV-storage-DC-flexible (PSC) technology provides new solutions for building energy management, effectively promoting the local consumption of building-based PV power and ensuring stable and reliable interaction between buildings and the grid. With the development of PSC-Storage-DC-flexible building clusters, coordinated operation among multiple buildings has become an important direction. The promotion of PSC-Storage-DC-flexible building clusters can not only improve the PV consumption rate within buildings but also optimize energy efficiency. However, the complex energy flow relationships among multiple buildings in a building cluster pose certain challenges to the overall coordinated operation of the cluster. How to fully leverage the independent operational advantages of each building in the overall optimized operation of the building cluster, and effectively utilize the power support between the grid and buildings, has become a key issue in improving the overall performance and stability of the building cluster and reducing operating costs.

[0003] A search of existing technical literature revealed that the paper "Stackelberg Game-Based Optimal Dispatch for PEDF Park and Power Grid Interaction under Multiple Incentive Mechanisms" (W.Chen,Y.Zhao,X.Wu, et al.Stackelberg Game-Based Optimal Dispatch for PEDF Park and Power Grid Interaction under Multiple Incentive Mechanisms[J].Energy Engineering,vol.121,no.10,pp.3075-3093,2024) proposes an optimal dispatch strategy for the interaction between PEDF park and power grid based on Stackelberg game theory. This strategy reduces electricity consumption costs, improves power utilization, and increases the penetration rate of renewable energy. The paper "Learning-aided distributionally robust optimization of DC distribution network with buildings to the grid" (Z. Wei, H. Xu, S. Chen, et al. Learning-aided distributionally robust optimization of DC distribution network with buildings to the grid [J]. Sustainable Cities and Society, vol. 113, 2024) constructs a distributed robust optimization model of DC distribution network based on PEDF system and proposes a solution method based on deep learning-assisted model to obtain the system's scheduling strategy. However, the above paper only studies a single PEDF system and does not consider the scheduling strategy under the cooperative operation of multiple PEDF systems.The paper "Grid-connected multi-microgrid system operational scheduling optimization: A hierarchical improved marine predators algorithm" (L.Li, B.Ji, G.Liu, et al. Grid-connected multi-microgrid system operational scheduling optimization: A hierarchical improved marine predators algorithm[J]. Energy, vol.294, 2024) proposes a method for optimizing the operation and scheduling of multi-microgrid systems based on an improved marine predators algorithm. This method considers internal power interaction and load demand response, reducing the total operating cost of the system. However, the heuristic optimization algorithm in the above-mentioned paper often lacks sufficient flexibility and adaptability when facing complex high-dimensional and dynamic environments, and is prone to getting trapped in local optima, resulting in the inability to make optimal scheduling decisions. In recent years, with the development of artificial intelligence technology, existing research has applied deep reinforcement learning (DRL) algorithms with environmental perception and decision-making capabilities to the economic scheduling and energy management of power systems. The paper "Economic Dispatch Strategy for Distributed Integrated Energy System in Industrial Park Based on Multi-Agent Deep Reinforcement Learning" (Zhang Fan, Wu Donghao, Chen Yuping, et al. Economic Dispatch Strategy for Distributed Integrated Energy System in Industrial Park Based on Multi-Agent Deep Reinforcement Learning [J]. Journal of Electric Power System and Automation, 2022, 34(12):18-26.) proposes a multi-energy coordinated and complementary optimization scheduling method based on multi-agent deep Q-networks, realizing multi-energy coordinated and complementary optimization in the integrated energy network system. However, the multi-agent DRL algorithm in this paper still has room for improvement in decision-making efficiency and performance. At the same time, the method does not consider flexible loads with adjustable capabilities when constructing the model, so its application to PEDF buildings will have certain limitations. Summary of the Invention

[0004] To address the aforementioned technical challenges, a method for optimizing the operation of optical-storage-flexible building clusters based on multi-agent deep reinforcement learning is proposed, including:

[0005] Construct an optimized operation model for building clusters that include multiple building units;

[0006] The building cluster optimization problem of the building cluster optimization operation model is formulated as a partially observable Markov decision process, in which the agent of each building unit makes decisions based on local observation information;

[0007] We apply a multi-agent reinforcement learning algorithm to solve partially observable Markov decision processes by optimizing the agent's decision-making through a strategy of centralized training and decentralized execution.

[0008] Offline training is performed to optimize the agent's strategy, and the building cluster is scheduled in real time during the actual operation phase based on the optimized strategy.

[0009] As a preferred embodiment of the photovoltaic-storage-rigid-flexible building cluster optimization operation method based on multi-agent deep reinforcement learning described in this invention, each building unit in the building cluster optimization operation model includes a photovoltaic system, a battery, a rigid load, and a flexible load.

[0010] The flexible load includes transferable loads and reduceable loads, and its power output is adjusted by regulating the flexible load.

[0011] As a preferred embodiment of the photovoltaic-storage-direct-connection-flexible building cluster optimization operation method based on multi-agent deep reinforcement learning described in this invention, the building cluster optimization operation model includes constructing a daily operating cost objective function for the photovoltaic-storage-direct-connection-flexible building cluster using the building cluster maintenance cost, the interaction cost between the building cluster and the power grid, the building cluster interaction cost, carbon emission cost, and flexible load dispatch subsidy.

[0012] As a preferred embodiment of the optical-storage-direct-flexible building cluster optimization operation method based on multi-agent deep reinforcement learning described in this invention, the partially observable Markov decision process includes:

[0013] The state space of each building unit's agent making decisions based on local observation information;

[0014] The action space in which an agent can choose operations at each time step;

[0015] A reward function that evaluates the effect of each agent performing an action.

[0016] As a preferred embodiment of the optical storage direct-flexible building cluster optimization operation method based on multi-agent deep reinforcement learning described in this invention, the multi-agent reinforcement learning algorithm includes,

[0017] One Actor network, a target Actor network, two Critic networks, and a target Critic network.

[0018] As a preferred embodiment of the optical storage direct-flexible building cluster optimization operation method based on multi-agent deep reinforcement learning described in this invention, the step of solving the decision-making of the agent through centralized training and decentralized execution includes,

[0019] During the training phase, each agent selects actions through the Actor network, and the actions of all agents are evaluated and the network parameters are updated through a centralized Critic network.

[0020] During the execution phase, each agent performs its own action based on local observation information.

[0021] As a preferred embodiment of the optical storage direct-flexible building cluster optimization operation method based on multi-agent deep reinforcement learning described in this invention, the offline training to optimize agent strategies includes:

[0022] Initialize the Actor network and Critic network parameters, the corresponding target network parameters, and the experience replay pool for each agent;

[0023] Initialize environment state s t ;

[0024] For an intelligent agent, based on its local environment... i,t Get action a i,t until all agents have chosen an action;

[0025] Execute joint action a t =(a 1,t ,a 2,t ,…,a N,t ), and receive a global reward r t and enter the next state s t+1 ;

[0026] Sample data (s) t ,a t ,r t ,s t+1 Stored in the experience replay pool D;

[0027] B mini-batch samples are drawn from the experience replay pool D;

[0028] Update the network parameters of agent i.

[0029] Another objective of this invention is to provide an optimized operation system for optical-storage-directed flexible building clusters based on multi-agent deep reinforcement learning. First, this invention constructs a PEDF building cluster optimization operation model with the objective function of minimizing total operating cost. Second, considering the privacy of individual building state information, the PEDF building cluster optimization operation problem is described as a partially observable Markov decision process (POMDP). Then, an improved multi-agent dual-delay deep deterministic policy gradient (MATD3) algorithm is proposed to solve the POMDP. This method expands the agent's exploration range in the solution space by introducing an additional action network that extends the action space; simultaneously, it combines a bidirectional long short-term memory (BiLSTM) network and an attention mechanism to enhance the agent's ability to capture key features in the state variables; furthermore, to address the underestimation of the Q-value of the target evaluation network, an averaging operation is employed to optimize the stability of the target evaluation network. Finally, the trained agent model is used to adaptively generate PEDF building cluster optimization operation strategies online. The PEDF building cluster optimization operation method proposed in this invention can significantly improve the energy management and scheduling efficiency of building clusters, thereby reducing system operating costs while optimizing energy utilization.

[0030] As a preferred embodiment of the optical storage direct-flexible building cluster optimization operation system based on multi-agent deep reinforcement learning described in this invention, it is characterized by including a construction module for constructing a building cluster optimization operation model containing multiple building units.

[0031] The decision module is used to describe the building cluster optimization problem of the building cluster optimization operation model as a partially observable Markov decision process, in which the agent of each building unit makes decisions based on local observation information.

[0032] The solution module is used to solve partially observable Markov decision processes by applying a multi-agent reinforcement learning algorithm, and optimizing the agent's decision-making through a strategy of centralized training and decentralized execution.

[0033] The optimization module is used for offline training to optimize the agent's strategy and for real-time scheduling of the building cluster based on the optimized strategy during the actual operation phase.

[0034] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the aforementioned method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning.

[0035] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning.

[0036] The beneficial effects of this invention are as follows: (1) The PEDF optimization operation problem is constructed as a POMDP process. Under the constructed POMDP framework, key indicators such as maintenance cost, interaction cost, carbon emission cost, and flexible load dispatch subsidy of each PEDF building are comprehensively considered. At the same time, under the premise of protecting the data privacy of each building, the reward function designed by the proposed POMDP takes into account the building operation pressure that may be caused by the centralized transfer of flexible loads, effectively alleviating the situation of excessive load concentration at a certain moment.

[0037] (2) A novel multi-agent deep reinforcement learning (MADRL) method is proposed. The proposed method introduces a new Actor network based on MATD3 to enhance the agent's search capability in the early stages and its stable exploration capability in the later stages. Furthermore, the introduction of BiLSTM and attention mechanisms effectively enhances the agent's ability to capture key information. These improvements significantly improve the learning speed and quality of the proposed MATD3 during the training phase, enabling it to quickly generate more accurate scheduling instructions in the PEDF optimization problem.

[0038] (3) The MATD3-based optimized operation method for PEDF building clusters can effectively reduce the load peak-valley difference and daily operating costs of PEDF building clusters. Compared with the original load curve, the proposed method, through the scheduling of flexible loads, reduces the load peak-valley difference of the three PEDF buildings by 17.6%, 14.2%, and 6.4%, respectively, alleviating the load pressure on the power grid and optimizing energy efficiency. Compared with the other three DRL methods, the operating cost of the proposed method is reduced, indicating that the proposed method has significant economic advantages and provides reliable technical support for the sustainable operation of PEDF building clusters. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a structural framework diagram of a PEDF building cluster optimization operation method based on multi-agent deep reinforcement learning, which is provided as an embodiment of the present invention.

[0041] Figure 2This is a comparison of the convergence results of a multi-agent deep reinforcement learning-based optical storage straight-flexible building cluster optimization operation method provided in one embodiment of the present invention with other DRL methods.

[0042] Figure 3 This is a load curve before and after optimization of a method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning, provided as an embodiment of the present invention.

[0043] Figure 4 The results of PEDF building cluster optimization operation are provided in an embodiment of the present invention for an optical-storage-direct-flexible building cluster optimization operation method based on multi-agent deep reinforcement learning.

[0044] Figure 5 This is a comparison chart of the operating costs of a multi-agent deep reinforcement learning-based optical storage straight-flexible building cluster optimization operation method and other DRL methods for optimizing PEDF building clusters, as provided in one embodiment of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0046] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning, including:

[0047] Step 101: Construct an optimal operating model for the PEDF building cluster with the objective function of minimizing total operating cost. Secondly, considering the privacy of individual building state information, the optimal operating problem for the PEDF building cluster is described as a partially observable Markov decision process (POMDP).

[0048] In step 101, the buildings in the PEDF building cluster model constructed by this invention mainly consist of photovoltaics, batteries, rigid loads, and flexible loads. Rooftop photovoltaics provide energy to the loads; batteries provide energy storage services for the buildings, which can improve the flexibility of load-side adjustments.

[0049] In an optional embodiment of the present invention, the load in a building cluster is divided into rigid load and flexible load, wherein the flexible load includes transferable load and reduceable load, and each building can schedule the flexible load according to the subsidy price to adjust the load power in order to reduce the electricity cost.

[0050] In a preferred embodiment of the present invention, the objective function of the PEDF building cluster optimization operation model is expressed as:

[0051]

[0052]

[0053] Where C is the daily operating cost of the PEDF building cluster, T is the number of control periods within the control cycle, and N is the total number of PEDF buildings. t,om C t,grid C t,unit C t,carbon C t,flex K represents the maintenance cost of the building cluster at time t, the interaction cost between the building cluster and the power grid, the interaction cost of the building cluster, the carbon emission cost, and the flexible load dispatch subsidy. ma,PV and K ma,BT These are the maintenance costs for photovoltaic panels and batteries, respectively. Let be the output power of the photovoltaic system, the charging power of the battery, and the discharging power of the battery in the i-th building at time t, respectively. and These represent the electricity purchase price and the electricity sales price at time t, respectively. and This represents the interaction power between the i-th building and the power grid at time t; and c represents the interaction power of the i-th and j-th buildings at time t, respectively; carbon and ε carbon These are the carbon emission penalty cost and the carbon emission coefficient, respectively. and These are the subsidy unit prices for reducing flexible load and transferring flexible load at time t, respectively. and These represent the reduction and transfer amounts of the i-th building's flexible load at time t, respectively.

[0054] It should be noted that in traditional methods, the cost of flexible loads is often reflected in the objective function as a penalty term, meaning that flexible load scheduling increases costs. In this invention, the scheduling cost of flexible loads is incorporated into the objective function as a subsidy value, more clearly quantifying the value of flexible load adjustment and further emphasizing the effect of user response behavior on system optimization through positive incentives.

[0055] Traditional methods often fail to consider the carbon emission costs caused by carbon emissions during operation. The method of this invention takes into account the carbon emission penalty, which can further reflect the carbon emission intensity during the operation process and make the operation strategy more in line with the "dual carbon" target requirements.

[0056] In an optional embodiment of the present invention, the constraint condition includes: the actual power of the photovoltaic system is limited by the rated power.

[0057]

[0058] in, Let be the actual photovoltaic output power of the i-th building at time t; and Let be the minimum and maximum photovoltaic output power of the i-th building at time t, respectively.

[0059] Energy storage systems need to consider power limitations and SOC constraints:

[0060]

[0061] in, and These represent the charging or discharging power of the battery in the i-th building at time t; and These represent the maximum power of the battery charging or discharging in the i-th building at time t; It is a 0-1 variable used to represent the charging / discharging state of the battery; Let be the state of charge of the i-th battery in the building at time t. and Let $t$ be the minimum and maximum states of charge of the battery in the $i$-th building at time $t$.

[0062] Power constraints that allow interaction between building clusters and the power grid:

[0063]

[0064] in, Let be the interaction power between the i-th building and the power grid at time t; and Let be the minimum and maximum interaction power between the i-th building and the power grid at time t, respectively.

[0065] Power constraints that allow interaction between building clusters:

[0066]

[0067] in, Let be the interaction power between the i-th and j-th buildings at time t; and Let be the minimum and maximum interaction power of the i-th and j-th buildings at time t, respectively.

[0068] Flexible load power constraints:

[0069]

[0070] in, Let be the reduction amount of the flexible load within the i-th building at time t; and These represent the maximum and minimum reduction amounts of the flexible load within the i-th building at time t; Let be the amount of flexible load transferred within the i-th building at time t; and These represent the maximum and minimum transfer amounts of the flexible load within the i-th building at time t.

[0071] Step 102 proposes an improved multi-agent dual-delay deep deterministic policy gradient (MATD3) algorithm to solve POMDP. The proposed method expands the exploration range of the agent in the solution space by introducing an additional action network that expands the action space.

[0072] It should be noted that the solution method of this invention is achieved through multi-agent deep reinforcement learning. The specific steps are as follows: ① Based on the research objective of this invention, namely the PEDF optimization problem, the values ​​in the POMDP are constructed by combining the objective function, specifically in the state space s, action space a, and reward function r; ② The agents are trained using the improved MATD3 algorithm, that is, the agents representing each building are trained using the MATD3 algorithm. Each agent will perform action a based on its own observation of s to maximize the collective reward r, thereby achieving optimal fitting of the deep neural network parameters in MATD3, thus achieving the training objective; ③ The trained agents can make the optimal action that minimizes the collective reward value based on the real-time state of the PEDF cluster, thereby completing the formulation of the optimization strategy.

[0073] The solution process of traditional deep reinforcement learning is the same as that of this invention. The improvement of this invention is mainly reflected in the construction of the POMDP model and the improvement of the solution algorithm, as detailed below:

[0074] ① In the construction of the reward function of the POMDP model, the traditional method often constructs it in the form of an objective function. The penalty term added by this invention is used to avoid the building's power being too high at a certain moment due to flexible load transfer, which would affect its safe and stable operation. This improvement can effectively reduce the impact of the optimization operation strategy on safe and stable operation in the traditional method; this is specifically reflected in step 102.

[0075] ② For the MATD3 algorithm, the improved method of this invention is reflected in steps 103 and 104, including the following three points:

[0076] (1) The traditional MATD3 algorithm uses only one Actor network to reflect the actions of the output agent. A single Actor network struggles to adapt its optimization performance as the number of iterations increases, leaving room for improvement in the final training performance. This invention adds an Actor network with the same structure to the TD3 network structure, creating a dual-Actor network. This method allows the algorithm to select a suitable Actor network based on different iteration counts, improving its early-stage search capability and later-stage stability.

[0077] (2) Traditional MATD3 uses deep neural networks to acquire system state in real time, but MATD3 based on multilayer perceptron (MLP) networks may not be able to effectively extract temporal features when acquiring system state. This invention introduces a BiLSTM network to enable the agent to fully extract state variables, allowing the agent to make decisions by making full use of time series features;

[0078] (3) The traditional MATD3 algorithm takes the minimum value of the two target Critic networks to obtain the predicted value of the Q function, which leads to an underestimation of the Q value, causing the agent to fail to make the optimal decision. This invention improves the accuracy of the target Critic network estimation by using the average value of the value functions of the two target Critic networks as the predicted value of the Q function, enabling the agent to learn better actions.

[0079] In an optional embodiment of the present invention, the optimized operation of the PEDF building cluster can be regarded as a sequential decision problem based on a multi-agent system. Since each building operates in a decentralized manner, only a portion of the information in the system can be observed. Therefore, the present invention formulates the optimized operation problem as a POMDP.

[0080] In an optional embodiment of the present invention, POMDP includes a state space, an action space, and a reward function.

[0081] In a preferred embodiment of the present invention, the state space is defined as follows: Due to privacy restrictions, agents in each PEDF building can only take actions by observing their local state information. The environmental state observed by the agent in the i-th building at time t can be defined as:

[0082]

[0083] in, This represents the total power demand of the load. Power required for flexible loads.

[0084] The sum of the observation information of each agent constitutes the global state s. t , can be represented as:

[0085] st =(o 1,t ,o 2,t ,...,o N,t )

[0086] Action space: Each sub-agent selects an action based on the state information within its region. The action space a of the agent in the i-th building at time t is... i,t It can be defined as:

[0087]

[0088] In a multi-agent environment, the sum of the actions of each sub-agent constitutes the joint action a. t , can be represented as:

[0089] a t =(a 1,t ,a 2,t ,...,a N,t )

[0090] Reward function: global state s t After each agent performs an action, it will transition to a new global state s. t+1 Each agent receives a reward r t Based on the constructed objective function, the reward function is set as follows:

[0091]

[0092] in, Let $t$ be the maintenance cost of the i-th building at time $t$, the interaction cost with the power grid, the interaction cost between building clusters, the carbon emission cost, and the flexible load dispatch subsidy, respectively.

[0093] To avoid excessive load power caused by the concentrated transfer of flexible loads at the same time, which could affect the safe operation of the main power grid, this invention further adds a reward correction mechanism, namely, adding a penalty for exceeding the original load peak, as shown below:

[0094]

[0095] Where K is the penalty factor. Let t be the total load power in the original i-th building.

[0096] The optimized reward function can be expressed as:

[0097]

[0098] In MADRL, the goal is to maximize long-term returns, i.e., cumulative discounted rewards, through policy optimization. Based on the reward functions of each agent, the global reward function r is obtained. t for:

[0099]

[0100] Here, γ is the discount factor, which reflects the current value of future rewards.

[0101] Step 103: Combine a bidirectional long short-term memory (BiLSTM) network and an attention mechanism to enhance the agent's ability to capture key features in the state variables;

[0102] In an optional embodiment of the present invention, the constructed POMDP is solved using an improved MATD3 algorithm. The MATD3 algorithm comprises an Actor network and a target Actor network, as well as two Critic networks and a target Critic network. The proposed MATD3 algorithm adopts a "centralized training, distributed execution" architecture. During the training phase, each agent selects actions through the Actor network, and the actions of all agents are evaluated and network parameters are updated through a centralized Critic network. During the execution phase, each agent executes its own actions based on local observation information. This invention proposes an improved MATD3 algorithm, enhancing the algorithm's exploration capability and avoiding getting trapped in local optima.

[0103] In a preferred embodiment of the present invention, firstly, the proposed method adds an Actor network with the same structure to the TD3 network structure, forming a dual-Actor network. In early training, the Actor network with a larger loss function is selected to participate in the TD3 algorithm training; this strategy can enhance the algorithm's exploration ability in the early stages. In later training, the Actor network with a smaller loss function is given priority, thereby reducing the magnitude of parameter updates and stabilizing the algorithm's convergence process in the later stages. This process can be represented as follows:

[0104]

[0105] in, and Here, represents the loss functions of the two Actor networks; D represents the state distribution. For the expectation of the state distribution; π(s) t ;θ1) and π(s t ;θ2) represents the two Actor networks in state s t The generated action; θ1 and θ2 are the parameters of the two Actor networks, respectively; θ Q represents the parameters of the Critic network; epoch represents the number of iterations during training; α represents the threshold.

[0106] Furthermore, to avoid instability in the entire training process due to excessively large differences in parameters between the two Actor networks trained at different stages, the proposed method updates the parameters of the two networks through a soft update mechanism. The specific expression is as follows:

[0107]

[0108] θ j =εθ i ′+(1-ε)θ j ′ i≠j

[0109] Where, θ i The parameters of the selected Actor network; θ j For the parameters of another Actor network; θ i ′ represents the parameters of the selected Actor network before the update; The learning rate for the selected Actor network; For the loss function with respect to θ i The gradient of θ′; j ′ represents the parameters of the other Actor network before the update; ε is a constant close to 1.

[0110] In a preferred embodiment of the present invention, secondly, in view of the fact that the TD3 algorithm based on the multilayer perceptron (MLP) network cannot fully extract the time features in the running state data and cannot effectively utilize the time series state data in the environment, the present invention introduces the Bi-LSTM network and the attention mechanism into the Actor network in the TD3 algorithm, so that the agent can make decisions by making full use of the time series features.

[0111] BiLSTM networks simultaneously explore forward and backward features in data by combining forward and backward LSTM networks. The proposed method embeds the BiLSTM network into the Actor network, enhancing the network's ability to capture state variable features. The parameters of the BiLSTM network satisfy the following relationship:

[0112]

[0113] Among them, h t and h t ′ represent the hidden features of the data in the forward and backward channels, respectively; H t Features after merging; x t The input is time series data; σ is the activation function; W y b is the weight matrix; y This is a bias term.

[0114] It should be noted that this invention assigns different weights to the extracted features through an attention mechanism, accurately reflecting the degree of influence of different features on the output and reducing unnecessary information redundancy. The attention mechanism can be represented as follows:

[0115]

[0116] Where, k i For attention scoring functions; and W a H is the attention parameter matrix; i The input features are tanh, the activation function is b. a For the bias term; β i For attention weights; H a Z is the output of the attention module; Z is the number of input features.

[0117] Step 104: To address the issue of underestimated Q-value in the target evaluation network, an averaging operation is employed to optimize the stability of the target evaluation network.

[0118] In a preferred embodiment of the present invention, finally, to address the problem of underestimated Q-value in the target Critic network in the TD3 algorithm, the present invention uses the average value function of the two target Critic networks as the predicted value of the Q-function, thereby improving the accuracy of the target Critic network estimation. Specifically, this can be expressed as follows:

[0119]

[0120] Among them, y target The Q-value of the target Critic network. and These are the Q-value functions of the two target Critic networks, respectively; and The parameters are for two target Critic networks; The action output by the Actor network.

[0121] Step 105: Utilize the trained agent model to adaptively generate an online PEDF building cluster optimization operation strategy. The PEDF building cluster optimization operation method proposed in this invention can significantly improve the energy management and scheduling efficiency of building clusters, thereby reducing system operating costs while optimizing energy utilization.

[0122] In an optional embodiment of the present invention, the application process of the proposed method includes offline training of all agents using the proposed improved MATD3 algorithm and online optimization of the PEDF building cluster based on the trained model.

[0123] In a preferred embodiment of the present invention, the specific steps of the offline training phase include:

[0124] Step 1: Initialize the Actor network, Critic network parameters, corresponding target network parameters, and experience replay pool for each agent.

[0125] Step 2: Initialize environment state s t ;

[0126] Step 3: For agent i, based on its local environment o i,t Get action a i,t ;

[0127] Step 4: Repeat step 2 until all agents have selected an action;

[0128] Step 5: Execute the combined action a t =(a 1,t ,a 2,t ,…,a N,t ), and receive a global reward r t and enter the next state s t+1 ;

[0129] Step 6: Transfer the sample data (s) t ,a t ,r t ,s t+1 Stored in the experience replay pool D;

[0130] Step 7: Draw B mini-batch samples from the experience replay pool D;

[0131] Step 8: Update the network parameters of agent i;

[0132] Step 9: Repeat step 7 until all agent network parameters have been updated;

[0133] Step 10: Repeat steps 3-9 until all training time slots are completed;

[0134] Step 11: Repeat steps 2-10 until all rounds of training are completed.

[0135] In a preferred embodiment of the present invention, the specific steps of the online application stage include:

[0136] Step 1: Deploy the trained model;

[0137] Step 2: Input the initial state of all agents;

[0138] Step 3: All agents execute scheduling actions;

[0139] Step 4: All agents receive a reward and proceed to the next state;

[0140] Step 5: Repeat steps 3 and 4 until the optimization run for all time periods is completed.

[0141] Example 2, refer to Figures 1-5 This is a second embodiment of the present invention, providing a method for optimizing the operation of a light-storage-direct-flexible building cluster based on multi-agent deep reinforcement learning, comprising:

[0142] This embodiment uses three PEDF buildings to simulate the optimized operation of a PEDF building cluster. First, load power data, photovoltaic power data, and energy storage status information of the three PEDF buildings are collected and divided into training and test sets. Second, four algorithms—MATD3, MASAC, MADDPG, and MATD3—are selected as benchmark algorithms for comparison. The training set is used to train these algorithms, and the average reward obtained by the agent during training is shown below. Figure 2 As shown. Then, to verify the effectiveness of the proposed method, the trained MATD3 algorithm was used to optimize the scheduling of the selected PEDF cluster. The scheduling results for a 24-hour test day are shown below. Figure 3-4 As shown. Finally, to verify the superiority of the proposed method, three comparative algorithms—MASAC, MADDPG, and MATD3—were used to optimize the scheduling of the PEDF building cluster on a test day. The costs of the four methods on this test day are as follows: Figure 5 As shown.

[0143] Reference Figure 1 First, information such as photovoltaic output, load power, energy storage device status, and time-of-use electricity price is collected from each building. Then, this data is used as state input to train the improved MATD3 algorithm. The improved MATD3 algorithm proposed in this invention introduces a new Actor network to enhance the agent's early search capabilities and improve the stability of later decisions, based on the original Actor network. Furthermore, the proposed method combines a BiLSTM network and an attention mechanism to improve the Actor network's ability to extract key feature information. During training, firstly, based on the state information of each building, the agent selects an action through the Actor network and receives a corresponding reward based on the selected action. Next, the rewards obtained by all agents are evaluated through a centralized Critic network to achieve a global value judgment. During this process, the state, action, reward, and next-time state of each agent are stored as experience samples in a shared experience replay pool. Subsequently, the network parameters are updated by sampling from the experience pool until the model converges. After the training phase, the agent model will be deployed to each building. The agent will make optimization decisions based on local building information to complete the optimized operation of the building cluster.

[0144] Reference Figure 2 ,from Figure 2 As can be seen, the proposed method converges faster than other DRL algorithms, reaching convergence after approximately 400 iterations. In contrast, MATD3 exhibits a slower convergence speed than the improved MATD3. Although its reward value gradually increases, its convergence speed is relatively slow, failing to rapidly improve performance in the initial stage. MASAC and MADDPG also have slow convergence speeds, showing almost no significant improvement in the early stages of training and maintaining a relatively slow growth in the later stages. This is mainly because the proposed method enhances the agent's exploration ability by introducing a dual-Actor network, enabling it to quickly identify better strategies in the early stages of training, thus promoting rapid convergence. Regarding the reward value obtained, the proposed method can achieve a high reward value in a shorter training time. In contrast, although MATD3 is relatively stable, its final convergence reward value is lower than that of the improved MATD3, while the reward values ​​of MASAC and MADDPG fluctuate greatly and do not reach convergence within the same time frame. This is mainly because the proposed method introduces a BiLSTM network and an attention mechanism, which enables the Actor network to extract key information more effectively from complex multi-agent environments, enhancing the accuracy and stability of decision-making. The introduction of the BiLSTM network allows the model to better utilize historical information for bidirectional learning, while the attention mechanism helps the network automatically focus on features crucial to decision-making, avoiding interference from redundant information. Therefore, it can be seen that the model trained using the proposed method outperforms other DRL algorithms.

[0145] Reference Figure 3 The load curve before optimization was obtained by predicting the load power of buildings on typical days. As shown in the figure, the proposed method, under flexible load scheduling and price guidance, achieved stable load scheduling for buildings. The load demand of each building was effectively scheduled to periods with lower electricity prices, reducing peak-hour power demand and thus achieving load balance and optimized power consumption. After optimization, the peak-to-valley difference of building load decreased significantly. Specifically, the peak-to-valley difference decreased by 17.6% for building A, 14.2% for building B, and 6.4% for building C. This result indicates that through optimized scheduling, load fluctuations in buildings were effectively controlled, helping to alleviate the load pressure on the power grid and optimize energy efficiency. Furthermore, compared to the original load curve, the optimized load curve did not exhibit excessively large load differences at certain times. This is mainly due to the inclusion of a penalty function related to the load difference before and after optimization in the reward function of the POMDP model, which avoids excessive load concentration, ensures the smoothness and stability of the optimization process, and demonstrates the effectiveness of the optimization strategy.

[0146] Reference Figure 4 ,from Figure 4 As can be seen, during the off-peak electricity period from 1:00 to 7:00, buildings meet their electricity needs by purchasing electricity from the grid and discharging batteries. From 11:00 to 15:00 and from 19:00 to 21:00, buildings have higher electricity demands. Due to peak electricity prices, buildings reduce their grid purchases and increase battery discharge and power exchange with other buildings, effectively lowering overall operating costs. Simultaneously, adjustments are made to flexible loads, shifting transferable loads to lower-price periods and rationally reducing loads that can be reduced, effectively lowering peak loads and operating costs during peak electricity consumption periods.

[0147] Reference Figure 5 Yes, it can be seen that compared with other DRL methods, the method proposed in this invention effectively reduces the operating costs of each building. Specifically, in building A, the optimized operating cost of the proposed method is 11.2% lower than MASAC, 12.2% lower than MADDPG, and 5.4% lower than MATD3. In building B, the optimized operating cost of the proposed method is 11.7% lower than MASAC, 13.6% lower than MADDPG, and 14.9% lower than MATD3. In building C, the optimized operating cost of the proposed method is 9.6% lower than MASAC, 15.5% lower than MADDPG, and 6.6% lower than MATD3. This demonstrates that the proposed method has significant economic advantages in energy scheduling and cost optimization for building clusters.

[0148] Example 3 is the third embodiment of the present invention, which differs from the previous two embodiments in that:

[0149] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0151] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0152] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0153] Example 4, the fourth embodiment of the present invention, provides an optimized operation system for optical storage, direct and flexible building clusters based on multi-agent deep reinforcement learning, including...

[0154] The building module is used to construct an optimized operating model for building clusters containing multiple building units;

[0155] The decision module is used to describe the building cluster optimization problem of the building cluster optimization operation model as a partially observable Markov decision process, in which the agent of each building unit makes decisions based on local observation information.

[0156] The solution module is used to solve partially observable Markov decision processes by applying a multi-agent reinforcement learning algorithm, and optimizing the agent's decision-making through a strategy of centralized training and decentralized execution.

[0157] The optimization module is used for offline training to optimize the agent's strategy and for real-time scheduling of the building cluster based on the optimized strategy during the actual operation phase.

[0158] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for optimizing the operation of optical storage-flexible building clusters based on multi-agent deep reinforcement learning, characterized in that: include, Construct an optimized operation model for building clusters that include multiple building units; The building cluster optimization problem of the building cluster optimization operation model is formulated as a partially observable Markov decision process, in which the agent of each building unit makes decisions based on local observation information; We apply a multi-agent reinforcement learning algorithm to solve partially observable Markov decision processes by optimizing the agent's decision-making through a strategy of centralized training and decentralized execution. Offline training is performed to optimize the agent's strategy, and the building cluster is scheduled in real time during the actual operation phase based on the optimized strategy.

2. The method for optimizing the operation of a light-storage-direct-flexible building cluster based on multi-agent deep reinforcement learning as described in claim 1, characterized in that: Each building unit in the building cluster optimization operation model includes a photovoltaic system, a battery, a rigid load, and a flexible load; The flexible load includes transferable loads and reduceable loads, and its power output is adjusted by regulating the flexible load.

3. The method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning as described in claim 2, characterized in that: The optimized operation model for the building cluster includes constructing a daily operating cost objective function for the photovoltaic-storage-DC-flexible building cluster based on the maintenance cost of the building cluster, the interaction cost between the building cluster and the power grid, the interaction cost of the building cluster, carbon emission costs, and flexible load dispatch subsidies.

4. The method for optimizing the operation of a light-storage-direct-flexible building cluster based on multi-agent deep reinforcement learning as described in claim 3, characterized in that: The partially observable Markov decision process includes, The state space of each building unit's agent making decisions based on local observation information; The action space of the operations that an agent can choose at each time step ； A reward function that evaluates the effect of each agent performing an action.

5. The method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning as described in claim 4, characterized in that: The multi-agent reinforcement learning algorithm includes, One Actor network, a target Actor network, two Critic networks, and a target Critic network.

6. The method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning as described in claim 5, characterized in that: The method of optimizing the agent's decision-making through a strategy of centralized training and decentralized execution includes... During the training phase, each agent selects actions through the Actor network, and the actions of all agents are evaluated and the network parameters are updated through a centralized Critic network. During the execution phase, each agent performs its own action based on local observation information.

7. The method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning as described in claim 6, characterized in that: The offline training to optimize the agent strategy includes... Initialize the Actor network and Critic network parameters, the corresponding target network parameters, and the experience replay pool for each agent; Initialize environment state s t ; For an intelligent agent, based on its local environment... i,t Get action a i,t until all agents have chosen an action; Execute joint action a t =(a 1,t ,a 2,t ,…,a N,t ), and receive a global reward r t and enter the next state s t+1 ; Sample data (s) t ,a t ,r t ,s t+1 Stored in the experience replay pool D; B mini-batch samples are drawn from the experience replay pool D; Update the network parameters of agent i.

8. A system for optimizing the operation of a cluster of optical-storage-linear-flexible buildings based on multi-agent deep reinforcement learning, employing the method for optimizing the operation of such clusters as described in any one of claims 1 to 7, characterized in that... include: The building module is used to construct an optimized operating model for building clusters containing multiple building units; The decision module is used to describe the building cluster optimization problem of the building cluster optimization operation model as a partially observable Markov decision process, in which the agent of each building unit makes decisions based on local observation information. The solution module is used to solve partially observable Markov decision processes by applying a multi-agent reinforcement learning algorithm, and optimizing the agent's decision-making through a strategy of centralized training and decentralized execution. The optimization module is used for offline training to optimize the agent's strategy and for real-time scheduling of the building cluster based on the optimized strategy during the actual operation phase.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning, as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for optimizing the operation of a light-storage-direction-flexible building cluster based on multi-agent deep reinforcement learning, as described in any one of claims 1 to 7.

Images