Energy storage operation optimization method and system based on hierarchical multi-agent reinforcement learning

Abstract

The invention discloses an energy storage operation optimization method and system based on hierarchical multi-agent reinforcement learning. The method comprises the following steps: firstly, initializing a hierarchical multi-agent reinforcement learning decision framework, and defining an interlayer communication synchronization mechanism; then, designing an upper layer reward function for balancing the electric charge income and the battery life loss cost, and optimizing an upper layer planning strategy; a lower layer reward function integrating current smoothness and budget tracking precision is designed, and a lower layer execution strategy is optimized; and finally, deploying the optimized upper-layer planning strategy and the optimized lower-layer execution strategy to an energy storage scheduling task, generating a charging and discharging power budget by an upper-layer planning agent in a day-ahead scheduling stage, and receiving a budget instruction and performing power adjustment by a lower-layer execution agent. According to the invention, by constructing the hierarchical decision-making architecture, coordinated optimization of short-term economic benefit and long-term life protection of the energy storage equipment is realized.

Description

Technical Field

[0001] This invention belongs to the field of power grid energy storage operation optimization technology, specifically involving an energy storage optimization method and system based on hierarchical multi-agent reinforcement learning. Background Technology

[0002] As the penetration rate of renewable energy in the power system continues to increase, the operational optimization of energy storage systems, as a key technology for balancing grid supply and demand and improving power system stability, is becoming increasingly prominent. Traditional energy storage optimization methods mainly focus on maximizing short-term economic benefits, achieving peak-valley electricity price arbitrage through frequent charging and discharging operations, but neglect the long-term lifespan degradation of energy storage devices. This short-sighted optimization strategy leads to premature degradation of energy storage devices and a significant reduction in their overall life-cycle economic efficiency.

[0003] Patent application CN120165417A primarily coordinates hybrid energy storage systems through model optimization and frequency division control, focusing on improving system investment, operating costs, and reliability. Patent application CN119864839A focuses on the configuration and scheduling of microgrid energy storage to improve renewable energy utilization, reduce costs, and ensure system stability. Patent application CN117955133A optimizes energy storage site selection, capacity, and daily operation plans through improved particle swarm optimization algorithms to enhance voltage stability, reduce costs, and minimize network losses.

[0004] Existing technologies have optimized the operation of energy storage systems from multiple perspectives, but some shortcomings still exist. First, the economic efficiency of energy storage devices throughout their entire life cycle is neglected. The life-cycle degradation models in existing technologies cannot accurately reflect the complex degradation mechanism of batteries under actual operating conditions, resulting in a significant deviation between the optimization results and the actual operating performance.

[0005] Secondly, existing technologies generally suffer from a single decision-making level, employing a flat decision-making structure that fails to effectively handle the coupling relationships between decision variables at different time scales. This mixture of long-term planning and short-term execution decisions leads to a sharp increase in computational complexity, makes it difficult to guarantee real-time performance, and limits the targeted application of different optimization algorithms, failing to fully leverage the advantages of various algorithms in specific problems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application provides a method and system for optimizing energy storage operation based on hierarchical multi-agent reinforcement learning. By constructing a hierarchical decision-making architecture, it achieves coordinated optimization of short-term economic benefits and long-term life protection of energy storage equipment.

[0007] The first aspect of this application provides an energy storage operation optimization method based on hierarchical multi-agent reinforcement learning, which adopts the following technical solution: Initialize the hierarchical multi-agent reinforcement learning decision framework, with the upper-layer planning agent comprising the upper-layer planning agent and the lower-layer execution agent, and define the inter-layer communication mechanism; Construct a quantitative model of the monetization cost of battery health status, design an upper-level reward function that balances electricity revenue and battery life depreciation cost, and optimize the upper-level planning strategy. Design the state space, action space, and reward function of the lower-level executive agent, construct a lower-level reward function that integrates current smoothness and budget tracking accuracy, and optimize the lower-level execution strategy; Based on the optimized upper-layer planning strategy and lower-layer execution strategy, the energy storage scheduling task is executed. The upper-layer planning agent generates the charging and discharging power budget during the day-ahead scheduling phase, and the lower-layer execution agent receives the budget instructions and performs power adjustment.

[0008] Furthermore, the upper-layer planning agent generates a power budget allocation vector and sends it to the lower-layer execution agent; the lower-layer execution agent performs real-time power adjustment and feeds back the execution result to the upper-layer planning agent; The inter-layer message format between the upper-layer planning agent and the lower-layer execution agent is defined as follows: ; in, For inter-layer messages, For the energy storage system's operating timestamp, Represents the agent identifier. Indicates the type of energy storage control message. For controlling the load of the energy storage system.

[0009] Furthermore, the upper-level reward function is represented by the difference between the electricity revenue incentive term and the battery loss penalty term; The calculation method for the electricity revenue incentive is as follows: calculate the electricity revenue for a single time period based on the electricity price and charging / discharging power budget at each moment; sum up the revenue of all 24 time periods, and then multiply by the revenue weight to obtain the total electricity revenue. The battery loss penalty is calculated as follows: combining the charging and discharging power budget, battery state of charge, and battery health status at each moment, the loss cost for a single time period is calculated through the battery life loss cost function; the gains of all 24 time periods are summed up and multiplied by the life loss weight to obtain the cumulative loss cost. The State of Health (SOH) of the battery represents the degree of battery degradation.

[0010] Furthermore, regarding the return weights and lifetime loss weight The balance relationship is achieved using a dynamic weight adaptive adjustment mechanism: ; ; In the formula, and For weight adjustment parameters, For a moment Battery health status.

[0011] Furthermore, the battery life loss cost function is the product of the life loss base coefficient, the power influence factor, the SOC deviation penalty, and the health state decay. The life loss baseline coefficient is the basic aging rate of the battery under battery temperature and historical cumulative cycle number. The power impact factor item has the absolute value of the charging and discharging power budget as the base and the index as the power impact coefficient related to the health status. The SOC deviation penalty term has the natural constant e as the base and the exponent is the product of the temperature sensitivity coefficient and the absolute SOC deviation; the absolute SOC deviation is the absolute value of the difference between the current power level and the optimal power level. The health status degradation term uses the remaining health loss of the battery as the base and the exponent as a health status penalty coefficient related to the number of charge-discharge cycles; the remaining health loss of the battery is expressed as... .

[0012] Furthermore, the lower-level reward function is the sum of the current smoothing penalty term, the budget tracking penalty term, and the constraint violation penalty term; The current smoothing penalty term is the product of the square of the current change value at adjacent time points and the current smoothing weight coefficient, and is negative. The budget tracking penalty term is the product of the square of the remaining deviation at the current time and the budget tracking weight coefficient, and is negative. The constraint violation penalty term is the product of the result of the multi-constraint comprehensive judgment function and the constraint violation penalty coefficient, and is negative.

[0013] Furthermore, the current smoothing weight coefficient is adaptively adjusted based on the battery health status and battery temperature; the adjusted current smoothing weight coefficient is the product of the base current smoothing weight, the health status adjustment factor, and the temperature adjustment factor. The health status regulation factor is represented as , The time is indicated; the temperature adjustment factor is based on the natural constant e, and the exponent is the ratio of the absolute value of the temperature difference to the temperature sensitivity constant; the temperature difference is the difference between the current battery temperature and the reference temperature.

[0014] Furthermore, the budget tracking weight coefficient is dynamically adjusted based on the deviation between electricity price and remaining budget; the adjusted budget tracking weight coefficient is the product of the basic budget tracking weight, the electricity price sensitivity factor, and the deviation amplification factor. The electricity price sensitivity factor is the ratio of the real-time electricity price to the historical average electricity price, summed with 1; the deviation amplification factor is the ratio of the absolute value of the remaining deviation at the current moment to the rated power of the energy storage system, summed with 1.

[0015] Furthermore, the multi-constraint comprehensive judgment function includes four sub-constraints, including a SOC constraint indication function, a current constraint indication function, a voltage constraint indication function, and a temperature constraint indication function, all of which are Boolean type; Within the sliding time window, the ratio of the number of violations of each sub-constraint to the total number of samples is calculated, and the output of the sub-constraint is converted into a constraint violation severity index. The multi-constraint comprehensive judgment function is expressed as follows: ; in, , , and These are the severity indicators of constraint violations for the SOC constraint indicator function, current constraint indicator function, voltage constraint indicator function, and temperature constraint indicator function, respectively. By classifying the penalties according to the severity of the violation, the constraint violation penalty coefficient is obtained. .

[0016] The second aspect of this application provides an energy storage operation optimization system based on hierarchical multi-agent reinforcement learning, employing the energy storage operation optimization method as provided in the first aspect of this application. The system includes: The decision framework initialization module is used to initialize the hierarchical multi-agent reinforcement learning architecture. The upper-layer planning module includes the upper-layer planning agent and the lower-layer execution agent, and defines the inter-layer communication and synchronization mechanism. The upper-level strategy optimization module is used to build a quantitative model of the monetization cost of battery health status, design an upper-level reward function that balances electricity revenue and battery life loss cost, and optimize the upper-level planning strategy. The lower-level strategy optimization module designs the state space, action space, and reward function of the lower-level execution agent, constructs a lower-level reward function that integrates current smoothness and budget tracking accuracy, and optimizes the lower-level execution strategy. The hierarchical collaborative decision-making module is used to deploy the optimized upper-level planning strategy and lower-level execution strategy to the energy storage scheduling task. The upper-level planning agent generates the charging and discharging power budget during the day-ahead scheduling phase, and the lower-level execution agent receives the budget instructions and performs power adjustment.

[0017] The beneficial effects of this invention are that, compared with the prior art, 1. This application, through the innovative design of a hierarchical decision-making architecture, achieves multi-timescale coordination for energy storage system operation optimization, effectively solving the technical challenge of balancing short-term gains with long-term lifespan losses in traditional methods. The hierarchical decision-making framework decomposes the complex energy storage optimization problem into relatively independent but coordinated sub-problems, significantly reducing computational complexity and improving solution efficiency and real-time performance.

[0018] 2. Regarding battery life protection, this application innovatively introduces a monetization cost quantification mechanism for battery health status, transforming the abstract concept of lifespan loss into a concrete economic cost indicator. This allows the optimization algorithm to balance short-term electricity cost savings with long-term equipment depreciation within a unified cost framework. This quantification mechanism not only improves the accuracy of lifespan loss assessment but also provides a unified evaluation standard for comparing the economics of different types of energy storage devices.

[0019] 3. The multi-agent reinforcement learning framework of this application possesses excellent environmental adaptability and self-learning capabilities, enabling it to continuously optimize decision-making strategies in complex and ever-changing power grid operation environments. Compared to traditional rule-based or mathematical programming methods, reinforcement learning algorithms can automatically extract optimal strategies from historical operational experience, eliminating the need for manual parameter tuning and rule design. Furthermore, the multi-agent architecture allows agents at different levels to employ the most suitable algorithm type, fully leveraging the technical advantages of each algorithm.

[0020] 4. Regarding computational efficiency, the hierarchical architecture design of this application allows the upper-layer planning agent to make decisions on a longer time scale, while the lower-layer execution agent focuses on fine-grained control on a short time scale, avoiding the curse of dimensionality problem caused by mixed optimization of decision variables at different time scales in traditional methods. This design not only improves solution efficiency but also enhances the scalability and modularity of the system. Attached Figure Description

[0021] Figure 1 Implementation architecture diagram for optimizing energy storage lifespan. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The embodiments described in this application are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this invention.

[0023] As an example of this case, a specific implementation method and system for energy storage operation optimization based on hierarchical multi-agent reinforcement learning is described, with reference to... Figure 1 , Figure 1Implementation architecture diagram for optimizing energy storage operation system.

[0024] Step 1: System architecture initialization and agent design; 1.1: Initializing the Hierarchical Multi-Agent Reinforcement Learning System Architecture Construct a hierarchical multi-agent reinforcement learning architecture for energy storage lifetime optimization systems, including upper-layer planning agents. and lower-level execution agents The network structure. Specifically: 1.1.1: The upper-level planning agent adopts the ActorCritic architecture with the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm and is responsible for coordinating and making decisions on the day-ahead power budget of the energy storage system.

[0025] In the upper-level planning intelligent agent, This is the policy network for the upper-layer agents, with the upper-layer state as the input. The output is the power budget allocation strategy; For the value network of the upper-level intelligent agent, evaluate the expected reward value in a given state; 1.1.2: The lower-level execution agent adopts the dual-delay deep deterministic policy gradient algorithm (TD3) architecture and is responsible for the real-time power regulation and control of the energy storage system.

[0026] In the lower-level execution agent For the lower-level intelligent agent, the Actor network takes the lower-level state as input. The output is a deterministic power regulation action; The first Critic network for the lower-level agent evaluates the Q-values ​​of state-action pairs. The second Critic network for the lower-level agent forms a double-Q learning mechanism with the first Critic network. This is the parameter vector of the neural network, including the weight matrix and the bias vector. For a moment The state vector contains information about the current operating state of the energy storage system; For a moment The action vector represents the control action chosen by the agent in the current state.

[0027] 1.2: Establish inter-layer communication protocols and data interaction interfaces; Design an inter-layer communication architecture for intelligent agents in an energy storage system based on asynchronous message passing, and define a dedicated message format optimized for energy storage lifetime. Inter-layer messages Format definition: ; The parameters for the inter-layer message format are explained in Table 1 below.

[0028] Table 1. Parameter Explanation of Energy Storage System Message Format

[0029] Inter-layer communication mechanism: The upper-layer agent generates a power budget allocation vector every 24 hours (day-ahead) and sends it to the lower-layer agent via a message queue. The lower-layer agent performs real-time power adjustment every minute and feeds back the execution results to the upper-layer agent through a buffer mechanism, realizing the information transmission for hierarchical control of the energy storage system.

[0030] 1.3: Design an initialization strategy for the intelligent agent network parameters of the energy storage system; This embodiment is based on initializing the weights of the energy storage control neural network using the Xavier initialization method, and a dedicated initialization strategy is designed for the energy storage system. The specific improvements are as follows: For the output layer of the energy storage system strategy network, a smaller initialization variance is used ( This avoids overly aggressive initial power control strategies and prevents unnecessary damage to battery life.

[0031] Among them, soft update rate The value range is 0.001-0.01, and the value is designed to suit the long-term operation characteristics of energy storage systems. This ensures the stability of target network parameter updates and avoids drastic fluctuations in energy storage control strategies that could affect battery life.

[0032] This application constructs a hierarchical multi-agent reinforcement learning decision-making framework, decomposing the energy storage optimization problem into two layers: an upper planning layer and a lower execution layer. The upper planning agent is responsible for long-term energy storage charge-discharge budget allocation, employing a multi-agent near-end policy optimization algorithm with the battery health state monetization cost as the core objective function to formulate a day-ahead charge-discharge plan. The lower execution agent is responsible for real-time charge-discharge power control, employing a dual-delay deep deterministic policy gradient algorithm to achieve smooth control of the charge-discharge current while satisfying the upper-layer budget constraints.

[0033] Step 2: Upper-level planning and implementation of intelligent agents; 2.1: Define the state space, action space, and reward function of the upper-level planning agent; 2.1.1: State space of the upper-level planning agent exist State vector at time step Represented as:

[0034] in, , , These are the battery health status (values ​​range from [0,1], where 1 represents a brand new battery and 0 represents a completely failed battery), battery state of charge (values ​​range from [0,1], representing the ratio of current stored capacity to rated capacity), and ambient temperature (unit: degrees Celsius). , and They are time points arrive The 24-hour hourly load forecast (in kW), photovoltaic output forecast (in kW), and electricity price information (yuan / kWh).

[0035] As an optional approach in this embodiment, the battery state of health (SOH) can be determined by combining an online estimation method based on capacity decay and a frequency domain analysis method based on electrochemical impedance spectroscopy (EIS).

[0036] As an optional approach in this embodiment, the State of Charge (SOC) is obtained by: calculating the real-time SOC using the ampere-hour integration method, and then dynamically correcting the cumulative error of the ampere-hour integration using the extended Kalman filter method.

[0037] As an optional method in this embodiment, load forecast value The load forecasting model is obtained using a Long Short-Term Memory (LSTM) network combined with meteorological data; the load forecasting model is expressed as: ; in, For the first Load forecast at time of day Represents the LSTM model. for Historical load data for the 24 hours prior to the current time. and The first Forecast temperature and humidity values ​​at any given time; This is a weekday type.

[0038] As an optional method in this embodiment, photovoltaic power output prediction The method of obtaining satellite cloud imagery combined with meteorological data correction is employed. ; in, For the first Predicted photovoltaic output at any given time For photovoltaic installed capacity, For the first Satellite cloud image data at any given time. and They are respectively Predicted temperature and wind speed values ​​at any given time; For convolutional neural networks used to extract cloud map features, This is a correction factor for the attenuation of temperature and wind speed in numerical weather forecasts.

[0039] As an optional method in this embodiment, electricity price information A forecasting model based on real-time electricity prices and day-ahead electricity prices is used: ; in, for Forecasted electricity prices at any time Basic electricity price, For market supply and demand functions, As a policy adjustment factor; and express The electricity demand and supply at any given moment.

[0040] 2.1.2: Action Space of Upper-Level Intelligent Agents The feasible region is defined as the 24-hour charge / discharge power budget allocation vector:

[0041] in, No. The charging and discharging power estimate at any given time, with positive values ​​indicating discharging and negative values ​​indicating charging, and the unit is kW; and These are the maximum allowable discharge power and the maximum allowable charging power of the energy storage system, respectively. In this embodiment, the action of the upper-level planning agent represents the charging and discharging power budget allocated to the energy storage system.

[0042] 2.2: Establish a quantitative model for the monetization cost of battery health status, and design a reward function that comprehensively considers electricity revenue and battery life depreciation costs; ; in, This indicates that the upper-level intelligent agent is at time... The reward value obtained; time The revenue weighting is dynamically adjusted according to the battery health status; For a moment Lifetime loss weight; The time step is typically 1 hour; The cost function is the battery life loss rate. 2.2.1: Regarding return weights and lifetime loss weight A dynamic weight adaptive adjustment mechanism is adopted to achieve an effective balance between short-term gains and long-term lifespan loss: ; ; in, , These are weighting adjustment parameters, set to 2 and 3 respectively in this embodiment. When the battery health state is high (SOH is close to 1). Increased focus on short-term gains; when battery health is low (SOH close to 0). Increased size, with a greater focus on lifespan protection.

[0043] 2.2.2: Battery lifespan degradation cost function Defined as: ; In the formula, The basic coefficient of lifespan loss is related to battery temperature. and number of charge / discharge cycles Related, ,in, The basic loss coefficient is obtained through accelerated aging experiments; The activation energy was obtained by fitting aging data at different temperatures using the Arrhenius equation. The cyclic aging index was obtained through regression analysis of long-term cyclic experimental data. Let be the ideal gas constant. This is the rated cycle life.

[0044] This is the power influence factor. The power impact factor related to health status is as follows: the lower the state of health (SOH), the greater the impact of charge / discharge power on lifespan; in this implementation, The method is based on an online identification method for changes in battery internal resistance. SOC deviation penalty This is the temperature-dependent SOC deviation sensitivity coefficient; in this embodiment, This was obtained through thermodynamic modeling and experimental data fusion. This indicates the optimal state of charge (SOC) of the battery. Based on the electrochemical characteristics of lithium-ion batteries, the internal stress of the battery is minimized and the lifespan loss is lowest when the SOC is around 50%. This is a term related to the decline in health status. This is a health status penalty coefficient related to the number of cycles; in this embodiment, Time-varying parameter estimation based on historical data of battery capacity decay.

[0045] 2.3: Implementation of a multi-agent proximal policy optimization training algorithm for upper-layer planning agents. A distributed training architecture is adopted, with each agent maintaining an independent policy network and value network. Policy network updates utilize a pruning objective function.

[0046] This application establishes a dynamic tracking and monetization cost quantification model for battery health status. By monitoring key parameters such as battery capacity decay and internal resistance growth in real time, it constructs a multi-dimensional evaluation index system that reflects the true health status of the battery. It transforms battery life loss into quantifiable economic costs, establishing a precise mapping relationship between cycle number, depth of discharge, charge / discharge rate, and life loss, providing accurate cost references for upper-level planning and decision-making.

[0047] Step 3: Design and implementation of the lower-level executive agent; design the state space, action space, and reward function of the lower-level executive agent; 3.1: Design the state space of the lower-level executing agent ,but State vector at time step for: ; in, This represents the battery's state of charge, with a value range of [0,1]. This represents the battery current; a positive value indicates the discharge current, and a negative value indicates the charging current. Battery voltage; Battery temperature; For the upper-level power budget, a positive value indicates planned discharge, and a negative value indicates planned charging; The remaining budget deviation represents the cumulative budget execution deviation from the start of the current hour to the end of the current hour, calculated as follows: ; The formula means that, starting from the beginning of the current hour... up to the current moment middle, Power budget at any moment and actual execution power The cumulative deviation, in minutes. Weighted summation.

[0048] A positive value indicates that the actual performance is lower than the budget, and power needs to be increased; a negative value indicates that the actual performance is higher than the budget, and power needs to be reduced.

[0049] 3.2: Designing the action space of the lower-level execution agent The lower-level execution agent is in At any time, based on the state vector Selected action vector Real-time charging and discharging power adjustment ; ; In the formula, This represents the actual output power. The range of values ​​is , This is the upper limit for power adjustment; for real-time adjustment... Control is achieved through a comprehensive adjustment based on both the current deviation and the predicted deviation, expressed as: ; In the formula, The feedback control gain is used to correct existing deviations; in this embodiment, it is set to 0.6. This is the feedforward control gain, used to prevent future deviations; in this embodiment, it is set to 0.4. This is the natural power output predicted based on the current battery state.

[0050] As an optional approach in this embodiment, the power adjustment upper limit... It is necessary to simultaneously satisfy multiple constraints, including the maximum charge and discharge power constraint of the battery, the SOC safety boundary constraint, and the temperature protection constraint, and finally take the minimum value of the three as the global upper limit.

[0051] 3.3: Construct the reward function and current smoothing objective for the lower-level executive agent; Lower-level reward function The primary focus is on current smoothness and budget tracking accuracy. Current smoothness is represented by the square of the difference in battery current between adjacent time points; a larger squared difference indicates poorer current smoothness, and vice versa. Budget tracking accuracy is represented by the square of the remaining budget deviation; a larger squared deviation indicates poorer budget tracking accuracy, and vice versa. The lower-level reward function is expressed as: ; In the lower-level reward function: (1) The current smoothing weighting coefficient has the dimension of A. -2 Adaptive adjustment based on battery health status and temperature: ; In the formula, The base current smoothing weight; As a health condition regulating factor, more attention is paid to current smoothing during battery aging; This is a temperature adjustment factor that increases smoothing requirements when the temperature deviates from the optimal value.

[0052] (2) The weighting factor for budget tracking is in kW. -2 Dynamic adjustments based on electricity price and budget deviations:

[0053] In the formula, Track weights for the basic budget; Electricity price is a sensitive factor, and budget execution is given more attention when electricity prices are high; This is the deviation amplification factor; the larger the deviation, the heavier the penalty. For real-time electricity prices, The historical average electricity price This refers to the rated power of the energy storage system. (3) To constrain the penalty coefficient for violations, To constrain the severity of violations; The multi-constraint comprehensive judgment function includes four sub-constraints, and is expressed as follows: .

[0054] Each sub-constraint is a Boolean type, taking the value 0 or 1. Specifically: SOC constraint indicator function is ; The current constraint indication function is ; The voltage constraint indication function is ; Temperature constraint indication function is ; In the expressions of the four sub-constraints, This is a SOC (State of Charge) limit violation flag. and These are the maximum permissible SOC and the minimum permissible SOC, respectively. This is an overcurrent limit indicator. Maximum allowable current; This is a voltage over-limit indicator. and These are the maximum and minimum permissible voltages, respectively. This is a temperature limit violation indicator. and These are the maximum and minimum operating temperatures, respectively.

[0055] The sub-constraint indicator function is of Boolean type. In this embodiment, the discrete judgment is transformed into a constraint violation severity index by extending the time dimension. Specifically: within a sliding time window (e.g., 60 seconds, sampling once per second), calculate the proportion of violation time for each sub-constraint, that is, the ratio of the number of violations of each sub-constraint to the total number of samples.

[0056] The ratio is then converted into a specific numerical value for the constraint violation penalty coefficient through a tiered penalty system based on the degree of violation:

[0057] As an optional embodiment of this application, a battery state monitoring and safety assurance mechanism is provided to provide real-time and accurate battery state information to the upper-layer planning agent, while providing safety constraints and guarantees to the lower-layer execution agent. Specifically, a composite model based on cycle counting and calendar aging is used to update the battery's state of health (SOH) in real time, and the update result is provided as input to the battery life loss cost function in step 2.2.

[0058] A safety verification module for the execution agent is constructed to determine in real time whether the control command violates operational constraints such as voltage, power, and SOC, and to provide real-time feedback for the constraint violation indication function in step 3.2.

[0059] An extended Kalman filter is introduced to dynamically estimate the battery SOC, ensuring that the SOC-related terms of the reward function of the upper-level planning agent have practical reference value.

[0060] Deploy a battery temperature management subsystem to collect temperature information in real time and supplement the thermal state variables in the lower state space.

[0061] This embodiment deeply integrates existing battery management technology with reinforcement learning framework to ensure that the agent's decisions are based on accurate physical states and avoid control failures caused by state estimation errors.

[0062] As an optional embodiment of this application, a cascaded battery coordination control mechanism is provided. This mechanism is used in energy storage systems containing multiple cascaded battery modules with inconsistent health states to coordinate the power and safety control of each module, extend the overall system lifespan, and improve system energy efficiency and control stability. Specifically, the state space and control parameters constructed for a single intelligent agent in step 1 are expanded into a multi-module architecture, allowing each battery module to independently maintain its parameter vector, thus achieving personalized modeling of state descriptions and control strategies.

[0063] Based on the upper-level state space defined in step 2, indices such as SOC and SOH of multiple battery modules are introduced to form a joint state representation, thereby enhancing the policy network's adaptability to system heterogeneity.

[0064] In step 3, the lower-level execution agent outputs the total power of the system. This embodiment introduces a power decomposition mechanism to allocate control commands according to the module health and status, ensuring that the control commands are implemented at the physical level by module.

[0065] Based on the constraint mechanism in step 3, the constraint verification is extended to the module level, adding inter-module SOC equalization constraints and power distribution equalization requirements.

[0066] This embodiment can solve the problems of large differences and difficult control of cascaded battery modules in the prior art. It introduces a multi-module difference perception and collaborative regulation mechanism under the reinforcement learning architecture, realizes system-level coupling of battery state estimation, power control, constraint execution and health management, and effectively improves the compatibility and scheduling stability of energy storage system for cascaded batteries.

[0067] In the application of cascaded battery utilization, this application specifically develops a differentiated parameter adaptive mechanism, which can effectively address the technical challenge of inconsistent performance parameters of cascaded batteries. By establishing a personalized model parameter library and an online learning mechanism, the system can automatically identify and adapt to the characteristic differences of different battery modules, achieving coordinated and optimized control of cascaded battery packs.

[0068] As an optional embodiment of this application, a multi-source information fusion and prediction mechanism is employed to enhance the environmental perception capability of the energy storage system's intelligent agent, ensure the accuracy of predicted variables in the state space, and provide high-confidence decision-making basis for upper and lower layer agents. Specifically, key predicted variables are provided for the upper-layer state space constructed in step 2.1, including photovoltaic output, load change trends, and electricity price fluctuations, ensuring the accuracy and reliability of the predicted components. The level of prediction accuracy directly affects the calculation accuracy of the upper-layer reward function in step 2.2, thereby affecting the training stability and operational performance of the reinforcement learning strategy.

[0069] The forecast results can provide a basis for formulating the day-ahead charge and discharge power budget in step 3.1, enabling the system to operate with foresight. Forecast errors (such as photovoltaic and load forecast deviations) provide a correction reference for real-time power regulation in step 3.2, enhancing control robustness.

[0070] This application integrates a multi-source information fusion mechanism for grid load forecasting and energy storage state estimation, combining historical operating data, meteorological information, load characteristics, and other multi-dimensional information to construct a high-precision load forecasting model. Simultaneously, it establishes a multi-state online estimation algorithm for energy storage devices, updating key parameters such as battery state of charge, health status, and temperature status in real time, providing accurate state information input for intelligent agent decision-making.

[0071] As an optional embodiment of this application, a training algorithm is designed for the agent to improve the learning efficiency and control performance of the distributed agent in energy storage scheduling, ensuring that the agent can converge stably in complex environments and obtain a robust optimal policy. Specifically: for the two-layer agent architecture proposed in step 1, a collaborative training process is designed for the upper-layer multi-agent proximal policy optimization (MAPPO) algorithm and the lower-layer deterministic policy gradient (TD3) algorithm to ensure that the upper and lower layer policies improve synchronously at their respective scales.

[0072] Improve the asynchronous message passing mechanism in step 1.2 to increase the efficiency of training data synchronization between distributed agents and alleviate the impact of communication bottlenecks on the training process.

[0073] By combining the long-term planning characteristics of the upper-level decision-making agent in step 2 with the rapid response characteristics of the lower-level control agent in step 3, a global experience pool and a local experience pool are designed respectively to realize a hierarchical differentiated sample replay strategy and improve sample utilization efficiency.

[0074] To address the system security constraints involved in step 3.2, a penalty mechanism is introduced during the training process to enhance the policy's ability to perceive constraint boundaries and ensure the deployability and security of the training results.

[0075] By designing a provable safety constraint verification module, this application ensures that the energy storage system strictly adheres to safety boundaries under any operating conditions. Unlike the soft constraint or penalty function methods used in existing technologies, the safety verification mechanism in this application can mathematically prove the determinism of constraint satisfaction, eliminating the risk of constraint violation that may occur in traditional methods.

[0076] As an optional embodiment of this application, a system integration and performance evaluation mechanism is provided to integrate the multiple modules designed in all the above embodiments of this application, construct a complete energy storage lifetime optimization system, and comprehensively evaluate the system performance under a unified framework. Specifically: By integrating a two-layer intelligent agent architecture, state space modeling and hierarchical decision-making mechanism, state monitoring method, coordination and control strategy, prediction mechanism and training algorithm, an end-to-end intelligent energy storage system is constructed.

[0077] The design incorporates a performance evaluation system that includes indicators such as economy, reliability, response speed, and lifespan impact. This system is used to quantitatively assess the operational effectiveness of the proposed control strategies and provide a basis for decision-making in practical deployment.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for optimizing energy storage operation based on hierarchical multi-agent reinforcement learning, characterized in that, Initialize the hierarchical multi-agent reinforcement learning decision framework, with the upper-layer planning agent comprising the upper-layer planning agent and the lower-layer execution agent, and define the inter-layer communication mechanism; Construct a quantitative model of the monetization cost of battery health status, design an upper-level reward function that balances electricity revenue and battery life depreciation cost, and optimize the upper-level planning strategy. Design the state space, action space, and reward function of the lower-level executive agent, construct a lower-level reward function that integrates current smoothness and budget tracking accuracy, and optimize the lower-level execution strategy; Based on the optimized upper-layer planning strategy and lower-layer execution strategy, the energy storage scheduling task is executed. The upper-layer planning agent generates the charging and discharging power budget during the day-ahead scheduling phase, and the lower-layer execution agent receives the budget instructions and performs power adjustment.

2. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The upper-layer planning agent generates a power budget allocation vector and sends it to the lower-layer execution agent; the lower-layer execution agent performs real-time power adjustment and feeds back the execution result to the upper-layer planning agent. The inter-layer message format between the upper-layer planning agent and the lower-layer execution agent is defined as follows: ； in, For inter-layer messages, For the energy storage system's operating timestamp, Represents the agent identifier. Indicates the type of energy storage control message. For controlling the load of the energy storage system.

3. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The upper-level reward function is represented by the difference between the electricity revenue incentive term and the battery loss penalty term; The calculation method for the electricity revenue incentive is as follows: calculate the electricity revenue for a single time period based on the electricity price and charging / discharging power budget at each moment; sum up the revenue of all 24 time periods, and then multiply by the revenue weight to obtain the total electricity revenue. The battery loss penalty is calculated as follows: combining the charging and discharging power budget, battery state of charge, and battery health status at each moment, the loss cost for a single time period is calculated through the battery life loss cost function; the gains of all 24 time periods are summed up and multiplied by the life loss weight to obtain the cumulative loss cost. The State of Health (SOH) of the battery represents the degree of battery degradation.

4. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 3, characterized in that, For the yield weight and lifetime loss weight The balance relationship is achieved using a dynamic weight adaptive adjustment mechanism: ； ； In the formula, and For weight adjustment parameters, For a moment Battery health status.

5. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 3, characterized in that, The battery life loss cost function is the product of the life loss base coefficient, the power influence factor, the SOC deviation penalty, and the health state decay. The life loss baseline coefficient is the basic aging rate of the battery under battery temperature and historical cumulative cycle number. The power impact factor item has the absolute value of the charging and discharging power budget as the base and the index as the power impact coefficient related to the health status. The SOC deviation penalty term has the natural constant e as the base and the exponent is the product of the temperature sensitivity coefficient and the absolute SOC deviation; the absolute SOC deviation is the absolute value of the difference between the current power level and the optimal power level. The health status degradation term uses the remaining health loss of the battery as the base and the exponent as a health status penalty coefficient related to the number of charge-discharge cycles; the remaining health loss of the battery is expressed as... .

6. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The lower-level reward function is the sum of the current smoothing penalty, the budget tracking penalty, and the constraint violation penalty. The current smoothing penalty term is the product of the square of the current change value at adjacent time points and the current smoothing weight coefficient, and is negative. The budget tracking penalty term is the product of the square of the remaining deviation at the current time and the budget tracking weight coefficient, and is negative. The constraint violation penalty term is the product of the result of the multi-constraint comprehensive judgment function and the constraint violation penalty coefficient, and is negative.

7. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 6, characterized in that, The current smoothing weight coefficient is adaptively adjusted based on the battery health status and battery temperature; the adjusted current smoothing weight coefficient is the product of the base current smoothing weight, the health status adjustment factor, and the temperature adjustment factor. The health status regulation factor is represented as , The time is indicated; the temperature adjustment factor is based on the natural constant e, and the exponent is the ratio of the absolute value of the temperature difference to the temperature sensitivity constant; the temperature difference is the difference between the current battery temperature and the reference temperature.

8. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 6, characterized in that, The budget tracking weight coefficient is dynamically adjusted based on the deviation between electricity price and remaining budget; the adjusted budget tracking weight coefficient is the product of the basic budget tracking weight, the electricity price sensitivity factor, and the deviation amplification factor. The electricity price sensitivity factor is the ratio of the real-time electricity price to the historical average electricity price, summed with 1; the deviation amplification factor is the ratio of the absolute value of the remaining deviation at the current moment to the rated power of the energy storage system, summed with 1.

9. The energy storage operation optimization method based on hierarchical multi-agent reinforcement learning according to claim 6, characterized in that, The multi-constraint comprehensive judgment function includes four sub-constraints, including the SOC constraint indication function, the current constraint indication function, the voltage constraint indication function, and the temperature constraint indication function, all of which are Boolean type; Within the sliding time window, the ratio of the number of violations of each sub-constraint to the total number of samples is calculated, and the output of the sub-constraint is converted into a constraint violation severity index. The multi-constraint comprehensive judgment function is expressed as follows: ； in, , , and These are the severity indicators of constraint violations for the SOC constraint indicator function, current constraint indicator function, voltage constraint indicator function, and temperature constraint indicator function, respectively. By classifying the penalties according to the severity of the violation, the constraint violation penalty coefficient is obtained. .

10. An energy storage operation optimization system based on hierarchical multi-agent reinforcement learning, running the energy storage operation optimization method as described in any one of claims 1-9, characterized in that, The system includes: The decision framework initialization module is used to initialize the hierarchical multi-agent reinforcement learning architecture. The upper-layer planning module includes the upper-layer planning agent and the lower-layer execution agent, and defines the inter-layer communication and synchronization mechanism. The upper-level strategy optimization module is used to build a quantitative model of the monetization cost of battery health status, design an upper-level reward function that balances electricity revenue and battery life loss cost, and optimize the upper-level planning strategy. The lower-level strategy optimization module designs the state space, action space, and reward function of the lower-level execution agent, constructs a lower-level reward function that integrates current smoothness and budget tracking accuracy, and optimizes the lower-level execution strategy. The hierarchical collaborative decision-making module is used to deploy the optimized upper-level planning strategy and lower-level execution strategy to the energy storage scheduling task. The upper-level planning agent generates the charging and discharging power budget during the day-ahead scheduling phase, and the lower-level execution agent receives the budget instructions and performs power adjustment.

Images