Energy storage system control method based on deep reinforcement learning and adaptive predictive control

Abstract

The invention discloses an energy storage system control method based on deep reinforcement learning and adaptive prediction control, and relates to the technical field of energy storage systems, and the method comprises the steps: obtaining the operation data of an energy storage system, constructing a total cost function, and setting the constraint condition of the energy storage system; constructing a hybrid control algorithm based on deep reinforcement learning and adaptive prediction control, and making a decision; formulating a convergence condition and an emergency response strategy according to the decision; and optimizing the hybrid control algorithm to realize safety optimization of the energy storage system. The invention provides a hybrid control algorithm, and efficient scheduling and safety control of the energy storage system are realized by using the synergistic effect of deep reinforcement learning and adaptive prediction control.

Description

Technical Field

[0001] The present invention relates to the technical field of energy storage systems, and in particular to an energy storage system control method based on deep reinforcement learning and adaptive predictive control. Background Art

[0002] With the increase in the proportion of renewable energy power generation, the volatility and instability of the power grid have increased significantly; as a balancing tool in the power system, the energy storage system can alleviate load fluctuations and promote the use of renewable energy; however, traditional control methods are mostly centered on a single objective function, and it is difficult to balance multiple constraints in practical applications, especially when the battery is frequently charged and discharged, temperature and SOC state of charge factors will affect the safety and life of the battery; in addition, existing methods are usually difficult to deal with emergencies and lack flexible response capabilities. Therefore, it is particularly necessary to propose a safe optimization control method based on dynamic programming.

[0003] Existing energy storage system control methods have the following defects: First, power scheduling lacks precision, and traditional methods mostly use static optimization strategies, which are difficult to adapt to the needs of nonlinear fluctuations in the power grid; second, there is a lack of comprehensive management of temperature and SOC, and most existing methods ignore the long-term impact of temperature and SOC on battery health; in addition, the ability to cope with emergencies is insufficient, and traditional methods find it difficult to quickly respond to sudden changes in load or fluctuations in renewable energy power generation, which may cause grid frequency instability or overload of the energy storage system. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to provide a hybrid algorithm control framework, which combines the adaptability of deep reinforcement learning with the foresight of adaptive predictive control, so that the energy storage system can automatically adjust its charging and discharging strategy when the power demand changes and the system state fluctuates, thereby achieving optimal cost management and safety assurance.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides a method for controlling an energy storage system based on deep reinforcement learning and adaptive predictive control, which includes obtaining operating data of the energy storage system, constructing a total cost function and setting constraints for the energy storage system; constructing a hybrid control algorithm based on deep reinforcement learning and adaptive predictive control to make decisions; formulating convergence conditions and emergency response strategies based on the decisions; and optimizing the hybrid control algorithm to achieve safe optimization of the energy storage system.

[0008] As an optimal solution of the energy storage system control method based on deep reinforcement learning and adaptive predictive control described in the present invention, the operating data of the energy storage system includes historical load data, temperature change data and SOC change data; the construction of the total cost function refers to the minimum operating cost of the energy storage system as the goal, covering the operating cost, temperature management cost and SOC penalty cost of the energy storage system.

[0009] As a preferred solution of the energy storage system control method based on deep reinforcement learning and adaptive predictive control described in the present invention, the calculation formula of the total cost function is as follows:

[0010]

[0011] C op (t) = a·P(t) 2 +b·P(t)+c

[0012] C temp (t) = λ·(T(t)-T ref ) 2

[0013] C soc (t) = μ·(SOC(t)-SOC ref ) 2

[0014] C adaptive (t)=θ·Var(ΔP(t),ΔT(t),ΔSOC(t))

[0015] Among them, C total is the total cost function of the energy storage system; C op (t) is the basic operating cost of the energy storage system at time t; C temp (t) is the temperature management cost; C soc (t) is the SOC penalty cost; C adaptive (t) is the adaptive control cost; a, b and c are the coefficients of the energy storage system operation cost; λ is the weight coefficient of the temperature management cost; μ is the weight coefficient of the SOC penalty cost; T ref is the safety value of the reference temperature; SOC ref is the safety value of the reference SOC; Var(·) is the variance of the system state change rate, which serves as a penalty term for the adjustment range of the regulation strategy. The parameter θ adjusts the weight of the adaptive control cost in the total cost.

[0016] As a preferred solution of the energy storage system control method based on deep reinforcement learning and adaptive predictive control described in the present invention, the constraint conditions include power constraint, temperature dynamic adjustment constraint and SOC interval dynamic constraint; the power constraint refers to setting the upper and lower limits of power to prevent instantaneous power overload from causing battery overheating or shortening of life. The specific formula is as follows:

[0017] P min ≤P(t)≤P max

[0018] Where, P(t) is the instantaneous power; P min and P max The power upper and lower limits are as follows: The temperature dynamic adjustment constraint refers to automatically adjusting the power output based on the real-time temperature change to ensure that the temperature is within the preset range. The specific formula is as follows:

[0019] ΔT(t)=α·(P(t)-P(t-1)),T min ≤T(t)≤T max

[0020] Wherein, T(t) is the real-time temperature; the SOC range dynamic constraint refers to preventing the battery from being damaged due to over-discharge or over-charge by means of the SOC upper and lower limit constraints, and ensuring that the SOC of the system remains within a reasonable range during operation. The specific formula is as follows:

[0021] SOC min ≤SOC(t)≤SOC max

[0022] Wherein, SOC(t) is the SOC during operation.

[0023] As a preferred solution of the energy storage system control method based on deep reinforcement learning and adaptive predictive control described in the present invention, wherein: the hybrid control algorithm refers to the coordinated work of deep reinforcement learning DRL and adaptive predictive control APC, and the use of the adaptability of deep reinforcement learning DRL and the predictiveness of adaptive predictive control APC enables the system to make optimal decisions in a dynamic environment; the deep reinforcement learning DRL is used to train an intelligent agent that can learn charging and discharging strategies, learn historical data through a reinforcement learning model, and guide real-time decision-making, including the following steps: constructing a simulation environment by setting the state space and action space of the system; defining a reward function, using the reduction of the total cost as the reward signal, that is, the system is within the safety range of temperature and SOC. The system maintains the minimum cost within the range; iterative optimization is performed through the DRL model so that the model converges to the optimal strategy after multiple rounds of learning; the adaptive predictive control is used to adjust the power output in real time and to roll forward the prediction of future power demand and system status to provide forward-looking decision support, including the following steps: forecasting future loads based on the time series analysis method; formulating a rolling optimization mechanism to update the power dispatch decision immediately after executing the prediction results in each time period; adjusting the current power output according to the prediction error and the output of the DRL model; balancing the short-term and long-term costs by introducing a discount factor, calculating the cost of the future time period using the expected value, and constructing the state transfer equation of the hybrid control algorithm. The specific formula of the state transfer equation of the hybrid control algorithm is as follows:

[0024] C total (t) = min P(t) (C op (t)+C temp (t)+C soc (t)+C adaptive (t)+γ·E P(t+1) [C total (t+1)])

[0025] Among them, C total is the total cost function of the energy storage system; γ is the discount factor.

[0026] As a preferred solution of the energy storage system control method based on deep reinforcement learning and adaptive predictive control described in the present invention, the convergence condition refers to the convergence condition defined in the solution process. When the energy storage system meets the following conditions in all time periods, the corresponding strategy is determined to be the optimal strategy:

[0027] |T(t)-T ref |<ò T

[0028] |SOC(t)-SOC ref |<ò SOC

[0029] |ΔP(t)|<ò P

[0030] Among them, T is the temperature tolerance value; SOC Tolerance value of SOC; P is the power tolerance value.

[0031] As a preferred solution of the energy storage system control method based on deep reinforcement learning and adaptive predictive control described in the present invention, the emergency response strategy includes a quick response mode, a temperature priority mode and a strategy switching; the quick response mode means that when the load suddenly rises, the system will temporarily increase the discharge power to maintain the frequency stability of the power grid. This adjustment is quickly decided by the deep reinforcement learning DRL module, and the increase in discharge power is judged based on the difference between past load data and current power output; the temperature priority mode includes: in an emergency, abnormal temperature rise may threaten the safety of the energy storage system, and the hybrid control algorithm will give priority to protecting the temperature safety of the system, automatically reduce the charging and discharging power, or even stop discharging to avoid temperature exceeding the limit; after the temperature gradually returns to the normal range, the system will gradually restore normal charging and discharging operations through rolling prediction; the strategy switching means that when an emergency occurs, the deep reinforcement learning DRL module automatically switches to the emergency mode according to the trained strategy, and calls a special operation sequence when the environment suddenly changes to avoid chain reactions caused by emergencies, and flexibly adapts to various emergency situations.

[0032] On the second aspect, in order to further solve the safety problems existing in the energy storage system, the present invention provides an energy storage system control system based on deep reinforcement learning and adaptive predictive control, which includes: a cost construction module, which is used to obtain the operating data of the energy storage system, construct a total cost function, and set the power constraints, temperature dynamic adjustment constraints and SOC interval dynamic constraints of the energy storage system; a decision-making module, which is used to construct a hybrid control algorithm based on deep reinforcement learning and adaptive predictive control and make optimal decisions; a decision convergence module, which is used to use convergence conditions to judge the effectiveness of the optimal decision and formulate an emergency response strategy; a safety optimization module, which is used to optimize the hybrid control algorithm.

[0033] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the energy storage system control method based on deep reinforcement learning and adaptive predictive control as described in the first aspect of the present invention is implemented.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the energy storage system control method based on deep reinforcement learning and adaptive predictive control as described in the first aspect of the present invention is implemented.

[0035] Beneficial effects of the present invention: The present invention proposes a hybrid control algorithm, which realizes efficient scheduling and safe control of the energy storage system by utilizing the synergistic effect of deep reinforcement learning and adaptive predictive control; utilizing the adaptability of deep reinforcement learning, the energy storage system can quickly adjust the charging and discharging power under different load fluctuations to achieve rapid response to the dynamic environment; through the short-term load prediction of the adaptive predictive control APC module, the system can predict the power demand in advance and adjust the power output at the current moment, thereby improving the scheduling flexibility of the system, so that the system not only adapts to the current environment, but also has the ability of forward-looking control; the multi-layer constraints and emergency response mechanism in the system ensure the stability and safety of the energy storage system in emergency situations such as sudden load increase and temperature fluctuation, and the temperature priority mode and strategy switching mechanism provide the system with more flexible safety protection in extreme environments; the hybrid control algorithm realizes the dual optimization of safety and economy by incorporating the penalty costs of power, temperature and SOC into the total cost function, and optimizes the operating cost of the system while ensuring battery life and system safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0037] Figure 1 This is an overall flow chart of the energy storage system control method based on deep reinforcement learning and adaptive predictive control in Example 1.

[0038] Figure 2 This is a schematic diagram of the structure of the computer device in Example 3.

[0039] Figure 3 This is a schematic diagram of the total cost changing over time in Example 4.

[0040] Figure 4 Schematic diagram of SOC variation over time in Example 4. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0043] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0044] Example 1

[0045] Reference Figure 1 , which is the first embodiment of the present invention, provides an energy storage system control method based on deep reinforcement learning and adaptive predictive control.

[0046] The existing energy storage system control methods have the following main problems: First, power scheduling lacks precision, and traditional methods mostly use static optimization strategies, which are difficult to adapt to the needs of nonlinear fluctuations in the power grid; second, there is a lack of comprehensive management of temperature and SOC, and most existing methods ignore the long-term impact of temperature and SOC on battery health; in addition, the ability to cope with emergencies is insufficient, and traditional methods find it difficult to quickly respond to sudden changes in load or fluctuations in renewable energy power generation, which may cause grid frequency instability or overload of the energy storage system.

[0047] The present application provides an effective solution to the above-mentioned problems. Next, a plurality of embodiments will be combined to explain in detail how to implement the energy storage system control method based on deep reinforcement learning and adaptive predictive control.

[0048] Figure 1 The overall flow chart of the energy storage system control method based on deep reinforcement learning and adaptive predictive control is shown, including:

[0049] S1: Obtain the operating data of the energy storage system, construct the total cost function and set the constraints of the energy storage system.

[0050] Preferably, the operating data of the energy storage system includes historical load data, temperature change data and SOC change data.

[0051] Preferably, constructing a total cost function means balancing economy and safety with the minimum operating cost of the energy storage system as the goal, covering the operating cost, temperature management cost and SOC penalty cost of the energy storage system. The total cost function decomposes the cost components and controls the impact of the costs item by item to ensure that the system achieves the best balance between stability and cost.

[0052] Specifically, the calculation formula of the total cost function is as follows:

[0053]

[0054] C op (t) = a·P(t) 2 +b·P(t)+c

[0055] C temp (t) = λ·(T(t)-T ref ) 2

[0056] C soc (t) = μ·(SOC(t)-SOC ref ) 2

[0057] C adaptive (t)=θ·Var(ΔP(t),ΔT(t),ΔSOC(t))

[0058] Among them, C total is the total cost function of the energy storage system; C op C (t) is the basic operating cost of the energy storage system at time t, which mainly includes the charging and discharging costs of the battery. This cost item is mainly used to reflect the energy consumption of the energy storage system to maintain the load balance of the power grid; temp (t) is the temperature management cost, which is used to control the temperature of the energy storage battery to operate within a safe range. Excessive temperature may cause battery performance degradation and even lead to safety risks. Therefore, this cost item can effectively reflect the increased cost when the temperature deviates from the safe value; C soc (t) is the SOC penalty cost, i.e., the state of charge penalty cost, which ensures that the SOC is maintained within a reasonable range and avoids damage to the battery due to excessive charge and discharge; C adaptive (t) is the adaptive control cost, which indicates the magnitude of the system adjustment strategy and controls the rapidly changing power output; a, b and c are the coefficients of the energy storage system operation cost, which indicate the degree of influence of the charging and discharging power on the system cost; λ is the weight coefficient of the temperature management cost, which is used to adjust the influence of the temperature management cost on the total system cost; μ is the weight coefficient of the SOC penalty cost, which is used to adjust the influence of the SOC penalty cost on the total system cost; T ref is the safety value of the reference temperature; SOC refis the safety value of the reference SOC; Var(·) is the variance of the system state change rate, which is used as a penalty term for the adjustment range of the regulation strategy. The parameter θ adjusts the weight of the adaptive control cost in the total cost. The total cost function controls the temperature management cost C temp (t) and SOC penalty cost C soc (t) Avoid the situation where the system temperature is too high or the SOC is too low during operation, thereby improving the safety and economy of the system.

[0059] Furthermore, constraints mean that in order to ensure the safe operation of the energy storage system, multiple physical constraints need to be defined to keep the energy storage battery within a reasonable power output, temperature and SOC range. These constraints ensure that the system can operate stably under extreme conditions and ensure that the energy storage system does not exceed the physical limits of the battery or damage the performance of the battery during normal operation.

[0060] Preferably, the constraint conditions include power constraint, temperature dynamic adjustment constraint and SOC interval dynamic constraint.

[0061] Specifically, power constraint refers to setting upper and lower limits of power to prevent instantaneous power overload from causing battery overheating or shortening of life. The specific formula is as follows:

[0062] P min ≤P(t)≤P max

[0063] Where, P(t) is the instantaneous power; P min and P max The upper and lower limits of power.

[0064] Specifically, the temperature dynamic adjustment constraint refers to automatically adjusting the power output based on real-time temperature changes to ensure that the temperature is within the preset range. The specific formula is as follows:

[0065] ΔT(t)=α·(P(t)-P(t-1)),T min ≤T(t)≤T max

[0066] Wherein, T(t) is the real-time temperature.

[0067] Specifically, the dynamic constraint of the SOC range refers to preventing the battery from being damaged due to over-discharge or over-charge by constraining the upper and lower limits of the SOC, and ensuring that the SOC of the system remains within a reasonable range during operation. The specific formula is as follows:

[0068] SOC min ≤SOC(t)≤SOC max

[0069] Wherein, SOC(t) is the SOC during operation.

[0070] Preferably, by constructing a comprehensive total cost function, the problem of how to balance economy and safety in the energy storage system is solved. Different from the prior art that only considers the operating cost of the power system, the present invention also considers the management of battery temperature and SOC state, which not only avoids battery damage caused by excessive temperature or excessive discharge, but also ensures that the operating cost of the system is effectively controlled.

[0071] S2: Build a hybrid control algorithm based on deep reinforcement learning and adaptive predictive control to make decisions.

[0072] Preferably, the hybrid control algorithm refers to the collaborative work of deep reinforcement learning DRL and adaptive predictive control APC, utilizing the adaptability of deep reinforcement learning DRL and the predictiveness of adaptive predictive control APC, so that the system can make optimal decisions in real time in a dynamic environment.

[0073] Specifically, deep reinforcement learning DRL is used to train an intelligent agent that can learn charging and discharging strategies. It learns historical data through a reinforcement learning model (such as a deep Q network or a deep deterministic policy gradient algorithm) to guide real-time decision-making, including the following steps: construct a simulation environment by setting the state space and action space of the system, where the state space includes power, temperature, and SOC, and the action space includes power increase and power decrease.

[0074] The reward function is defined with the reduction of total cost as the main reward signal, and the reward is higher when the system maintains the minimum cost within the safe range of temperature and SOC.

[0075] Iterative optimization is performed through the DRL model so that the model converges to the optimal strategy after multiple rounds of learning, where the DRL model is a deep Q network DQN model or a deep deterministic policy gradient algorithm DDPG model.

[0076] Specifically, adaptive predictive control is used to adjust power output in real time and make rolling forecasts of future power demand and system status to provide forward-looking decision support, including the following steps: predicting future loads based on a time series analysis method, where the time series analysis method is an LSTM neural network or an ARIMA model.

[0077] A rolling optimization mechanism is developed to update the power scheduling decision immediately after executing the forecast results in each time period.

[0078] According to the prediction error and the output of the DRL model, the current power output is adjusted to ensure the accuracy and stability of real-time control.

[0079] Preferably, by introducing a discount factor to balance short-term and long-term costs, using the expected value to calculate the cost of the future time period, the state transfer equation of the hybrid control algorithm is constructed. The specific formula of the state transfer equation of the hybrid control algorithm is as follows:

[0080] C total (t) = min P(t) (C op (t)+C temp (t)+C soc (t)+C adaptive (t)+γ·E P(t+1) [C total (t+1)])

[0081] Among them, C total is the total cost function of the energy storage system; γ is the discount factor, which is used to balance short-term and long-term costs. The cost of the future time period t+1 is calculated by the expected value, thereby providing forward-looking support for current decisions.

[0082] Preferably, a hybrid control algorithm is constructed by combining deep reinforcement learning and adaptive predictive control. Compared with traditional predictive control methods, the introduction of deep reinforcement learning enables the system to make optimal decisions in real time under dynamic environments, significantly improving the adaptability and decision-making efficiency of the energy storage system. Especially when the power demand fluctuates greatly, DRL can optimize the charging and discharging strategy by learning historical load data and battery status, thereby reducing the overall operating cost and avoiding safety risks of excessive temperature or too low SOC. The APC module provides forward-looking decision support by predicting future load and temperature changes in real time, enabling the system to adjust power output in advance within a period of time in the future to avoid the impact of emergencies on system performance. It can effectively balance short-term and long-term costs, allowing the energy storage system to maintain stable operation in a complex and changing environment.

[0083] S3: Formulate convergence conditions and emergency response strategies based on decisions.

[0084] Preferably, the convergence condition refers to defining the convergence condition in the solution process in order to determine the validity of the optimal solution. When the energy storage system meets the following conditions in all time periods, the corresponding strategy is determined to be the optimal strategy:

[0085] |T(t)-T ref |<ò T

[0086] |SOC(t)-SOC ref |<ò SOC

[0087] |ΔP(t)|<ò P

[0088] Among them, T is the temperature tolerance value; SOC Tolerance value of SOC; P is the power tolerance value, which is used to control the system to achieve the optimal solution during dynamic adjustment.

[0089] Specifically, the emergency response strategy refers to the fact that during the actual operation of the energy storage system, emergencies (such as a sudden increase in grid load or a sharp change in ambient temperature) may cause the energy storage system to withstand high power output or exceed the safe temperature range in a short period of time. Therefore, the hybrid control algorithm designs a multi-level emergency response mechanism to enable the system to quickly restore stability in an emergency situation.

[0090] Preferably, the emergency response strategy includes a quick response mode, a temperature priority mode and a strategy switching.

[0091] Specifically, the fast response mode means that when the load suddenly increases, the system will temporarily increase the discharge power to maintain the frequency stability of the power grid. This adjustment is quickly decided by the deep reinforcement learning DRL module, which judges the increase in discharge power based on the difference between past load data and current power output. The fast response mode effectively improves the flexibility of the energy storage system and enhances the load tracking capability without affecting SOC and temperature safety.

[0092] Specifically, the temperature priority mode includes: in emergency situations, abnormal temperature rise may threaten the safety of the energy storage system. The hybrid control algorithm will prioritize protecting the system's temperature safety, automatically reduce the charging and discharging power, or even stop discharging to avoid temperature exceeding the limit.

[0093] After the temperature gradually returns to the normal range, the system will gradually resume normal charging and discharging operations through rolling prediction. The temperature priority mode mainly monitors temperature data in real time through adaptive predictive control (APC) to ensure that temperature fluctuations are within a reasonable range.

[0094] Specifically, strategy switching means that when an emergency occurs, the deep reinforcement learning DRL module automatically switches to emergency mode according to the trained strategy. This strategy calls a special operation sequence when the environment changes suddenly to avoid chain reactions caused by emergencies. Strategy switching can also flexibly adapt to a variety of emergency situations, such as giving priority to ensuring grid stability when the load is too high or giving priority to ensuring the safety of energy storage system equipment when the temperature exceeds the limit.

[0095] Preferably, the stability problem of the energy storage system in emergency situations is effectively solved by utilizing an emergency response strategy. Different from the single emergency response mechanism commonly used in the prior art, the present invention enables the system to flexibly adapt to different emergencies by switching between different strategies. For example, when the load increases sharply, the system can quickly adjust the discharge power; when the temperature rises abnormally, the system will give priority to ensuring temperature safety and avoid battery damage by reducing power output. This flexible response mechanism significantly improves the reliability and safety of the energy storage system.

[0096] S4: Optimize the hybrid control algorithm to achieve safe optimization of the energy storage system.

[0097] Preferably, optimizing the hybrid control algorithm includes the following steps: in the daily operation of the energy storage system, obtaining the system's historical load data, temperature change, and SOC change data, and organizing the data into input samples for model training through data cleaning and feature extraction.

[0098] Based on the collected data, multiple rounds of training are performed using a deep reinforcement learning model (such as the DQN model or the DDPG model). The reward function in the training process aims to reduce the total cost while taking into account multiple safety constraints. Through the training of the deep learning model, a policy network that can quickly adapt to different environments is obtained.

[0099] The adaptive predictive control (APC) module uses the long short-term memory (LSTM) neural network to predict future power demand and temperature changes, and uses the predicted values ​​as an important reference for the current control strategy. During the actual operation of the system, the adaptive predictive control (APC) module will continuously adjust the prediction parameters to maintain the accuracy of the prediction results.

[0100] The DRL strategy network is combined with the APC control strategy to form a complete hybrid control solution, in which the power output and temperature management strategies are continuously optimized during system operation through the intelligent decision-making of DRL and the predictive feedback of APC.

[0101] After the energy storage system is put into operation, the system will monitor the power, temperature and SOC changes in real time through sensors, and feed the monitoring data back to the control center. When it is detected that the power, temperature or SOC deviates from the safe range, the hybrid control system will trigger the corresponding adjustment mechanism, such as fast response mode, temperature priority mode or strategy switching.

[0102] It should be noted that the hybrid control algorithm proposed in the present invention can be widely used in various complex power grid scenarios. First, for high-load power grid scenarios, such as in power systems with large load fluctuations, the rapid response capability of the energy storage system is crucial. The hybrid control algorithm can dynamically adjust the charging and discharging power of the energy storage system according to load changes to avoid power grid instability caused by load overload; secondly, for renewable energy access scenarios, such as wind power, photovoltaics and other renewable energy power generation scenarios with strong volatility, the energy storage system can be used as a tool to balance power generation fluctuations. The hybrid control algorithm of the present invention can predict power demand through APC when wind power or photovoltaics fluctuates, and adjust the power output of the energy storage system through DRL to achieve effective balance of renewable energy fluctuations; in addition, in microgrids or independent power supply systems, the energy storage system is the main power dispatching equipment. Through the hybrid control algorithm, the energy storage system can provide efficient and reliable power dispatching in the microgrid to ensure the stability and economy of the power supply system; finally, in power peak shaving and valley filling, by using the hybrid control algorithm, the energy storage system can release power during peak power consumption and store power during low power consumption, thereby achieving the function of peak shaving and valley filling, reducing the peak-to-valley difference of the power grid and improving the operating efficiency of the overall power system.

[0103] In summary, the present invention proposes a hybrid control algorithm, which realizes efficient scheduling and safe control of energy storage systems by utilizing the synergistic effect of deep reinforcement learning and adaptive predictive control; utilizing the adaptability of deep reinforcement learning, the energy storage system can quickly adjust the charging and discharging power under different load fluctuations to achieve rapid response to dynamic environments; through the short-term load prediction of the adaptive predictive control APC module, the system can predict the power demand in advance and adjust the power output at the current moment, thereby improving the scheduling flexibility of the system, so that the system not only adapts to the current environment, but also has the ability of forward-looking control; the multi-layer constraints and emergency response mechanisms in the system ensure the stability and safety of the energy storage system in emergency situations such as sudden load increase and temperature fluctuation, and the temperature priority mode and strategy switching mechanism provide the system with more flexible safety protection in extreme environments; the hybrid control algorithm realizes the dual optimization of safety and economy by incorporating the penalty costs of power, temperature and SOC into the total cost function, and optimizes the operating cost of the system while ensuring battery life and system safety.

[0104] Embodiment 2 is an embodiment of the present invention, which provides an energy storage system control system based on deep reinforcement learning and adaptive predictive control, including: a cost construction module, which is used to obtain the operating data of the energy storage system, construct a total cost function, and set the power constraints, temperature dynamic adjustment constraints and SOC interval dynamic constraints of the energy storage system; a decision-making module, which is used to construct a hybrid control algorithm based on deep reinforcement learning and adaptive predictive control and make optimal decisions; a decision convergence module, which is used to use convergence conditions to judge the effectiveness of the optimal decision and formulate an emergency response strategy; a safety optimization module, which is used to optimize the hybrid control algorithm.

[0105] Embodiment 3 is an embodiment of the present invention, which is different from the previous embodiment in that:

[0106] like Figure 2 As shown, if the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0107] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0108] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0109] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0110] Example 4 is an embodiment of the present invention, which provides a method for controlling an energy storage system based on deep reinforcement learning and adaptive predictive control. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0111] This example simulates the operation of the method of the present invention and the traditional method in the energy storage system, collects historical load data, temperature change data and SOC change data from the energy storage system, and constructs the total cost function of the energy storage system, including operation cost, temperature management cost, SOC penalty cost and adaptive control cost. By reasonably setting the weight coefficient of each cost, the system can minimize the economic cost while ensuring safety, and obtain the relationship between the total cost of the method of the present invention and the traditional method and the change of SOC data over time in real time, such as Figure 3 and Figure 4 Shown is a schematic diagram of total cost and SOC changing over time.

[0112] from Figure 3 and Figure 4It can be seen that in terms of total cost, the present invention incorporates the penalty costs of power, temperature and SOC into the total cost function by combining deep reinforcement learning with adaptive predictive control. Compared with traditional technologies, the present invention achieves dual optimization of safety and economy and optimizes the operating cost of the system. In terms of SOC, the present invention ensures the stability and safety of the energy storage system in sudden situations such as sudden load increase and temperature fluctuation by utilizing multi-layer constraints and emergency response mechanisms.

[0113] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A control method for an energy storage system based on deep reinforcement learning and adaptive predictive control, characterized in that: include: Obtain the operating data of the energy storage system, construct the total cost function and set the constraints of the energy storage system; Build hybrid control algorithms based on deep reinforcement learning and adaptive predictive control to make decisions; formulating convergence conditions and emergency response strategies based on said decisions; The hybrid control algorithm is optimized to achieve safe optimization of the energy storage system.

2. The energy storage system control method based on deep reinforcement learning and adaptive predictive control according to claim 1, characterized in that: The operation data of the energy storage system includes historical load data, temperature change data and SOC change data; The construction of the total cost function refers to taking the minimum operating cost of the energy storage system as the goal, covering the operating cost, temperature management cost and SOC penalty cost of the energy storage system.

3. The energy storage system control method based on deep reinforcement learning and adaptive predictive control according to claim 2, characterized in that: The calculation formula of the total cost function is as follows: C op (t)=a·P(t) 2 +b·P(t)+c C temp (t)=λ·(T(t)-T ref ) 2 C soc (t)=μ·(SOC(t)-SOC ref ) 2 C adaptive (t)=θ·Var(ΔP(t),ΔT(t),ΔSOC(t)) Among them, C total is the total cost function of the energy storage system; C op (t) is the basic operating cost of the energy storage system at time t; C temp (t) is the temperature management cost; C soc (t) is the SOC penalty cost; C adaptive (t) is the adaptive control cost; a, b and c are the coefficients of the energy storage system operation cost; λ is the weight coefficient of the temperature management cost; μ is the weight coefficient of the SOC penalty cost; T ref is the safety value of the reference temperature; SOC ref is the safety value of the reference SOC; Var(·) is the variance of the system state change rate, which serves as a penalty term for the adjustment range of the regulation strategy. The parameter θ adjusts the weight of the adaptive control cost in the total cost.

4. The energy storage system control method based on deep reinforcement learning and adaptive predictive control according to claim 3, characterized in that: The constraints include power constraints, temperature dynamic adjustment constraints and SOC interval dynamic constraints; The power constraint refers to setting the upper and lower limits of power to prevent instantaneous power overload from causing battery overheating or shortening of life. The specific formula is as follows: P min ≤P(t)≤P max Where, P(t) is the instantaneous power; P min and P max are the upper and lower limits of power; The temperature dynamic adjustment constraint refers to automatically adjusting the power output based on real-time temperature changes to ensure that the temperature is within a preset range. The specific formula is as follows: ΔT(t)=α·(P(t)-P(t-1)),T min ≤T(t)≤T max Wherein, T(t) is the real-time temperature; The SOC range dynamic constraint refers to preventing the battery from being damaged due to over-discharge or over-charge by constraining the upper and lower limits of the SOC, and ensuring that the SOC of the system remains within a reasonable range during operation. The specific formula is as follows: SOC min ≤SOC(t)≤SOC max Wherein, SOC(t) is the SOC during operation.

5. The energy storage system control method based on deep reinforcement learning and adaptive predictive control according to claim 4, characterized in that: The hybrid control algorithm refers to the collaborative work of deep reinforcement learning DRL and adaptive predictive control APC, which utilizes the adaptability of deep reinforcement learning DRL and the predictability of adaptive predictive control APC to enable the system to make optimal decisions in a dynamic environment; The deep reinforcement learning (DRL) is used to train an intelligent agent that can learn charging and discharging strategies, learn historical data through a reinforcement learning model, and guide real-time decision-making, including the following steps: Construct a simulation environment by setting the state space and action space of the system; Define a reward function, with the reduction of total cost as the reward signal, that is, the system maintains the minimum cost within the safe range of temperature and SOC; Iterative optimization is performed through the DRL model, so that the model converges to the optimal strategy after multiple rounds of learning; The adaptive predictive control is used to adjust power output in real time and to make rolling predictions of future power demand and system status to provide forward-looking decision support, including the following steps: Forecast future load based on time series analysis method; Develop a rolling optimization mechanism to update the power scheduling decision immediately after executing the forecast results in each time period; Adjust the current power output according to the prediction error and the output of the DRL model; By introducing a discount factor to balance short-term and long-term costs, and using the expected value to calculate the cost of the future time period, the state transfer equation of the hybrid control algorithm is constructed. The specific formula of the state transfer equation of the hybrid control algorithm is as follows: C total (t)=min P(t) (C op (t)+C temp (t)+C soc (t)+C adaptive (t)+γ·E P(t+1) [C total (t+1)]) Among them, C total is the total cost function of the energy storage system; γ is the discount factor.

6. The energy storage system control method based on deep reinforcement learning and adaptive predictive control according to claim 5, characterized in that: The convergence condition refers to the convergence condition defined in the solution process. When the energy storage system meets the following conditions in all time periods, the corresponding strategy is determined to be the optimal strategy: |T(t)-T ref |<ò T |SOC(t)-SOC ref |<ò SOC |ΔP(t)|<ò P Among them, T is the temperature tolerance value; SOC Tolerance value of SOC; P is the power tolerance value.

7. The energy storage system control method based on deep reinforcement learning and adaptive predictive control according to claim 6, characterized in that: The emergency response strategy includes a rapid response mode, a temperature priority mode and a strategy switch; The fast response mode means that when the load suddenly increases, the system will temporarily increase the discharge power to maintain the frequency stability of the power grid. This adjustment is quickly decided by the deep reinforcement learning DRL module, which determines the increase in discharge power based on the difference between past load data and current power output; The temperature priority mode includes: In emergencies, abnormal temperature rise may threaten the safety of the energy storage system. The hybrid control algorithm will prioritize the temperature safety of the system, automatically reduce the charge and discharge power, or even stop discharging to avoid temperature exceeding the limit. After the temperature gradually returns to the normal range, the system will gradually resume normal charging and discharging operations through rolling prediction; The strategy switching means that when an emergency occurs, the deep reinforcement learning DRL module automatically switches to emergency mode according to the trained strategy, calls a special operation sequence when the environment suddenly changes, avoids chain reactions caused by emergencies, and flexibly adapts to various emergency situations.

8. An energy storage system control system based on deep reinforcement learning and adaptive predictive control, based on the energy storage system control method based on deep reinforcement learning and adaptive predictive control according to any one of claims 1 to 7, characterized in that: include, The cost construction module is used to obtain the operating data of the energy storage system, construct the total cost function, and set the power constraint, temperature dynamic adjustment constraint, and SOC interval dynamic constraint of the energy storage system; A decision-making module, which is used to build a hybrid control algorithm based on deep reinforcement learning and adaptive predictive control to make optimal decisions; The decision convergence module is used to use the convergence conditions to judge the effectiveness of the optimal decision and formulate emergency response strategies; Safety optimization module, used to optimize the hybrid control algorithm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the energy storage system control method based on deep reinforcement learning and adaptive predictive control described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the energy storage system control method based on deep reinforcement learning and adaptive predictive control as described in any one of claims 1 to 7 are implemented.

Images