Energy storage scheduling methods, software products, and equipment based on deep reinforcement learning

Abstract

This invention provides an energy storage scheduling method, program product, and device based on deep reinforcement learning. The energy storage scheduling method includes: at each scheduling decision moment, acquiring real-time operating data of the energy storage system and external time-series data related to the scheduling decision moment; fusing the external time-series data with the real-time operating data after feature processing to form the input state vector of the deep reinforcement learning model; constructing a comprehensive reward function including an economic benefit term, a battery degradation cost penalty term, and a battery health and safety constraint penalty term; inputting the input state vector into the deep reinforcement learning model, using maximizing the comprehensive reward function as the training objective, driving the deep reinforcement learning model to learn a strategy, and outputting a charging and discharging strategy corresponding to the scheduling decision moment. This invention enables the charging and discharging strategy to dynamically balance immediate benefits with battery life-cycle losses and safety risks, achieving comprehensive optimal scheduling across all dimensions of the system.

Description

Technical Field

[0001] This invention relates to the field of energy storage, and in particular to an energy storage scheduling method, program product, and equipment based on deep reinforcement learning. Background Technology

[0002] With the large-scale integration of renewable energy into the grid, energy storage systems are playing an increasingly crucial role in improving the flexibility and economy of grid operation. The core of their efficient operation lies in formulating scientific charging and discharging strategies to maximize the overall system benefits or achieve other operational goals while meeting the physical constraints of the energy storage devices themselves.

[0003] However, existing energy storage scheduling schemes based on traditional optimization algorithms (such as ant colony optimization) still have significant limitations: In terms of optimization objectives, most methods are solely focused on maximizing immediate electricity revenue, failing to incorporate factors such as battery health degradation costs and operational safety risks into a unified optimization framework, making it difficult for the resulting strategies to achieve optimal performance throughout the system's entire lifecycle; in terms of search mechanisms, traditional algorithms are prone to getting trapped in local optima and struggle to effectively balance exploration and utilization under complex constraints, resulting in insufficient economic efficiency and robustness of the scheduling schemes; in terms of environmental adaptability, these methods heavily rely on preset fixed parameters, and once battery performance deteriorates or market prices fluctuate drastically, the original strategies may quickly become ineffective, lacking the ability to learn online and dynamically adjust to cope with uncertainty.

[0004] Therefore, the current energy storage dispatching field urgently needs a new intelligent dispatching method that can simultaneously take into account multiple objectives such as economy, safety and battery life, and has both global optimization capabilities and online adaptive learning characteristics, so as to support the efficient, reliable and long-term stable operation of energy storage systems in real and complex environments. Summary of the Invention

[0005] One objective of this invention is to construct a comprehensive reward function that integrates economic benefits, battery degradation costs, and health and safety constraints, enabling the charging and discharging strategy to dynamically balance immediate benefits with battery lifecycle losses and safety risks, thereby achieving comprehensive optimal scheduling across all dimensions of the system.

[0006] A further objective of this invention is to achieve adaptive adjustment of the balance between exploration and utilization during training by real-time calculation of the policy stability evaluation value and dynamic adjustment of the target entropy value, thereby improving the convergence efficiency of model training and the stability of the final policy.

[0007] Another further objective of this invention is to improve the online adaptability and robustness of the scheduling system to dynamic and uncertain environments by designing an environmental mutation entropy correction amount, so as to enable the model to actively enhance its exploration intensity when it senses drastic changes in the operating environment.

[0008] In particular, according to a first aspect of the present invention, the present invention provides an energy storage scheduling method based on deep reinforcement learning, applied to an energy storage system including battery cells, the energy storage scheduling method comprising:

[0009] At each scheduling decision moment, acquire the real-time operating data of the energy storage system and the external time-series data associated with the scheduling decision moment;

[0010] External time-series data is processed for features and then fused with real-time running data to form the input state vector of the deep reinforcement learning model;

[0011] Construct a comprehensive reward function, which includes at least an economic benefit term, a battery degradation cost penalty term, and a battery health and safety constraint penalty term.

[0012] The input state vector is fed into the deep reinforcement learning model, with the goal of maximizing the comprehensive reward function. This drives the deep reinforcement learning model to learn policies and outputs the charging and discharging policies corresponding to the scheduling decision time.

[0013] Optionally, real-time operating data includes the charge / discharge rate, terminal voltage, charge / discharge current, and state of charge of the battery cells; external time-series data includes historical electricity price data, historical electricity load data, and future environmental data; and the step of fusing the external time-series data with the real-time operating data after feature processing includes:

[0014] Time-series features were extracted from historical electricity load data and future environmental data to obtain future electricity load characteristics and future photovoltaic power output characteristics.

[0015] Historical electricity price data, future electricity load characteristics, future photovoltaic power output characteristics, charge / discharge rate, terminal voltage, charge / discharge current, and state of charge are integrated.

[0016] Optionally, the expression for the comprehensive reward function is:

[0017] ;

[0018] in, For the overall reward value, For economic benefits, As a penalty item for battery degradation costs, To balance economic benefits with the cost adjustment factor for battery life, This is a penalty item for battery health and safety constraints.

[0019] Optionally, the calculation steps for the battery degradation cost penalty term include:

[0020] Based on the charging and discharging current, real-time operating temperature, and depth of discharge of the battery cells within the current scheduling step, calculate the equivalent aging ampere-hour throughput increment generated by the scheduling step.

[0021] The increase in equivalent aging ampere-hour throughput is added to the historically accumulated total equivalent aging ampere-hour throughput.

[0022] Based on the updated total equivalent aging ampere-hour throughput, the incremental proportion of battery capacity loss caused within the current scheduling step is calculated using a preset battery capacity decay empirical model.

[0023] The incremental percentage of battery capacity loss, combined with the rated battery capacity of the energy storage system and the total battery purchase cost, is converted into an economic penalty value that characterizes the reduction in battery life.

[0024] Optionally, the battery health and safety constraint penalty items include a state of charge exceeding limit penalty item, a high-rate charge / discharge penalty item, and a state of charge change penalty item. The state of charge exceeding limit penalty item is used to constrain the state of charge of the battery cell to operate within a preset safe range. The high-rate charge / discharge penalty item is used to suppress excessively high charge and discharge current of the battery cell. The state of charge change penalty item is used to smooth the charge and discharge process of the battery cell and suppress drastic changes in the state of charge between adjacent scheduling decision times.

[0025] Optionally, the process of driving a deep reinforcement learning model to learn a policy also includes:

[0026] Based on the changing characteristics of the internal state parameters of the deep reinforcement learning model, the policy stability evaluation value of the current training stage is calculated. The internal state parameters include at least one or more of the following: policy gradient, action selection distribution, and value function prediction.

[0027] The target entropy value of the deep reinforcement learning model is dynamically adjusted based on the policy stability evaluation value, where the target entropy value is used to characterize the randomness and uncertainty of the policy.

[0028] Optionally, the step of dynamically adjusting the target entropy value of the deep reinforcement learning model based on the policy stability evaluation value includes:

[0029] When the policy stability assessment value exceeds a preset first threshold, the target entropy value is reduced to enhance the policy's utilization of learned experience; and

[0030] When the policy stability assessment value is less than the preset second threshold, the target entropy value is increased to enhance the policy's exploration of unknown states, where the second threshold is less than the first threshold.

[0031] Optionally, after the step of dynamically adjusting the target entropy value of the deep reinforcement learning model based on the policy stability evaluation value, the method further includes:

[0032] Monitor the intensity of mutations in the overall reward value of the deep reinforcement learning model during training;

[0033] Calculate the environmental mutation entropy correction based on the mutation intensity and the current strategy stability assessment value;

[0034] The target entropy value, adjusted based on the strategy stability assessment value, is compared with the enhanced exploration entropy value determined by the environmental mutation entropy correction amount.

[0035] The larger of the two values ​​is selected as the final target entropy value to enhance the exploration capabilities of deep reinforcement learning models when the environment changes drastically.

[0036] According to a second aspect of the present invention, the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the deep reinforcement learning-based energy storage scheduling method described above.

[0037] According to a third aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the deep reinforcement learning-based energy storage scheduling method described above.

[0038] The energy storage scheduling method based on deep reinforcement learning of this invention constructs an input state vector by fusing real-time operational data and associated external time-series data at each scheduling decision point, and drives a deep reinforcement learning model to make decisions with the goal of maximizing the comprehensive reward function. This comprehensive reward function innovatively integrates economic benefits, battery degradation costs, and health and safety constraints within the same optimization framework. This allows the model's output charging and discharging strategy to automatically and dynamically balance immediate economic benefits with long-term battery lifespan degradation and operational safety risks, achieving comprehensive optimal scheduling across the entire lifecycle of the system. This overcomes the shortcomings of traditional methods that have a single objective and neglect long-term costs.

[0039] Furthermore, the energy storage scheduling method based on deep reinforcement learning in this invention calculates the policy stability evaluation value, which reflects the fluctuations of internal state parameters such as policy gradient and action distribution, in real time, and dynamically adjusts the target entropy value, which represents the randomness of the policy, accordingly. This method achieves adaptive adjustment of the balance between exploration and exploitation during training: when the policy becomes unstable, it automatically reduces the target entropy to enhance exploitation and consolidate existing knowledge; when the policy becomes rigid, it automatically increases the target entropy to encourage exploration and escape local optima. This mechanism significantly improves the convergence efficiency of model training and the stability of the final policy, effectively solving the problems of traditional optimization algorithms and reinforcement learning models with fixed exploration mechanisms easily getting trapped in local optima and experiencing large training fluctuations.

[0040] Furthermore, the energy storage scheduling method based on deep reinforcement learning in this invention monitors the mutation intensity of the comprehensive reward value during training and calculates the environmental mutation entropy correction amount by combining it with the current policy stability evaluation value. Then, it compares the target entropy value with an enhanced exploration entropy value determined by this correction amount and selects the larger one as the final target. This design enables the model to proactively and significantly increase its exploration intensity when it perceives drastic environmental changes (such as sharp fluctuations in electricity prices or load mutations), forcing the model to quickly try new strategies in the new environment. This greatly enhances the online adaptability and robustness of the scheduling system to dynamic and uncertain operating environments, overcoming the weaknesses of traditional methods that rely on fixed parameters and have poor environmental adaptability.

[0041] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0043] Figure 1 This is a schematic diagram of an energy storage scheduling method based on deep reinforcement learning according to an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the battery degradation cost penalty term calculation steps in an energy storage scheduling method based on deep reinforcement learning according to an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the adaptive entropy adjustment step in an energy storage scheduling method based on deep reinforcement learning according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the environmental change response step in an energy storage scheduling method based on deep reinforcement learning according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of a computer program product according to an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention;

[0049] Figure 7 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0050] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0051] This invention provides an energy storage scheduling method based on deep reinforcement learning, aiming to achieve intelligent and adaptive optimization of the charging and discharging strategy of an energy storage system through machine learning technology. This method is applicable to energy storage systems containing battery cells and can comprehensively consider multiple objectives such as economy, battery health, and operational safety in complex and variable operating environments, thereby formulating a scientific and efficient charging and discharging strategy.

[0052] Figure 1 This is a schematic diagram of an energy storage scheduling method based on deep reinforcement learning according to an embodiment of the present invention, as shown below. Figure 1 As shown, the energy storage dispatch method includes at least the following:

[0053] Step S101: At each scheduling decision moment, acquire real-time operating data of the energy storage system and external time-series data associated with the scheduling decision moment. Real-time operating data refers to measurements directly acquired from local sensors of the energy storage system that reflect the instantaneous physical state of the battery cells. For example, real-time operating data may include the charge / discharge rate, terminal voltage, charge / discharge current, and state of charge of the battery cells. Specifically, the charge / discharge rate indicates the ratio of the battery's current charge / discharge power to its rated capacity, reflecting the charge / discharge intensity; the terminal voltage indicates the potential difference between the positive and negative terminals of the battery; the charge / discharge current indicates the magnitude and direction of the current during the battery's charge / discharge process; and the state of charge indicates the proportion of the battery's current remaining capacity to its rated capacity. External time-series data refers to data with time-series characteristics related to the electricity market and operating environment, with its time axis associated with the current scheduling decision moment. For example, external time-series data may include historical electricity price data, historical electricity load data, and future environmental data. Among them, historical electricity price data is used to indicate the transaction price of the electricity market in the past period; historical electricity load data is used to indicate the changes in electricity consumption or power consumption on the user side in the past period; and future environmental data is used to indicate the conditions such as sunshine and weather in the subsequent period, such as photovoltaic power output conditions.

[0054] Step S102 involves fusing the external time-series data with real-time operational data after feature processing to form the input state vector of the deep reinforcement learning model. For historical electricity load data and future environmental data with strong time-series dependencies, time-series feature extraction techniques (such as Long Short-Term Memory networks) can be used to obtain future electricity load features and future photovoltaic output features, respectively. The extracted future electricity load features, future photovoltaic output features, and historical electricity price data (which do not require complex transformations) are then fused with the real-time operational data of the battery cells (charge / discharge rate, terminal voltage, charge / discharge current, and state of charge) to form a fixed-dimensional input state vector. This vector integrates the system's internal real-time state, external historical patterns, and future scenario predictions, providing the model with a complete information view required for global optimization decisions.

[0055] Step S103: Construct a comprehensive reward function, which includes at least an economic benefit term, a battery degradation cost penalty term, and a battery health and safety constraint penalty term. The economic benefit term relates to the economic returns of charging and discharging behavior in the electricity market. For example, it can be designed to generate profit from discharging when electricity prices are high and to charge and store energy when electricity prices are low. The battery degradation cost penalty term aims to achieve full lifecycle economic optimization. Based on a battery aging model, it quantifies in real-time the small, irreversible losses to battery capacity caused by current charging and discharging operations (discharge depth, temperature, throughput, etc.) and converts these losses into equivalent economic costs as a negative reward. This forces the model to pay the corresponding battery lifespan loss cost when pursuing discharge revenue, thus automatically learning to find the optimal balance between short-term gains and long-term asset value. The battery health and safety constraint penalty term ensures the safe and reliable operation of the energy storage system. It applies negative incentives to behaviors that may harm battery health or exceed safe operating boundaries, prompting the model to intrinsically avoid high-risk operations during the learning process.

[0056] Step S104: The input state vector is fed into the deep reinforcement learning model. With maximizing the comprehensive reward function as the training objective, the deep reinforcement learning model learns a policy and outputs a charging / discharging policy corresponding to the scheduling decision moment. At the current scheduling decision moment, the model, based on the received input state vector, uses its internally learned policy network to perform forward computation and inference, evaluates the long-term value of different charging / discharging actions under the current internal and external environment, and generates a decision accordingly. Finally, it outputs an optimized charging / discharging policy (usually expressed as a specific charging / discharging power command) that precisely corresponds to the current scheduling decision moment.

[0057] Using the aforementioned energy storage scheduling method, an input state vector is constructed by fusing real-time operational data with associated external time-series data at each scheduling decision point. This vector drives a deep reinforcement learning model to make decisions with the objective of maximizing the comprehensive reward function. This comprehensive reward function innovatively integrates economic benefits, battery degradation costs, and health and safety constraints within the same optimization framework. This allows the model's output charging and discharging strategy to automatically and dynamically balance immediate economic benefits with long-term battery lifespan degradation and operational safety risks, achieving comprehensive optimal scheduling across the entire lifecycle of the system. This overcomes the shortcomings of traditional methods, which often have a single objective and neglect long-term costs.

[0058] In an optional embodiment, the expression for the comprehensive reward function can be:

[0059]

[0060] in, For the overall reward value, For economic benefits, As a penalty item for battery degradation costs, To balance economic benefits with the cost adjustment factor for battery life, This is a penalty item for battery health and safety constraints.

[0061] This invention integrates the optimization objectives of economic benefits, battery health, and operational safety into a unified mathematical framework by designing a comprehensive reward function, thereby achieving synergistic optimization of multiple objectives.

[0062] In one optional embodiment, the economic benefit term is defined as the net difference between the electricity sales revenue obtained by the energy storage system from discharging and the electricity purchase cost incurred from charging during the time interval between adjacent scheduling decision moments. In other words, the calculated immediate economic benefit at each scheduling decision moment t. This refers to the net economic benefits generated by actual charging and discharging behavior within a time step (or a market settlement cycle) from the previous decision time t-1 to the current time t.

[0063] Figure 2 This is a schematic diagram illustrating the battery degradation cost penalty term calculation steps in a deep reinforcement learning-based energy storage scheduling method according to an embodiment of the present invention, as shown below. Figure 2 As shown, the calculation steps for the battery degradation cost penalty term may include:

[0064] Step S201: Calculate the equivalent aging ampere-hour throughput increment generated by the scheduling step based on the charging and discharging current, real-time operating temperature, and depth of discharge of the battery cells within the current scheduling step.

[0065] Step S202: The incremental equivalent aging ampere-hour throughput is added to the historically accumulated total equivalent aging ampere-hour throughput.

[0066] Step S203: Based on the updated total equivalent aging ampere-hour throughput, calculate the incremental proportion of battery capacity loss caused within the current scheduling step using a preset battery capacity decay empirical model.

[0067] Step S204: The incremental percentage of battery capacity loss is combined with the rated capacity of the battery in the energy storage system and the total purchase cost of the battery to convert it into an economic penalty value that characterizes the battery life loss.

[0068] By embedding a battery aging model based on equivalent ampere-hour throughput into real-time scheduling decisions, dynamic quantification and economic penalties for battery life degradation costs are achieved. This enables energy storage systems to automatically balance energy gains and battery life losses when participating in the electricity market, avoid high-aging conditions such as high temperatures and deep charge-discharge cycles, improve the economic benefits throughout the entire life cycle, extend the actual service life of batteries, and indirectly enhance the safety of system operation.

[0069] In this embodiment, the current scheduling step size can be used as a starting point. The charging and discharging current of the internal battery Real-time operating temperature Current discharge depth Calculate the equivalent aging ampere-hour throughput increment generated within this step size. :

[0070]

[0071] in, The depth of discharge influence index; It is the electrochemical activation energy; This is the universal gas constant; This is a reference temperature.

[0072] Next, Total equivalent aging ampere-hour throughput accumulated to history middle:

[0073]

[0074] Then, based on the updated total equivalent aging ampere-hour throughput Using a pre-defined empirical model of battery capacity degradation (such as a power-law model), the incremental proportion of battery capacity loss within the current scheduling step is calculated. :

[0075]

[0076] in, and These are the parameters for model fitting.

[0077] Finally, the capacity loss ratio increment Converted to corresponding economic penalty value :

[0078]

[0079] in, This refers to the battery's rated capacity. This represents the total cost of purchasing the battery.

[0080] In one alternative embodiment, battery health and safety constraint penalty item It consists of a weighted average of the following three sub-penalty items:

[0081]

[0082] in, For the penalty term for exceeding the limit of the charged state, For high-rate charge / discharge electron penalties, This is a penalty term for charge state mutations. , , These are the weighting coefficients for each sub-penalty item, used to adjust the relative importance of different security constraints in the total penalty.

[0083] Used to constrain the state of charge of battery cells ( It operates within a preset safe range to avoid overcharging and over-discharging. Its penalty value increases non-linearly with the degree of deviation of the state of charge from the safe boundary, thus forming an effective soft boundary constraint. For example, a safe upper limit can be defined. The lower safety limit is 0.9. It is 0.2. The calculation method is as follows:

[0084] when hour,

[0085] when hour,

[0086] when hour,

[0087] in, For example, the boundary penalty coefficient can be taken as... A value of 100 is used to control the severity of the punishment.

[0088] This penalty is used to suppress excessively high charge and discharge currents in battery cells (i.e., high-rate operation) to reduce power and thermal stress on the battery and extend its lifespan. Its penalty value is proportional to the square of the charge and discharge current, causing the penalty to increase rapidly during high-rate operation. The calculation method is as follows:

[0089]

[0090] in, The battery charging and discharging current within the current scheduling step (discharging is positive, charging is negative). Its absolute value; This is the multiplier penalty factor, used to adjust the sensitivity to current amplitude.

[0091] This is used to smooth the charging and discharging process of battery cells and suppress drastic changes in state of charge between adjacent scheduling decision points (e.g., continuous full charging and discharging within a short period of time), thereby reducing cyclic aging stress. Its penalty value varies with... The absolute value of the change increases exponentially, thus affecting drastic changes. Fluctuations are subject to a significant penalty. The calculation method is as follows:

[0092]

[0093] in, This represents the change in state of charge between adjacent scheduling decision times. and For adjustment coefficients (e.g., can be taken as follows) For 10, (5), jointly controlling the shape and intensity of the penalty curve.

[0094] By weighting and combining the above three sub-penalty items, the battery health and safety constraint penalty item... It can comprehensively and quantitatively apply negative incentives to operational behaviors that damage battery health or threaten operational safety from three dimensions: boundary safety, power safety, and process smoothness, thereby guiding the deep reinforcement learning model to endogenously form a safe and stable charging and discharging behavior pattern during the policy learning process.

[0095] In driving the deep reinforcement learning model to learn policies, this method further introduces a hierarchical policy stability enhancement mechanism. This mechanism first adjusts the balance between exploration and exploitation during regular training by sensing fluctuations in the model's internal state, and then adds a sudden environmental change response mechanism to cope with unexpected changes under dynamic operating conditions. Specifically, this mechanism includes the following two main components:

[0096] The first step is adaptive entropy adjustment based on policy stability. Specifically, based on the changing characteristics of the internal state parameters (such as policy gradient, action selection distribution, and value function prediction) of the deep reinforcement learning model during training iterations, the policy stability evaluation value for the current training stage is calculated in real time. Subsequently, based on the policy stability evaluation value, the target entropy value in the training of the deep reinforcement learning model is dynamically adjusted. The target entropy value is used to quantify the randomness and uncertainty of the policy, and its level directly affects the balance between "utilizing known experience" and "exploring unknown states" in the model.

[0097] Figure 3 This is a schematic diagram of the adaptive entropy adjustment step in an energy storage scheduling method based on deep reinforcement learning according to an embodiment of the present invention, as shown below. Figure 3 As shown, it specifically includes:

[0098] Step S301: Calculate the policy stability evaluation value for the current training phase based on the change characteristics of the internal state parameters of the deep reinforcement learning model.

[0099] In step S302, when the policy stability evaluation value is greater than a preset first threshold (e.g., 0.8), the target entropy value is reduced to enhance the policy's utilization of learned experience.

[0100] Step S303: When the policy stability evaluation value is less than the preset second threshold (e.g., it can be taken as 0.4), the target entropy value is increased to enhance the policy's exploration of unknown states.

[0101] Step S304: When the strategy stability assessment value is between the first threshold and the second threshold (including the boundary), the target entropy value is kept relatively stable, and the current exploration-exploitation balance is maintained.

[0102] Using the above method, an adaptive adjustment of the balance between exploration and exploitation during training is achieved: when the policy becomes unstable, the target entropy is automatically reduced to enhance exploitation and consolidate existing knowledge; when the policy becomes rigid, the target entropy is automatically increased to encourage exploration and escape local optima. This mechanism significantly improves the convergence efficiency of model training and the stability of the final policy, effectively solving the problems of traditional optimization algorithms and reinforcement learning models with fixed exploration mechanisms being prone to getting trapped in local optima and experiencing large training fluctuations.

[0103] In this embodiment, the strategy stability evaluation value Calculated using the following formula:

[0104]

[0105] in, This represents the policy stability evaluation value at the current training stage (time t), with a value range of (0, 1]. The closer it is to 1, the more stable the strategy. It represents the standard deviation of the policy gradient norm over recent training steps, reflecting the volatility of the policy update direction; It represents the standard deviation of the movement distribution in recent training steps, reflecting the volatility of the randomness of movement selection; It represents the standard deviation of the action value function prediction over recent training steps, reflecting the volatility of the value estimate; , , These are the weighting coefficients for the corresponding volatility indicators, used to adjust the contribution ratio of each item to the stability assessment.

[0106] Based on the calculated strategy stability assessment value The target entropy value is determined by the following dynamic entropy adjustment equation. :

[0107]

[0108] in, The target entropy value is dynamically adjusted. The preset base entropy value; The standard entropy coefficient is used to determine the baseline level for the target entropy value; The stability response coefficient is used to control the stability of the strategy. The intensity and direction of adjustment of the target entropy value; To explore the incentive coefficient, The attenuation rate, This represents the current number of training steps. This represents the total number of training steps.

[0109] The second stage involves exploring and enhancing the response to environmental mutations. To further improve the model's adaptability in dynamic and uncertain environments, an environmental mutation response stage is added based on the aforementioned adjustments. Figure 4 This is a schematic diagram of the environmental abrupt change response step in an energy storage scheduling method based on deep reinforcement learning according to an embodiment of the present invention, as shown below. Figure 4 As shown, it specifically includes:

[0110] Step S401: Monitor the mutation intensity of the comprehensive reward value of the deep reinforcement learning model during training.

[0111] Step S402: Calculate the environmental mutation entropy correction based on the mutation intensity and the current strategy stability assessment value.

[0112] Step S403 compares the target entropy value, which is adjusted based on the strategy stability assessment value, with the enhanced exploration entropy value determined by the environmental mutation entropy correction amount.

[0113] Step S404: Select the larger of the two values ​​as the final effective target entropy value to enhance the exploration capability of the deep reinforcement learning model when the environment changes drastically.

[0114] Using the above method, when the model senses drastic changes in the environment (such as sharp fluctuations in electricity prices or sudden changes in load), it can proactively and significantly increase the exploration intensity, forcing the model to quickly try new strategies in the new environment. This greatly enhances the online adaptability and robustness of the scheduling system to dynamic and uncertain operating environments, and overcomes the weaknesses of traditional methods that rely on fixed parameters and have poor environmental adaptability.

[0115] In this embodiment, the environmental mutation entropy correction amount Calculated using the following formula:

[0116]

[0117] in, This represents the amount of entropy correction caused by sudden environmental changes; The response intensity coefficient is used to control the overall correction magnitude; For the most recent Within a step, the L2 norm change of the reward sequence obtained by the model; The standard deviation of historical rewards is used to standardize reward mutations and make them comparable. This indicates that the more unstable the strategy itself ( The smaller the value, the more sensitive it is to sudden environmental changes, and the greater the correction magnitude.

[0118] In obtaining the environmental mutation correction amount Then, the final effective, enhanced target entropy value is determined using the following formula. :

[0119]

[0120] in, The final, enhanced target entropy value; This is the basic target entropy value obtained in the first stage; The preset base entropy value; This represents the amount of entropy correction caused by sudden environmental changes.

[0121] The above formula means: the basic target entropy value obtained in the first stage. , and the basic entropy value and environmental mutation correction amount Jointly determined enhanced exploration entropy The two values ​​are compared, and the larger one is selected as the final target. This ensures that once an environmental mutation is detected, the target entropy value will be raised to a baseline that is significantly higher than the normal level. This forces the model to significantly increase its exploratory behavior after environmental changes, enabling it to quickly adapt to new environments.

[0122] This embodiment also provides a computer program product 11, a computer-readable storage medium 12, and a computer device 10. Figure 5 This is a schematic diagram of a computer program product according to an embodiment of the present invention. Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Figure 7 This is a schematic block diagram of a computer device according to an embodiment of the present invention.

[0123] Computer program product 11 includes computer program 111, which, when executed by processor 101, implements any of the aforementioned deep reinforcement learning-based energy storage scheduling methods. Computer-readable storage medium 12 stores the aforementioned computer program 111, which, when executed by processor 101, implements any of the aforementioned deep reinforcement learning-based energy storage scheduling methods. Computer device 10 may include memory 102, processor 101, and computer program 111 stored in memory 102 and running on processor 101.

[0124] The computer program 111 used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages.

[0125] Computer program 111 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of the invention, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of computer-readable program instructions.

[0126] For the purposes of this embodiment, computer program product 11 is a related product containing computer program 111. Computer-readable storage medium 12 is a tangible device capable of holding and storing computer program 111, and can be any device capable of containing, storing, communicating, propagating, or transmitting computer program 111 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 12 include: portable computer disks, hard disks, random access memory 102 (RAM), read-only memory 102 (ROM), erasable programmable read-only memory 102 (EPROM or flash memory), static random access memory 102 (SRAM), portable optical disc read-only memory 102 (CD-ROM), digital multifunction disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0127] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A deep reinforcement learning-based energy storage scheduling method, applied to an energy storage system containing battery cells, characterized in that, The energy storage dispatch method includes: At each scheduling decision moment, the real-time operation data of the energy storage system and the external time-series data associated with the scheduling decision moment are acquired. The real-time operation data includes the charge / discharge rate, terminal voltage, charge / discharge current and state of charge of the battery cell. The external time-series data includes historical electricity price data, historical electricity load data and future environmental data. The external time-series data is processed for features and then fused with the real-time running data to form the input state vector of the deep reinforcement learning model. A comprehensive reward function is constructed, which includes at least an economic benefit term, a battery degradation cost penalty term, and a battery health and safety constraint penalty term. The battery health and safety constraint penalty term includes a state of charge (SCC) exceeding limit sub-penalty term, a high-rate charge / discharge penalty term, and a SCC mutation sub-penalty term. The SCC exceeding limit sub-penalty term is used to constrain the SCC of the battery cell to operate within a preset safe range. The high-rate charge / discharge penalty term is used to suppress excessively high charge / discharge currents of the battery cell. The SCC mutation sub-penalty term is used to smooth the charge / discharge process of the battery cell and suppress drastic changes in SCC between adjacent scheduling decision times. The input state vector is input into the deep reinforcement learning model, with the goal of maximizing the comprehensive reward function, to drive the deep reinforcement learning model to learn the policy and output the charging and discharging policy corresponding to the scheduling decision time. The process of driving the deep reinforcement learning model to learn the policy also includes: Based on the changing characteristics of the internal state parameters of the deep reinforcement learning model, the policy stability evaluation value of the current training stage is calculated, wherein the internal state parameters include at least one or more of the following: policy gradient, action selection distribution, and value function prediction. The target entropy value of the deep reinforcement learning model is dynamically adjusted based on the policy stability evaluation value, wherein the target entropy value is used to characterize the randomness and uncertainty of the policy; The steps for dynamically adjusting the target entropy value of the deep reinforcement learning model based on the policy stability evaluation value include: When the policy stability assessment value exceeds a preset first threshold, the target entropy value is reduced to enhance the policy's utilization of learned experience; and When the policy stability assessment value is less than a preset second threshold, the target entropy value is increased to enhance the policy's exploration of unknown states, wherein the second threshold is less than the first threshold. Among them, the strategy stability evaluation value Calculated using the following formula: ； in, This represents the policy stability evaluation value at the current training stage (time t), with a value range of (0, 1]. The closer it is to 1, the more stable the strategy. It represents the standard deviation of the policy gradient norm over recent training steps, reflecting the volatility of the policy update direction; It represents the standard deviation of the movement distribution in recent training steps, reflecting the volatility of the randomness of movement selection; It represents the standard deviation of the action value function prediction over recent training steps, reflecting the volatility of the value estimate; , , These are the weighting coefficients for the corresponding volatility indicators, used to adjust the contribution ratio of each item to the stability assessment; Based on the calculated strategy stability assessment value The target entropy value is determined by the following dynamic entropy adjustment equation. : ； in, The target entropy value is dynamically adjusted. The preset base entropy value; The standard entropy coefficient is used to determine the baseline level for the target entropy value. The stability response coefficient is used to control the stability of the strategy. The intensity and direction of adjustment of the target entropy value; To explore the incentive coefficient, The attenuation rate, This represents the current number of training steps. This represents the total number of training steps.

2. The energy storage scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The steps of fusing the external time-series data after feature processing with the real-time running data include: Time-series features are extracted from the historical electricity load data and the future environmental data respectively to obtain the future electricity load characteristics and the future photovoltaic output characteristics; The historical electricity price data, the future electricity load characteristics, the future photovoltaic power output characteristics, the charge / discharge rate, the terminal voltage, the charge / discharge current, and the state of charge are fused together.

3. The energy storage scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The expression for the comprehensive reward function is: ； in, For the overall reward value, For the aforementioned economic benefit item, The battery degradation cost penalty item, To balance economic benefits with the cost adjustment factor for battery life, This refers to the battery health and safety constraint penalty item.

4. The energy storage scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The calculation steps for the battery degradation cost penalty include: Based on the charging and discharging current, real-time operating temperature, and depth of discharge of the battery cell within the current scheduling step, calculate the equivalent aging ampere-hour throughput increment generated by the scheduling step. The equivalent aging ampere-hour throughput increment is added to the historically accumulated total equivalent aging ampere-hour throughput; Based on the updated total equivalent aging ampere-hour throughput, the incremental proportion of battery capacity loss caused within the current scheduling step is calculated using a preset battery capacity decay empirical model. The incremental percentage of battery capacity loss, combined with the rated battery capacity and total battery purchase cost of the energy storage system, is converted into an economic penalty value representing the reduction in battery life.

5. The energy storage scheduling method based on deep reinforcement learning according to claim 1, characterized in that, After the step of dynamically adjusting the target entropy value of the deep reinforcement learning model based on the policy stability evaluation value, the method further includes: Monitor the mutation intensity of the overall reward value of the deep reinforcement learning model during training; Calculate the environmental mutation entropy correction amount based on the mutation intensity and the current strategy stability assessment value; The target entropy value, adjusted based on the strategy stability assessment value, is compared with the enhanced exploration entropy value determined by the environmental mutation entropy correction amount. The larger of the two values ​​is selected as the final effective target entropy value to enhance the exploration capability of the deep reinforcement learning model when the environment changes drastically. Among them, the environmental mutation entropy correction amount Calculated using the following formula: ； in, This represents the amount of entropy correction caused by sudden environmental changes; The response intensity coefficient is used to control the overall correction magnitude; For the most recent Within a step, the L2 norm change of the reward sequence obtained by the model; The standard deviation of historical rewards is used to standardize reward mutations and make them comparable. This indicates that the more unstable the strategy itself ( The smaller the value, the more sensitive it is to sudden environmental changes, and the greater the correction magnitude. In obtaining the environmental mutation correction amount Then, the final effective, enhanced target entropy value is determined using the following formula. : ； in, The final, enhanced target entropy value; This is the basic target entropy value obtained in the first stage; The preset base entropy value; This represents the amount of entropy correction caused by sudden environmental changes.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the energy storage scheduling method based on deep reinforcement learning as described in any one of claims 1 to 5.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the energy storage scheduling method based on deep reinforcement learning as described in any one of claims 1 to 5.

Images