A method for maximizing energy storage benefits based on a policy gradient algorithm
By adopting an energy storage control method based on policy gradient algorithm, and combining policy gradient and graph optimization, the adaptability and stability of energy storage system in complex environments are solved, and efficient optimization of multi-device collaborative control is achieved, thereby improving the revenue stability and flexibility of energy storage system.
Patent Information
- Application Number
- CN202511003945.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2045-07-21
AI Technical Summary
Existing energy storage control methods suffer from poor adaptability, insufficient strategy stability, and weak coordination among multiple devices when facing complex factors in the power system, such as real-time electricity price fluctuations, load forecasting errors, and multi-device operation coupling. This makes it difficult to achieve the cyclical evolution and continuous optimization of charging and discharging strategies.
A strategy gradient algorithm-based approach, combined with strategy gradient and graph optimization methods, is adopted. By constructing energy storage state feature vectors, charging and discharging actions, and reward signals as policy training inputs, an information entropy regularization term and a time-series modeling mechanism are introduced to build a hierarchical optimization structure, thereby realizing the coordinated control of multiple energy storage devices.
It enhances the economic efficiency and controllability of energy storage systems in the electricity market, ensures the stability and adaptability of strategies, achieves global consistency and synergistic optimization, and improves the revenue stability and flexibility of energy storage devices in complex environments.
Smart Images
Figure CN120896202B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent energy storage control, and in particular to a method for maximizing energy storage benefits based on a policy gradient algorithm. Background Technology
[0002] In the current power system dispatching and operation, energy storage devices serve as an important means of regulating energy supply and demand balance, and are widely used in scenarios such as new energy grid integration, electricity price arbitrage, and peak-valley regulation. Traditional energy storage control methods mainly rely on rule-based strategies or model predictive control, typically formulating charging and discharging plans by setting fixed thresholds, linear programming, or heuristic methods. These methods have significant limitations in parameter setting, adaptability, and handling of uncertainties, and are difficult to effectively address complex factors such as real-time electricity price fluctuations, load forecasting errors, and the coupling of multiple devices.
[0003] In recent years, with the rapid development of artificial intelligence technology, reinforcement learning methods have been gradually introduced into the field of energy storage control. By constructing policy functions and optimizing them based on historical interaction data, reinforcement learning has strong self-learning and environmental adaptability. However, the application of existing reinforcement learning in energy storage scheduling still faces many challenges: First, most methods are based on single-cycle optimization, ignoring the temporal dependence of energy storage states and the coupling relationship with long-term returns, making it difficult to achieve the cyclical evolution and continuous optimization of charging and discharging strategies; Second, in the collaborative control of multiple energy storage devices, it is difficult for strategies to share information, which can easily lead to local optima; Third, the lack of control over the stability of policy distribution and generalization ability during policy optimization leads to unstable training processes, large policy fluctuations, and reduced reliability of returns. Summary of the Invention
[0004] One objective of this invention is to propose a method for maximizing energy storage revenue based on a policy gradient algorithm. This invention integrates policy gradient and graph optimization methods to achieve coordinated control of energy storage, and has the advantages of high revenue, strong adaptability and policy stability.
[0005] A method for maximizing energy storage revenue based on a policy gradient algorithm according to an embodiment of the present invention includes the following steps:
[0006] Collect energy storage device status parameters, real-time electricity market price data, and load forecast data; preprocess the collected data to generate preprocessed data.
[0007] Feature extraction is performed on the preprocessed data to generate an energy storage state feature vector. The energy storage state feature vector is used as the system state input to initialize the charging and discharging strategy parameters and generate the initial charging and discharging strategy.
[0008] Based on the initial charge and discharge strategy, the charge and discharge actions for this cycle are generated, and the charge and discharge actions are applied to the energy storage device to perform the charge and discharge operations. The results of the charge and discharge operations are collected, and charge and discharge feedback information is generated.
[0009] By combining charging and discharging feedback information with charging and discharging actions, and calculating a reward signal based on the difference between immediate and expected benefits, the reward signal, energy storage state feature vector, and charging and discharging actions are used to update the charging and discharging strategy parameters to generate an updated charging and discharging strategy.
[0010] By utilizing the time-series dependencies of energy storage state feature vectors, charging and discharging actions, and reward signal sequences over multiple cycles, feedback optimization is performed on future cycle returns to generate a cycle-evolving charging and discharging strategy.
[0011] In multi-energy storage device scenarios, the energy storage state feature vectors, charging and discharging actions, and reward signals of each energy storage device are used for lower-level autonomous strategy optimization and upper-level device coordination to generate a globally optimal collaborative charging and discharging strategy.
[0012] Optionally, the energy storage device status parameters include state of charge, voltage, current, charging power and maximum charging power, discharging power and maximum discharging power, temperature and cycle life status, and the preprocessing includes time synchronization, outlier detection, missing value completion, noise filtering and normalization.
[0013] Optionally, the generation of the initial charge / discharge strategy specifically includes:
[0014] The preprocessed data is arranged in chronological order to construct a multidimensional data sequence with multiple consecutive time steps. Each time step includes energy storage device status parameters, real-time electricity market price data, and load forecast data.
[0015] Feature extraction is performed on multidimensional data sequences. Short-term dynamic change features between time steps are extracted through a one-dimensional convolution structure. The convolution results are then activated and compressed to obtain the feature response within a local time segment.
[0016] The feature response within a local time segment is input into a bidirectional long short-term memory neural network, and recursive feature encoding is performed simultaneously in both the forward and reverse directions of time to capture the trend pattern in the long time series. Finally, the corresponding energy storage state feature vector is output at the current time step.
[0017] The energy storage state feature vector is used as the system state input, and the charging and discharging strategy parameters are initialized.
[0018] The energy storage state feature vector and the initial charge / discharge strategy parameters are input into the strategy network. The initial charge / discharge strategy parameters are used to initialize the strategy network. The strategy network is a feedforward neural network with a two-layer fully connected structure. The first layer uses a modified linear unit as the activation function, and the second layer uses a hyperbolic tangent function as the activation function. The output is the initial charge / discharge strategy under the current system state.
[0019] Optionally, the generation of the charge / discharge feedback information specifically includes:
[0020] The charging and discharging actions for the current cycle are generated based on the initial charging and discharging strategy;
[0021] The generated charging and discharging actions are applied to the energy storage device, and the actual charging and discharging operations are executed according to the control parameters;
[0022] After the charging and discharging operation is completed, the energy storage state feature vector at the end of the charging and discharging operation is collected;
[0023] The immediate revenue for the current period is calculated by combining the charging and discharging actions with the real-time price of the electricity market. The immediate revenue is the economic return obtained by the energy storage device through charging and discharging during the operation, which is obtained by multiplying the electricity by the real-time electricity price.
[0024] The energy storage state feature vector after charging and discharging is combined with the immediate benefits to form charging and discharging feedback information.
[0025] Optionally, the generation of the updated charge / discharge strategy specifically includes:
[0026] The reward signal is calculated based on the difference between the immediate return and the expected return for the current period.
[0027] A policy training input consisting of energy storage state feature vectors, charging and discharging actions, and reward signals is constructed. The charging and discharging policy parameters are optimized using a policy network, and an optimization objective function is constructed.
[0028] The charging / discharging strategy parameters are updated using gradient ascent based on the optimization objective function;
[0029] The updated charge / discharge strategy parameters are applied to the energy storage state feature vector of the current cycle to generate an updated charge / discharge strategy for the next cycle's strategy iteration.
[0030] Optionally, the generation of the periodically self-evolving charge-discharge strategy specifically includes:
[0031] Construct a historical behavior trajectory sequence to obtain energy storage state feature vector sequences, charging and discharging action sequences, and reward signal sequences over multiple consecutive periods;
[0032] The three types of sequences are input into the time series modeling structure, and a gated recurrent unit network is used to extract the temporal state representation under each period to form a joint state representation vector sequence.
[0033] Based on the current period joint state representation vector, the charging and discharging policy probability distribution is generated using the policy function, and the policy loss function is constructed by combining the advantage function.
[0034] The policy loss function is trained using gradient ascent, the parameters of the policy function are updated, and a sliding time window mechanism is introduced in each update to control the historical behavior trajectory within the most recent k cycles as the basis for optimization.
[0035] The updated strategy function parameters are used in the strategy generation process of the next cycle. By combining the energy storage state feature vector of the latest cycle and the joint state representation vector corresponding to the current cycle, the cycle-evolving charging and discharging strategy of the current cycle is generated.
[0036] Optionally, the generation of the globally optimal cooperative charging and discharging strategy specifically includes:
[0037] In an operation scenario involving multiple energy storage devices, the energy storage state feature vector, charging and discharging actions in the current cycle, and corresponding reward signals of each energy storage device are extracted. A hierarchical optimization structure consisting of autonomous optimization of lower-level devices and coordination between upper-level devices is constructed. Based on the fusion optimization results, the globally optimal collaborative charging and discharging strategy is output and fed back to each energy storage device for execution.
[0038] In the lower-level autonomous optimization process, for the i-th energy storage device, a strategy function is constructed with the energy storage state feature vector, charging and discharging actions and reward signals as inputs, and a cumulative objective function is constructed by maximizing the cumulative discount reward;
[0039] In the upper-level collaborative optimization process, all energy storage devices are constructed as corresponding nodes in the graph structure. The energy storage state feature vector, charging and discharging actions and reward signals are used as node attributes. The edge weights are defined using the coupling relationship between each pair of devices. Aggregated features are obtained through the graph attention mechanism, and a collaborative loss function is constructed based on the similarity of policy distribution.
[0040] During the integration and optimization phase, a joint objective function is constructed to comprehensively consider both local benefit optimization and strategy consistency among devices.
[0041] Based on the joint objective function, the parameters of all policy functions are updated synchronously by minimizing the joint objective function through the gradient descent method, generating a globally optimal set of coordinated charging and discharging policies, and distributing the corresponding charging and discharging policies to each energy storage device.
[0042] The beneficial effects of this invention are:
[0043] This invention proposes a method for maximizing energy storage revenue based on a policy gradient algorithm. This method effectively overcomes the problems of poor adaptability, insufficient policy stability, and weak coordination among multiple devices in existing energy storage control technologies when dealing with complex dynamic environments. By constructing a policy training input consisting of energy storage state feature vectors, charging and discharging actions, and reward signals, and introducing an information entropy regularization term for the policy distribution into the objective function, the method enhances the policy's exploration ability and action diversity in the early training stage, effectively avoiding the policy from getting trapped in local optima. As the entropy coefficient gradually decays, the policy can converge stably in the later stage, ensuring the continuity and stability of revenue optimization.
[0044] This invention also introduces a time-series modeling mechanism for historical behavior trajectories, uses a gated recurrent neural network structure to extract multi-cycle evolutionary features, constructs a joint state representation vector, and combines the advantage function to construct a periodic strategy loss function. This enables long-term dependency modeling of energy storage behavior and generation of periodic self-evolving strategies, improving the control strategy's ability to perceive and anticipate future profit trends.
[0045] Furthermore, this invention designs a hierarchical optimization structure for multi-energy storage device scenarios. Through the lower-level autonomous strategy training and the strategy similarity coordination mechanism in the upper-level graph structure, the effective transmission and fusion of strategy features between devices is carried out based on the attention mechanism, which further improves the global consistency and collaborative optimization level of charging and discharging control. The overall method has the advantages of strong structural scalability, high revenue stability, and excellent adaptability to dynamic environments, which can effectively improve the economy and regulation capabilities of energy storage systems in the electricity market. Attached Figure Description
[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0047] Figure 1 This is a flowchart of a method for maximizing energy storage revenue based on a policy gradient algorithm proposed in this invention. Detailed Implementation
[0048] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0049] refer to Figure 1 A method for maximizing energy storage revenue based on a policy gradient algorithm includes the following steps:
[0050] Collect energy storage device status parameters, real-time electricity market price data, and load forecast data; preprocess the collected data to generate preprocessed data.
[0051] Feature extraction is performed on the preprocessed data to generate an energy storage state feature vector. The energy storage state feature vector is used as the system state input to initialize the charging and discharging strategy parameters and generate the initial charging and discharging strategy.
[0052] Based on the initial charge and discharge strategy, the charge and discharge actions for this cycle are generated, and the charge and discharge actions are applied to the energy storage device to perform the charge and discharge operations. The results of the charge and discharge operations are collected, and charge and discharge feedback information is generated.
[0053] By combining charging and discharging feedback information with charging and discharging actions, and calculating a reward signal based on the difference between immediate and expected benefits, the reward signal, energy storage state feature vector, and charging and discharging actions are used to update the charging and discharging strategy parameters to generate an updated charging and discharging strategy.
[0054] By utilizing the time-series dependencies of energy storage state feature vectors, charging and discharging actions, and reward signal sequences over multiple cycles, feedback optimization is performed on future cycle returns to generate a cycle-evolving charging and discharging strategy.
[0055] In multi-energy storage device scenarios, the energy storage state feature vectors, charging and discharging actions, and reward signals of each energy storage device are used for lower-level autonomous strategy optimization and upper-level device coordination to generate a globally optimal collaborative charging and discharging strategy.
[0056] This invention systematically improves the efficiency of energy storage control revenue optimization by constructing a complete process structure from data acquisition, state feature modeling, charging and discharging strategy generation and updating, periodic self-evolution optimization, to multi-device collaborative control. This method combines a strategy gradient algorithm to achieve continuous learning and adaptive adjustment of charging and discharging strategies, enabling dynamic response to market price fluctuations and load forecast changes. It ensures that the energy storage system continuously outputs high-yield strategies under different cycles and complex scenarios, and has the advantages of full-cycle, adaptive, and scalable operation.
[0057] In this embodiment, the energy storage device status parameters include state of charge, voltage, current, charging power and maximum charging power, discharging power and maximum discharging power, temperature and cycle life status. The preprocessing includes time synchronization, outlier detection, missing value completion, noise filtering and normalization.
[0058] This invention clarifies the composition of the state parameters of energy storage devices, covering charging and discharging electrical parameters, ambient temperature, and cycle life information. Combined with synchronization, filtering, and normalization operations in the data preprocessing process, it ensures the integrity and trainability of input features. This reduces the interference of data errors on policy training, provides stable basic data for subsequent feature extraction and policy modeling, and improves the accuracy and robustness of policy generation.
[0059] In this embodiment, the generation of the initial charge-discharge strategy specifically includes:
[0060] The preprocessed data is arranged in chronological order to construct a multidimensional data sequence with multiple consecutive time steps. Each time step includes energy storage device status parameters, real-time electricity market price data, and load forecast data.
[0061] Feature extraction is performed on multidimensional data sequences. Short-term dynamic change features between time steps are extracted through a one-dimensional convolution structure. The convolution results are then activated and compressed to obtain the feature response within a local time segment.
[0062] The feature response within a local time segment is input into a bidirectional long short-term memory neural network, and recursive feature encoding is performed simultaneously in both the forward and reverse directions of time to capture the trend pattern in the long time series. Finally, the corresponding energy storage state feature vector is output at the current time step.
[0063] The energy storage state feature vector is used as the system state input, and the charging and discharging strategy parameters are initialized.
[0064] The charging and discharging strategy parameters are a set of learnable parameters used to control the action distribution in the strategy gradient. The initialization method is to sample from a random distribution with a mean of zero and a fixed standard deviation to generate initial charging and discharging strategy parameters, ensuring that the strategy has sufficient exploration capabilities.
[0065] The energy storage state feature vector and the initial charge-discharge strategy parameters are input into the strategy network. The initial charge-discharge strategy parameters are used to initialize the strategy network. The strategy network is a feedforward neural network with a two-layer fully connected structure. The first layer uses the modified linear unit as the activation function, and the second layer uses the hyperbolic tangent function as the activation function. The output is the initial charge-discharge strategy under the current system state.
[0066] The initial charge and discharge strategy is a control output generated by a feedforward neural network based on the energy storage state feature vector and the initial strategy parameter input. The control output includes the charging power setpoint, the discharging power setpoint, the action time, and the action selection probability distribution, which are used to indicate the charging and discharging actions of the current cycle and their execution intensity.
[0067] This invention employs one-dimensional convolution combined with a bidirectional long short-term memory network for sequence modeling, enabling deep feature extraction of energy storage system states across multiple time scales. This ensures high sensitivity to market fluctuations and system trends from the initial strategy generation stage. By randomly initializing strategy parameters and inputting them into a feedforward neural network to generate initial charging and discharging strategies, the diversity and exploration capability of the strategy search space are effectively enhanced, improving the applicability and yield of the initial strategy under complex conditions.
[0068] In this embodiment, the generation of the charge / discharge feedback information specifically includes:
[0069] The charging and discharging actions for the current cycle are generated based on the initial charging and discharging strategy. The charging and discharging power is used as a set of control parameters to describe the charging and discharging power and its duration of action of the energy storage device in this cycle.
[0070] The generated charging and discharging actions are applied to the energy storage device, and the actual charging and discharging operations are executed according to the control parameters;
[0071] After the charging and discharging operation is completed, the energy storage state feature vector at the end of the charging and discharging operation is collected;
[0072] The immediate revenue for the current period is calculated by combining the charging and discharging actions with the real-time price of the electricity market. The immediate revenue is the economic return obtained by the energy storage device through charging and discharging during the operation, which is obtained by multiplying the electricity by the real-time electricity price.
[0073] The energy storage state feature vector after charging and discharging is combined with the immediate benefits to form charging and discharging feedback information. This feedback information is used to reflect the effect of charging and discharging actions within the current cycle and serves as the basis for optimizing charging and discharging strategies.
[0074] This invention constructs a feedback mechanism that links charging and discharging actions with real-time market data. By combining the changes in energy storage state after the action is executed, a complete charging and discharging feedback information chain is formed. This mechanism can comprehensively reflect the effect of action execution and market reaction, provide accurate evaluation indicators for strategy optimization, and ensure that the direction of strategy optimization is consistent with actual returns, thereby improving the accuracy of strategy adjustment and the ability to guide returns.
[0075] In this embodiment, the generation of the updated charging and discharging strategy specifically includes:
[0076] The reward signal is calculated based on the difference between the immediate return and the expected return for the current period.
[0077] A policy training input consisting of energy storage state feature vectors, charging / discharging actions, and reward signals is constructed. The policy network is then used to optimize the charging / discharging policy parameters, and an optimization objective function is constructed.
[0078]
[0079] Where J(θ) represents the optimization objective function, θ represents the charging / discharging strategy parameters, and H(π) represents the charging / discharging strategy parameters. θ ) represents the information entropy of the policy distribution output by the policy function, β represents the adjustment coefficient of the entropy regularization term, and s t Let a represent the energy storage state eigenvector. t Indicates the discharge action, π θ (a t |s t ) represents the current policy function, E represents the mathematical expectation, and r t This indicates a reward signal;
[0080] The adjustment coefficient of the entropy regularization term gradually decreases as training progresses, promoting exploration in the early stages and encouraging policy convergence in the later stages, which is suitable for the long-term stable benefit optimization objective in energy storage control.
[0081] β t =β0·exp(-λ·t);
[0082] Where, β t β0 represents the adjustment coefficient of the entropy regularization term at time step t, exp represents the natural exponential function, and λ represents the decay rate.
[0083] This invention introduces the information entropy term of the policy distribution output by the policy function into the policy gradient optimization objective function, which can enhance the exploration capability of the charging and discharging strategy during the training process. By regularizing the information entropy of the policy distribution, the policy network is prevented from getting trapped in local optima in the early training stage. This enables the energy storage device to continuously generate diverse and effective charging and discharging actions in the face of complex situations such as electricity price fluctuations and changes in load forecasting errors, expanding the space of optional actions. At the same time, in the subsequent training process, by gradually adjusting the entropy regularization coefficient, the strategy can be guided to gradually converge to a stable strategy with better performance and higher returns. This improves the charging and discharging flexibility and return stability of the entire energy storage system, ensuring that the policy optimization process takes into account both stability and robustness, and helps to achieve the full-cycle optimal control objective of maximizing energy storage returns.
[0084] The charging / discharging strategy parameters are updated using gradient ascent based on the optimization objective function;
[0085] The updated charge / discharge strategy parameters are applied to the energy storage state feature vector of the current cycle to generate an updated charge / discharge strategy for the next cycle's strategy iteration.
[0086] This invention introduces policy distribution information entropy as a regularization term into the policy gradient objective function. By adjusting the entropy coefficient, the balance between policy exploration and convergence is dynamically controlled, which significantly improves the generalization ability of the policy during the training phase. This mechanism can prevent the policy from getting stuck in local optima in the early stage and achieve fine-tuning of performance during the gradual convergence of the policy. It ensures that the generated charging and discharging strategies have diversity, stability and high profitability, effectively supporting the realization of the long-term optimal operation goal of the energy storage system.
[0087] In this embodiment, the generation of the periodically self-evolving charging and discharging strategy specifically includes:
[0088] Construct a historical behavior trajectory sequence to obtain energy storage state feature vector sequences, charging and discharging action sequences, and reward signal sequences over multiple consecutive periods;
[0089] The three types of sequences are input into the time series modeling structure, and the time series state representation under each period is extracted by the gated recurrent unit network to form a joint state representation vector sequence. The joint state representation vector sequence reflects the evolution of energy storage state, action execution trajectory and reward feedback change trend.
[0090] Based on the current periodic joint state representation vector, the charging and discharging policy probability distribution π is generated using the policy function. θ (a t |s t ,h t And combine the advantage function to construct the policy loss function:
[0091]
[0092] Where θ represents the parameters of the policy function, G(θ) represents the policy loss function, and A t π represents the dominance function value in period t. θ (a t |s t ,h t ) represents the policy probability distribution, describing the joint state representation vector h. t The energy storage state feature vector is s t At that time, charging and discharging actions are taken. t The probability, logπ θ (a t |s t ,h t H(π) represents the logarithm of the policy probability, λ represents the entropy regularization coefficient, and H(π) represents the logarithm of the policy probability. θ (·|s t ,h t )) represents the entropy value of the strategy probability distribution, and T represents the total number of cycles;
[0093] The policy loss function is trained using gradient ascent, the parameters of the policy function are updated, and a sliding time window mechanism is introduced in each update to control the historical behavior trajectory within the most recent k cycles as the basis for optimization, thereby improving the stability and generalization ability of the policy on the time scale.
[0094] The updated strategy function parameters are used in the strategy generation process of the next cycle. By combining the energy storage state feature vector of the latest cycle and the joint state representation vector corresponding to the current cycle, the cycle-evolving charging and discharging strategy of the current cycle is generated, ensuring the continuity, adaptability and optimization of the strategy.
[0095] This invention constructs behavioral trajectories for multiple cycles, extracts cross-cycle evolution patterns based on time series modeling structures, and combines joint state representation and advantage functions to construct a periodic self-evolutionary strategy optimization mechanism, which enhances the adaptability of the strategy in the time dimension. This mechanism can capture the inherent patterns of energy storage system behavior evolution over time, dynamically adjust the strategy output, and effectively improve the stability of the strategy's returns and control accuracy in the face of long-term dynamic changes.
[0096] In this embodiment, the generation of the globally optimal cooperative charging and discharging strategy specifically includes:
[0097] In an operation scenario involving multiple energy storage devices, the energy storage state feature vector, charging and discharging actions in the current cycle, and corresponding reward signals of each energy storage device are extracted. A hierarchical optimization structure consisting of autonomous optimization of lower-level devices and coordination between upper-level devices is constructed. Based on the fusion optimization results, the globally optimal collaborative charging and discharging strategy is output and fed back to each energy storage device for execution.
[0098] In the lower-level autonomous optimization process, for the i-th energy storage device, a policy function is constructed with the energy storage state feature vector, charging and discharging actions, and reward signals as inputs. The cumulative objective function is constructed by maximizing the cumulative discount reward.
[0099]
[0100] Among them, J i (θ i () represents the cumulative objective function, describing the i-th device's performance in the policy function. The expected cumulative reward is as follows: Indicating in strategy The expected value is given by γ, where γ represents the discount factor, T represents the total number of optimization cycles, and θ represents the expected value. i This represents the policy function parameters for the i-th device. This represents the reward signal obtained at time step t;
[0101] In the upper-level collaborative optimization process, all energy storage devices are constructed as corresponding nodes in a graph structure. The energy storage state feature vector, charging and discharging actions, and reward signals are used as node attributes. Edge weights are defined using the coupling relationship between each pair of devices. Aggregated features are obtained through a graph attention mechanism, and a collaborative loss function is constructed based on the similarity of policy distribution.
[0102]
[0103] Among them, L coop Let N represent the collaborative loss function, and N represent the number of energy storage devices. Let α represent the set of adjacent devices of the i-th device. ijThis represents the attention weight between the i-th device and the j-th device calculated by the graph attention mechanism. This represents the Kullback-Leibler divergence between the policy distributions of the i-th and j-th devices;
[0104] The coupling relationship between the devices is defined by strategy similarity, specifically using Kullback-Leibler divergence to measure the difference in action distribution of different energy storage devices under similar conditions. The higher the strategy similarity, the more consistent the strategy tendencies between the devices, and the larger the corresponding edge weight.
[0105] The attention weight is obtained by linearly mapping and splicing the node attributes between energy storage devices, using a modified linear unit function with leakage to calculate the unnormalized attention score, and then normalizing it in the device adjacency domain using the Softmax function. It is used to measure the weight ratio of neighboring devices in the process of policy information transmission.
[0106] In the fusion optimization phase, a joint objective function is constructed that comprehensively considers both local benefit optima and inter-device policy consistency:
[0107]
[0108] Among them, L total ρ represents the joint objective function, and ρ represents the weighting coefficient of the coordination loss term, which is used to adjust the balance between local optimization and global consistency.
[0109] Based on the joint objective function, the parameters of all policy functions are updated synchronously by minimizing the joint objective function through the gradient descent method, generating a globally optimal set of coordinated charging and discharging policies, and distributing the corresponding charging and discharging policies to each energy storage device.
[0110] This invention constructs a hierarchical optimization structure that combines upper and lower layers. At the device layer, it achieves individual optimality of strategies, while at the system layer, it introduces graph structures and graph attention mechanisms to model the policy coupling relationship between devices. By jointly optimizing the objective function, it achieves a synergistic improvement in policy consistency and local benefits. This method significantly enhances the overall control coordination in multi-device systems, improves collaborative discharge efficiency and benefit integration capabilities, and has good scalability and cluster control adaptability.
[0111] Example 1:
[0112] To verify the feasibility of this invention in practice, it was applied to a comprehensive energy dispatch center with multiple energy storage devices deployed. This center, which focuses on electricity demand response management for industrial and commercial loads, includes several energy storage devices of varying capacities and types, distributed across the power distribution systems of multiple plants. It possesses real-time electricity price access capabilities, short-term load forecasting capabilities, and a unified energy storage dispatch interface. This invention, by deploying a profit maximization control system driven by a strategy gradient algorithm, realizes closed-loop control logic from state perception, strategy generation, action execution to strategy updating. This effectively solves the problems of instability and insufficient optimization capabilities of existing energy storage dispatch schemes in dealing with multi-device collaboration, dynamic price fluctuations, and prediction error disturbances.
[0113] In this implementation environment, the data acquisition module first acquires real-time operational information of each energy storage device in a unified format, including voltage, current, state of charge, charging and discharging power, temperature, and cycle life. Simultaneously, it accesses real-time electricity price data released by the electricity market and local load forecast information. A preprocessing stage completes data time alignment, anomaly removal, missing data completion, and normalization. Subsequently, the feature extraction module inputs the multi-dimensional time-series data into a deep time-series sensing network to extract state vectors containing energy storage evolution trends and market environment characteristics. This extracts these state vectors and initializes a feedforward neural strategy network, outputting initial charging and discharging control actions. These actions are calculated directly by the strategy network based on the state vectors and initialization parameters, including the magnitude of charging and discharging power, duration of action, and probability distribution.
[0114] After the control action is applied to the device, the system records the action execution effect and the economic benefits obtained in real time. Combining the electricity price change trend, action deviation and historical performance, the system performs gradient updates on the current strategy function. The update process uses an entropy regularization term to control the exploration capability. At the same time, a strategy loss function is constructed to make the system gradually converge to the optimal solution of stable income. In order to improve the strategy's responsiveness to changes in time scale, this embodiment introduces a sliding time window mechanism. It uses the state, action and reward trajectory in multiple cycles to construct a periodic self-evolving strategy, so that the strategy function has the ability to predict and adjust for future trends.
[0115] In actual operation involving multiple energy storage devices, this embodiment adopts a hierarchical optimization structure, modeling each energy storage device as an independent strategy unit. Local strategy optimization is first completed at the lower layer, and then the strategy information between devices is aggregated through a graph attention mechanism to achieve upper-layer collaborative optimization under the constraint of strategy distribution consistency. The edge weights in the graph structure are automatically updated through a strategy similarity measurement function, and the attention mechanism weights the node information to ensure that the final fused strategy takes into account both local optimality and global coordination. According to actual operation feedback, the system stability is significantly improved, the strategy response speed is faster, the revenue volatility is significantly reduced, the scheduling between devices tends to be coordinated and consistent, and the overall energy utilization efficiency is improved.
[0116] To verify the performance of the present invention in practice, it was compared with traditional methods, and the results are shown in Table 1.
[0117] Table 1. Comparison between policy gradient-based energy storage optimization methods and traditional methods.
[0118] index Method of the present invention Traditional methods Strategy stability index 0.92 0.67 Standard deviation of returns 6.8 12.5 Number of rounds for strategy convergence 105 180 Average return improvement rate 18.3％ 0％ Multi-device policy consistency score 0.85 0.58 Load response accuracy 92.6％ 81.2％
[0119] Regarding the strategy stability index, the method of this invention exhibits higher consistency in multiple rounds of charging and discharging strategy updates, with a value of 0.92, which is better than the 0.67 of the traditional method. This improvement is mainly due to the periodic self-evolution mechanism and sliding time window processing method introduced in this invention, which effectively smooths strategy fluctuations and enables a relatively stable strategy output even in an environment of frequent changes in electricity prices, thereby enhancing the robustness of energy storage control.
[0120] Regarding the standard deviation of returns, the return volatility of this invention decreased to 6.8 after multiple runs, which is significantly better than the 12.5 of the traditional method. This indicates that the charging and discharging strategy generated by this invention not only has higher returns, but also significantly reduces the uncertainty of short-term returns caused by market changes. This is because its reward signal construction integrates expected value comparison, entropy regularization control and multi-period feedback mechanism, which has a stronger risk mitigation capability.
[0121] Regarding the number of policy convergence rounds, the method of this invention achieves stable policy convergence within 105 training rounds, while traditional methods require more than 180 rounds. This improvement in convergence speed stems from the bidirectional temporal modeling and gating mechanism adopted in this invention, which enables the energy storage state and action feedback features to form clear gradient directions in the early stages of policy training, shortening invalid attempts in the policy search process and thus significantly improving training efficiency.
[0122] In terms of average return improvement, traditional methods show virtually no return growth, while the method of this invention achieves an average improvement of 18.3%. This growth stems from the introduction of advantage function estimation and multi-scale time series modeling mechanisms into the strategy objective function, which makes the charging and discharging behavior more closely matched with market fluctuations, enabling the capture of better arbitrage opportunities and achieving the maximum economic return in a balanced manner across different cycles.
[0123] In terms of multi-device policy consistency score, this invention scores 0.85, far exceeding the 0.58 of the traditional method. This advantage is attributed to the fact that this invention uses a graph attention mechanism to jointly optimize the policies among multiple energy storage nodes, enabling different devices to make complementary decisions collaboratively when facing similar market situations, reducing policy conflicts and resource competition, thereby achieving consistency and efficiency in scheduling among devices.
[0124] In terms of load response accuracy, the method of this invention achieves 92.6%, a significant improvement over the 81.2% of traditional methods. This is fundamentally due to the introduction of a dynamic adjustment mechanism based on load forecasting during the strategy generation process. Combined with real-time feedback on charging and discharging actions, this mechanism can promptly compensate for forecast errors, thereby ensuring a rapid and accurate response of the energy storage system to load demands and effectively improving the overall service quality of the system.
[0125] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for maximizing energy storage revenue based on a policy gradient algorithm, characterized in that, Includes the following steps: Collect energy storage device status parameters, real-time electricity market price data, and load forecast data; preprocess the collected data to generate preprocessed data. Feature extraction is performed on the preprocessed data to generate an energy storage state feature vector. The energy storage state feature vector is used as the system state input to initialize the charging and discharging strategy parameters and generate the initial charging and discharging strategy. Based on the initial charge and discharge strategy, the charge and discharge actions for this cycle are generated, and the charge and discharge actions are applied to the energy storage device to perform the charge and discharge operations. The results of the charge and discharge operations are collected, and charge and discharge feedback information is generated. By combining charging and discharging feedback information with charging and discharging actions, and calculating a reward signal based on the difference between immediate and expected benefits, the reward signal, energy storage state feature vector, and charging and discharging actions are used to update the charging and discharging strategy parameters to generate an updated charging and discharging strategy. By utilizing the time-series dependencies of energy storage state feature vectors, charging and discharging actions, and reward signal sequences over multiple cycles, feedback optimization is performed on future cycle returns to generate a cycle-evolving charging and discharging strategy. In multi-energy storage device scenarios, the energy storage state feature vectors, charging and discharging actions, and reward signals of each energy storage device are used for lower-level autonomous strategy optimization and upper-level device coordination to generate a globally optimal collaborative charging and discharging strategy. The generation of the globally optimal cooperative charging and discharging strategy specifically includes: In an operation scenario involving multiple energy storage devices, the energy storage state feature vector, charging and discharging actions in the current cycle, and corresponding reward signals of each energy storage device are extracted. A hierarchical optimization structure consisting of autonomous optimization of lower-level devices and coordination between upper-level devices is constructed. Based on the fusion optimization results, the globally optimal collaborative charging and discharging strategy is output and fed back to each energy storage device for execution. During the lower-level autonomous optimization process, for the first... For each energy storage device, a strategy function is constructed with the energy storage state feature vector, charging and discharging actions, and reward signals as inputs. A cumulative objective function is constructed by maximizing the cumulative discount reward. In the upper-level collaborative optimization process, all energy storage devices are constructed as corresponding nodes in the graph structure. The energy storage state feature vector, charging and discharging actions and reward signals are used as node attributes. The edge weights are defined using the coupling relationship between each pair of devices. Aggregated features are obtained through the graph attention mechanism, and a collaborative loss function is constructed based on the similarity of policy distribution. During the integration and optimization phase, a joint objective function is constructed to comprehensively consider both local benefit optimization and strategy consistency among devices. Based on the joint objective function, the parameters of all policy functions are updated synchronously by minimizing the joint objective function through the gradient descent method, generating a globally optimal set of coordinated charging and discharging policies, and distributing the corresponding charging and discharging policies to each energy storage device.
2. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The energy storage device status parameters include state of charge, voltage, current, charging power and maximum charging power, discharging power and maximum discharging power, temperature and cycle life status. The preprocessing includes time synchronization, outlier detection, missing value completion, noise filtering and normalization.
3. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the initial charge / discharge strategy specifically includes: The preprocessed data is arranged in chronological order to construct a multidimensional data sequence with multiple consecutive time steps. Each time step includes energy storage device status parameters, real-time electricity market price data, and load forecast data. Feature extraction is performed on multidimensional data sequences. Short-term dynamic change features between time steps are extracted through a one-dimensional convolution structure. The convolution results are then activated and compressed to obtain the feature response within a local time segment. The feature response within a local time segment is input into a bidirectional long short-term memory neural network, and recursive feature encoding is performed simultaneously in both the forward and reverse directions of time to capture the trend pattern in the long time series. Finally, the corresponding energy storage state feature vector is output at the current time step. The energy storage state feature vector is used as the system state input, and the charging and discharging strategy parameters are initialized. The energy storage state feature vector and the initial charge / discharge strategy parameters are input into the strategy network. The initial charge / discharge strategy parameters are used to initialize the strategy network. The strategy network is a feedforward neural network with a two-layer fully connected structure. The first layer uses a modified linear unit as the activation function, and the second layer uses a hyperbolic tangent function as the activation function. The output is the initial charge / discharge strategy under the current system state.
4. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the charge / discharge feedback information specifically includes: The charging and discharging actions for the current cycle are generated based on the initial charging and discharging strategy; The generated charging and discharging actions are applied to the energy storage device, and the actual charging and discharging operations are executed according to the control parameters; After the charging and discharging operation is completed, the energy storage state feature vector at the end of the charging and discharging operation is collected; The immediate revenue for the current period is calculated by combining the charging and discharging actions with the real-time price of the electricity market. The immediate revenue is the economic return obtained by the energy storage device through charging and discharging during the operation, which is obtained by multiplying the electricity by the real-time electricity price. The energy storage state feature vector after charging and discharging is combined with the immediate benefits to form charging and discharging feedback information.
5. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the updated charge / discharge strategy specifically includes: The reward signal is calculated based on the difference between the immediate return and the expected return for the current period. A policy training input consisting of energy storage state feature vectors, charging and discharging actions, and reward signals is constructed. The charging and discharging policy parameters are optimized using a policy network, and an optimization objective function is constructed. The charging / discharging strategy parameters are updated using gradient ascent based on the optimization objective function; The updated charge / discharge strategy parameters are applied to the energy storage state feature vector of the current cycle to generate an updated charge / discharge strategy for the next cycle's strategy iteration.
6. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the periodic self-evolving charging and discharging strategy specifically includes: Construct a historical behavior trajectory sequence to obtain energy storage state feature vector sequences, charging and discharging action sequences, and reward signal sequences over multiple consecutive periods; The three types of sequences are input into the time series modeling structure, and a gated recurrent unit network is used to extract the temporal state representation under each period to form a joint state representation vector sequence. Based on the current period joint state representation vector, the charging and discharging policy probability distribution is generated using the policy function, and the policy loss function is constructed by combining the advantage function. The policy loss function is trained using gradient ascent, and its parameters are updated accordingly. A sliding window mechanism is introduced in each update round to control the most recent... The historical behavior trajectory within a certain period serves as the basis for optimization; The updated strategy function parameters are used in the strategy generation process of the next cycle. By combining the energy storage state feature vector of the latest cycle and the joint state representation vector corresponding to the current cycle, the cycle-evolving charging and discharging strategy of the current cycle is generated.