Energy storage income maximization method based on strategy gradient algorithm

Abstract

The invention discloses an energy storage income maximization method based on a strategy gradient algorithm, and the method comprises the following steps: collecting energy storage equipment state parameters, power market real-time price data and load prediction data, and carrying out the preprocessing; feature extraction is carried out, charging and discharging strategy parameters are initialized, and an initial charging and discharging strategy is generated; according to the initial charging and discharging strategy, generating a charging and discharging action of a cost cycle, executing the charging and discharging operation, and generating charging and discharging feedback information; a reward signal is calculated, and the reward signal, the energy storage state feature vector and the charging and discharging action are used for updating charging and discharging strategy parameters; performing feedback optimization on future cycle income by using a time sequence dependency relationship; and performing lower-layer autonomous strategy optimization and upper-layer equipment coordination, and generating a globally optimal cooperative charging and discharging strategy. The strategy gradient and graph optimization method is fused, energy storage cooperative control is achieved, and the method has the advantages of being high in income, high in adaptability and stable in strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent energy storage control, and in particular to a method for maximizing energy storage revenue based on a policy gradient algorithm. BACKGROUND

[0002] In the current dispatching and operation process of the power system, the energy storage device is widely used in new energy grid connection, electricity price arbitrage, peak-valley regulation and other scenarios as an important means to regulate the balance between energy supply and demand. The traditional energy storage control method mainly relies on rule-based strategies or model predictive control methods, which usually formulate the charging and discharging plan through setting fixed thresholds, linear programming or heuristic methods. Such methods have great limitations in parameter setting, adaptability and response to uncertain factors, and are difficult to effectively handle real-time electricity price fluctuations, load prediction errors and multi-device operation coupling and other complex factors.

[0003] In recent years, with the rapid development of artificial intelligence technology, reinforcement learning methods have been gradually introduced into the field of energy storage control. By constructing a policy function and optimizing based on historical interaction data, reinforcement learning has strong self-learning and environmental adaptability. However, the existing reinforcement learning in energy storage dispatching still faces many challenges: first, most methods are based on single-period optimization, ignoring the time sequence dependence of energy storage state and the coupling relationship of long-term revenue, making it difficult to realize the periodic evolution and continuous optimization of charging and discharging strategies; second, in the collaborative control of multiple energy storage devices, it is difficult to share information between strategies, which easily leads to local optimization; third, the strategy optimization process lacks control over the stability of the strategy distribution and the generalization ability, resulting in unstable training process, large strategy fluctuations and reduced revenue reliability. SUMMARY

[0004] One object of the present application is to provide a method for maximizing energy storage revenue based on a policy gradient algorithm. The present application combines policy gradient and graph optimization methods to achieve energy storage collaborative control, with the advantages of high revenue, strong adaptability and stable strategy.

[0005] According to the method for maximizing energy storage revenue based on a policy gradient algorithm, the following steps are included:

[0006] Collecting energy storage device state parameters, real-time electricity market price data and load prediction data, preprocessing the collected data to generate preprocessed data;

[0007] Extracting features from the preprocessed data to generate an energy storage state feature vector, inputting the energy storage state feature vector as the system state, initializing the charging and discharging strategy parameters, and generating an initial charging and discharging strategy;

[0008] According to the initial charging and discharging strategy, a charging and discharging action of the current period is generated, the charging and discharging action is applied to the energy storage device, a charging and discharging operation is performed, and a charging and discharging operation result is collected to generate charging and discharging feedback information;

[0009] The charging and discharging feedback information is combined with the charging and discharging action, and a reward signal is calculated according to the difference between the immediate reward and the expected reward. The reward signal, the energy storage state feature vector and the charging and discharging action are applied to update the charging and discharging strategy parameters to generate an updated charging and discharging strategy;

[0010] The energy storage state feature vectors, charging and discharging actions and reward signals in multiple periods are used to perform feedback optimization on future period rewards using time sequence dependency to generate a period self-evolution charging and discharging strategy;

[0011] In a multi-energy storage device scenario, the energy storage state feature vectors, charging and discharging actions and reward signals of each energy storage device are respectively optimized by a lower layer autonomous strategy and coordinated between upper layer devices to generate a globally optimal collaborative charging and discharging strategy.

[0012] Optionally, the energy storage device state parameters include state of charge, voltage, current, charging power and maximum charging power, discharging power and maximum discharging power, temperature and cycle life state, and the preprocessing includes time synchronization, outlier detection, missing value completion, noise filtering and normalization.

[0013] Optionally, the generation of the initial charging and discharging strategy specifically includes:

[0014] The preprocessed data is arranged in time sequence to construct a multi-dimensional data sequence of multiple consecutive time steps, each time step containing energy storage device state parameters, real-time price data of the power market and load prediction data;

[0015] Feature extraction is performed on the multi-dimensional data sequence, short-term dynamic change features between time steps are extracted through a one-dimensional convolution structure, and the convolution results are activated and compressed to obtain feature responses in local time segments;

[0016] The feature responses in the local time segments are input into a bidirectional long short-term memory neural network, and recursive feature encoding is performed in both time forward and reverse directions to capture trend patterns in long time sequences, and finally the corresponding energy storage state feature vector is output at the current time step;

[0017] The energy storage state feature vector is input as a system state, and the charging and discharging strategy parameters are initialized;

[0018] The energy storage state feature vector and initial charging and discharging strategy parameters are input into the strategy network, the initial charging and discharging strategy parameters are used to initialize the strategy network, the strategy network is a feedforward neural network comprising two layers of full connection structure, the first layer adopts a rectified linear unit as an activation function, and the second layer adopts a hyperbolic tangent function as an activation function, and outputs the initial charging and discharging strategy under the current system state.

[0019] Optionally, the generation of the charging and discharging feedback information specifically comprises:

[0020] The charging and discharging action of the current period is generated based on the initial charging and discharging strategy.

[0021] The generated charging and discharging action is applied to the energy storage device, and an actual charging and discharging operation is performed according to the control parameters.

[0022] After the charging and discharging operation is completed, the energy storage state feature vector at the end of the charging and discharging is collected.

[0023] The real-time price of the power market is combined with the charging and discharging action to calculate the immediate income in the current period, the immediate income is the economic return obtained by the energy storage device through charging and discharging during the action, and is obtained by integrating the power multiplied by the real-time price;

[0024] The energy storage state feature vector after charging and discharging and the immediate income are combined to form the charging and discharging feedback information.

[0025] Optionally, the generation of the updated charging and discharging strategy specifically comprises:

[0026] The reward signal is calculated based on the difference between the immediate income and the expected income of the current period.

[0027] The strategy training input composed of the energy storage state feature vector, the charging and discharging action and the reward signal is constructed, the charging and discharging strategy parameters are optimized by using the strategy network, and an optimization objective function is constructed.

[0028] The gradient ascent update is performed on the charging and discharging strategy parameters according to the optimization objective function.

[0029] The updated charging and discharging strategy parameters are applied to the energy storage state feature vector of the current period to generate an updated charging and discharging strategy, which is used for strategy iteration in the next period.

[0030] Optionally, the generation of the periodically evolving charging and discharging strategy specifically comprises:

[0031] The historical behavior trajectory sequence is constructed, and the energy storage state feature vector sequence, the charging and discharging action sequence and the reward signal sequence in multiple consecutive periods are obtained.

[0032] The three types of sequence are input into a time sequence modeling structure, a gated recurrent unit network is used to extract a time sequence state representation under each cycle to form a joint state representation vector sequence;

[0033] Based on the joint state representation vector of the current cycle, a strategy function is used to generate a charging and discharging strategy probability distribution, and a policy loss function is constructed in combination with an advantage function;

[0034] The policy loss function is trained by using a gradient ascent method, the parameters of the strategy function are updated, and a sliding time window mechanism is introduced in each round of update to control the historical behavior trajectory in the last k cycles as the optimization basis.

[0035] The updated strategy function parameters are used in the strategy generation process of the next cycle, the energy storage state feature vector of the latest cycle and the joint state representation vector corresponding to the current cycle are combined to generate the cycle self-evolution charging and discharging strategy of the current cycle.

[0036] Optionally, the generation of the globally optimal collaborative charging and discharging strategy specifically comprises:

[0037] In the operation scene containing multiple energy storage devices, the energy storage state feature vector of each energy storage device, the charging and discharging action in the current cycle and the corresponding reward signal are extracted respectively, a hierarchical optimization structure composed of lower device autonomous optimization and upper device coordination is constructed, and a globally optimal collaborative charging and discharging strategy is output based on the fusion optimization result and fed back to each energy storage device for execution.

[0038] In the lower layer autonomous optimization process, for the i-th energy storage device, a strategy function is constructed with the energy storage state feature vector, the charging and discharging action and the reward signal as input, and a cumulative target function is constructed by maximizing the cumulative discounted reward.

[0039] In the upper layer collaborative optimization process, all energy storage devices are constructed as corresponding nodes in a graph structure, the energy storage state feature vector, the charging and discharging action and the reward signal are taken as node attributes, the coupling relationship between each pair of devices is used to define edge weights, the aggregated features are obtained through a graph attention mechanism, and a collaborative loss function is constructed based on the strategy distribution similarity.

[0040] In the fusion optimization stage, a joint target function is constructed to comprehensively consider the local revenue optimality and the strategy consistency between devices.

[0041] Based on the joint target function, all strategy function parameters are synchronously updated by using a gradient descent method to minimize the joint target function, a globally optimal collaborative charging and discharging strategy set is generated, and the corresponding charging and discharging strategy is distributed to each energy storage device.

[0042] The beneficial effects of the present application are:

[0043] The energy storage benefit maximization method based on the policy gradient algorithm fully overcomes the poor adaptability, insufficient policy stability and weak multi-device coordination of existing energy storage control technologies in dealing with complex dynamic environments, the policy training input is constructed by the energy storage state feature vector, the charging and discharging action and the reward signal, and the information entropy regular term of the policy distribution is introduced into the target function, the exploration ability and action diversity of the policy in the early training stage are improved, the policy is effectively prevented from falling into local optimum, and with the gradual decay of the entropy coefficient, the policy can stably converge in the later stage, and the continuity and stability of benefit optimization are ensured.

[0044] The application also introduces a time sequence modeling mechanism of historical behavior trajectory, uses a gated recurrent neural network structure to extract multi-period evolution features, constructs a joint state representation vector, and constructs a periodic policy loss function combined with an advantage function, realizes long-term dependence modeling of energy storage behavior and generation of periodic self-evolution policy, and improves the perception ability and foresight of the control policy to the future benefit change trend.

[0045] In addition, the application designs a hierarchical optimization structure for a multi-energy storage device scene, through the lower layer autonomous strategy training and the strategy similarity coordination mechanism in the upper layer graph structure, based on the attention mechanism, the strategy features between devices are effectively transmitted and fused, and the global consistency and collaborative optimization level of the charging and discharging control are further improved, and the overall method has the advantages of strong structure scalability, high benefit stability and excellent adaptability to dynamic environment, and can effectively improve the economy and regulation and control ability of the energy storage system in the power market. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, and are used to explain the application together with embodiments of the application, and do not constitute a limitation on the application. In the drawings:

[0047] Figure 1 A flowchart of the energy storage benefit maximization method based on the policy gradient algorithm proposed by the application. DETAILED DESCRIPTION

[0048] The application will now be described in further detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, and only schematically show the basic structure of the application, and therefore only show the components related to the application.

[0049] REFERENCE Figure 1 An energy storage benefit maximization method based on a policy gradient algorithm, comprising the following steps:

[0050] Collecting energy storage device state parameters, real-time price data of the power market and load prediction data, preprocessing the collected data to generate preprocessed data;

[0051] feature extraction is performed on the pretreated data to generate an energy storage state feature vector, the energy storage state feature vector is taken as a system state input, a charging and discharging strategy parameter is initialized, and an initial charging and discharging strategy is generated;

[0052] a charging and discharging action of the current period is generated according to the initial charging and discharging strategy, the charging and discharging action is applied to the energy storage device, a charging and discharging operation is performed, and a charging and discharging operation result is collected to generate charging and discharging feedback information;

[0053] The charging and discharging feedback information is combined with the charging and discharging action, the difference between the immediate income and the expected income is calculated, the reward signal is calculated, the reward signal, the energy storage state feature vector and the charging and discharging action are applied to update the charging and discharging strategy parameter, and an updated charging and discharging strategy is generated.

[0054] The energy storage state feature vectors, the charging and discharging actions and the reward signal sequences in multiple periods are used to perform feedback optimization on the future period income by using the time sequence dependence relationship, and a period self-evolution charging and discharging strategy is generated.

[0055] In a multi-energy storage device scenario, the energy storage state feature vectors, the charging and discharging actions and the reward signals of each energy storage device are respectively subjected to lower-layer autonomous strategy optimization and upper-layer inter-device coordination to generate a globally optimal collaborative charging and discharging strategy.

[0056] The present application improves the income optimization efficiency of energy storage control by constructing a whole process structure from data acquisition, state feature modeling, charging and discharging strategy generation and updating, period self-evolution optimization to multi-device collaborative control. The method combines the strategy gradient algorithm to realize continuous learning and adaptive adjustment of the charging and discharging strategy, which can dynamically respond to market price fluctuations and load prediction changes, ensuring that the energy storage system continuously outputs high-yield strategies in different periods and complex scenarios, and has the advantages of whole period, self-adaptation and scalability.

[0057] In the embodiment, the energy storage device state parameters include state of charge, voltage, current, charging power and maximum charging power, discharging power and maximum discharging power, temperature and cycle life state, and the preprocessing includes time synchronization, outlier detection, missing value completion, noise filtering and normalization.

[0058] The present application defines the composition of the energy storage device state parameters, including charging and discharging electrical parameters, environmental temperature and cycle life information, and combines synchronization, filtering and normalization operations in the data preprocessing process to ensure the integrity and trainability of the input features, which reduces the interference of data errors on strategy training, provides stable basic data for subsequent feature extraction and strategy modeling, and improves the accuracy and robustness of strategy generation.

[0059] In the embodiment, the generation of the initial charging and discharging strategy specifically includes:

[0060] The preprocessed data is arranged in time sequence to construct a multi-dimensional data sequence of multiple continuous time steps, each time step containing energy storage device state parameters, real-time power market price data and load prediction data;

[0061] Feature extraction is performed on the multi-dimensional data sequence, short-term dynamic change features between time steps are extracted through a one-dimensional convolution structure, and the convolution results are activated and compressed to obtain feature responses within a local time segment;

[0062] The feature responses within the local time segment are input into a bidirectional long short-term memory neural network, which simultaneously performs recursive feature encoding in the forward and reverse directions to capture trend patterns in long time sequences, and finally outputs the corresponding energy storage state feature vector at the current time step;

[0063] The energy storage state feature vector is input as the system state, and the charge and discharge strategy parameters are initialized;

[0064] The charge and discharge strategy parameters are a set of learnable parameters used to control the action distribution in the strategy gradient, and the initialization method is to sample from a random distribution with a mean of zero and a fixed standard deviation to generate initial charge and discharge strategy parameters, ensuring that the strategy has sufficient exploration ability;

[0065] The energy storage state feature vector and the initial charge and discharge strategy parameters are input into the strategy network, and the initial charge and discharge strategy parameters are used to initialize the strategy network, which is a feedforward neural network containing two fully connected structures, the first layer uses a rectified linear unit as the activation function, and the second layer uses a hyperbolic tangent function as the activation function, and outputs the initial charge and discharge strategy under the current system state.

[0066] The initial charge and discharge strategy is the control output generated by inputting the energy storage state feature vector and the initial strategy parameters into the feedforward neural network, and the control output includes the charge power set value, the discharge power set value, the action time and the action selection probability distribution, which is used to indicate the charge and discharge action and its execution intensity in the current period.

[0067] The present application uses one-dimensional convolution combined with bidirectional long short-term memory network for sequence modeling, realizes deep feature extraction of energy storage system state in multiple time scales, ensures high sensitivity to market fluctuations and system trends in the early stage of strategy generation, generates initial charge and discharge strategy by randomly initializing strategy parameters and inputting into feedforward neural network, effectively enhances the diversity and exploration ability of strategy search space, and improves the applicability and yield rate of initial strategy in complex state

[0068] In the embodiment, the generation of the charge and discharge feedback information specifically includes:

[0069] The initial charging and discharging strategy is used to generate a charging and discharging action of the current period, and the charging and discharging action is a set of control parameters for describing the charging and discharging power and the action time of the energy storage device in the current period.

[0070] The generated charging and discharging action is applied to the energy storage device, and actual charging and discharging operation is performed according to the control parameters;

[0071] After the charging and discharging operation is completed, the energy storage state feature vector at the end of the charging and discharging is collected;

[0072] The real-time price of the power market is combined with the charging and discharging action to calculate the immediate income in the current period, and the immediate income is the economic return obtained by the energy storage device through charging and discharging during the action execution, which is obtained by integrating the power and the real-time price;

[0073] The energy storage state feature vector after charging and discharging and the immediate income are combined to form charging and discharging feedback information, which is used to reflect the effect of the charging and discharging action in the current period and serves as a basis for optimization of the charging and discharging strategy.

[0074] The present application forms a complete charging and discharging feedback information link by constructing a feedback mechanism of the charging and discharging action associated with the real-time market data and combining the energy storage state change after the action execution, which can comprehensively reflect the action execution effect and market reaction, provide accurate evaluation indexes for strategy optimization, ensure the consistency between the strategy optimization direction and the actual income, and improve the accuracy of strategy adjustment and the income guiding ability.

[0075] In the embodiment, the generation of the updated charging and discharging strategy specifically includes:

[0076] The reward signal is calculated based on the difference between the immediate income and the expected income in the current period;

[0077] The strategy training input composed of the energy storage state feature vector, the charging and discharging action and the reward signal is constructed, the charging and discharging strategy parameters are optimized by using the strategy network, and the optimization objective function is constructed:

[0078]

[0079] Wherein, J(θ) represents the optimization objective function, θ represents the charging and discharging strategy parameters, H(π θ ) represents the information entropy of the strategy distribution output by the strategy function, β represents the adjustment coefficient of the entropy regular term, s t represents the energy storage state feature vector, a t represents the discharging action, π θ (a t |s t ) represents the current strategy function, E represents the mathematical expectation, and r t represents the reward signal.

[0080] The adjustment coefficient of the entropy regularization term gradually decreases with the training, promoting exploration in the early stage and encouraging strategy convergence in the later stage, suitable for the long-term stable optimization goal of energy storage control:

[0081] β t = β0·exp(-λ·t);

[0082] Wherein, β t represents the adjustment coefficient of the entropy regularization term at time step t, β0represents the initial entropy coefficient, exp represents the natural exponential function, and λ represents the decay rate;

[0083] The present application introduces the information entropy term of the strategy distribution of the strategy function output in the strategy gradient optimization objective function, which can improve the exploration ability of the charging and discharging strategy in the training process. Through the regularization processing of the information entropy of the strategy distribution, the strategy network is prompted to avoid falling into local optimum in the early training stage, so that the energy storage device can continuously generate diversified and effective charging and discharging actions under the complex situation of facing price fluctuations and load prediction error changes, expand the action space, and at the same time in the subsequent training process, by gradually adjusting the entropy regularization coefficient, the strategy can be guided to gradually converge to a stable strategy with better performance and higher income, improve the charging and discharging flexibility and income stability of the whole energy storage system, ensure that the strategy optimization process considers stability and robustness, and help to realize the whole cycle optimal control goal of maximizing the energy storage income;

[0084] According to the optimization objective function, gradient ascent update is performed on the charging and discharging strategy parameters;

[0085] The updated charging and discharging strategy parameters are applied to the energy storage state feature vector of the current period to generate an updated charging and discharging strategy for the next period of strategy iteration.

[0086] The present application introduces the strategy distribution information entropy as a regularization term in the strategy gradient objective function, dynamically controls the balance between strategy exploration and convergence by adjusting the entropy coefficient, significantly improves the generalization ability of the strategy in the training stage, prevents the strategy from falling into local optimum in the early stage, and realizes fine optimization of performance in the process of gradually converging the strategy, ensures that the generated charging and discharging strategy has diversity, stability and high income, and effectively supports the realization of the long-term optimal operation goal of the energy storage system.

[0087] In the embodiment, the generation of the periodically evolved charging and discharging strategy specifically includes:

[0088] A historical behavior trajectory sequence is constructed to obtain a sequence of energy storage state feature vectors, a sequence of charging and discharging actions and a sequence of reward signals in multiple consecutive periods;

[0089] The three types of sequence are input into a time sequence modeling structure, a gated recurrent unit network is used to extract a time sequence state representation in each cycle to form a joint state representation vector sequence, which reflects the energy storage state evolution, action execution trajectory and reward feedback change trend;

[0090] Based on the joint state representation vector of the current cycle, a charging and discharging strategy probability distribution π θ (a t |s t ,h t ) is generated by using a policy function, and a policy loss function is constructed by combining an advantage function:

[0091]

[0092] Where θ represents the parameters of the policy function, G(θ) represents the policy loss function, A t t represents the advantage function value of the tth cycle, π θ (a t |s t ,h t ) represents the strategy probability distribution, which describes the probability of taking the charging and discharging action a t when the joint state representation vector is h t and the energy storage state feature vector is s t , logπ θ (a t |s t ,h t ) represents the logarithmic value of the strategy probability, λ represents the entropy regularization term adjustment coefficient, H(π θ (·|s t ,h t )) represents the entropy value of the strategy probability distribution, and T represents the total number of cycles.

[0093] The policy loss function is trained by using gradient ascent method to update the parameters of the policy function, and a sliding time window mechanism is introduced in each round of update to control the historical behavior trajectory in the last k cycles as the optimization basis to improve the stability and generalization ability of the policy in the time scale.

[0094] The updated policy function parameters are used in the policy generation process of the next cycle, combined with the energy storage state feature vector of the latest cycle and the joint state representation vector corresponding to the current cycle, to generate the charging and discharging strategy of the cycle self-evolution of the current cycle, ensuring the continuity, adaptability and optimization of the strategy.

[0095] The application extracts the cross-period evolution rule based on the time sequence modeling structure by constructing multiple periodic behavior trajectories, and enhances the adaptability of the strategy in the time dimension by combining the joint state representation and the advantage function to construct a periodic self-evolution strategy optimization mechanism. The mechanism can capture the internal pattern of the behavior of the energy storage system evolving over time, dynamically adjust the strategy output, and effectively improve the revenue stability and control accuracy of the strategy in the face of long-term dynamic change scenarios.

[0096] In the embodiment, the generation of the globally optimal coordinated charging and discharging strategy specifically includes:

[0097] In the operation scenario containing multiple energy storage devices, the energy storage state feature vector, the charging and discharging action in the current period and the corresponding reward signal of each energy storage device are extracted respectively, a hierarchical optimization structure composed of lower device autonomous optimization and upper device coordination is constructed, and a globally optimal coordinated charging and discharging strategy is output based on the fusion optimization result and fed back to each energy storage device for execution;

[0098] In the lower layer autonomous optimization process, for the i-th energy storage device, a strategy function is constructed with the energy storage state feature vector, the charging and discharging action and the reward signal as the input, and a cumulative objective function is constructed by maximizing the cumulative discounted reward:

[0099]

[0100] Wherein, J i (θ i ) represents the cumulative objective function, which describes the expected cumulative reward of the i-th device under the strategy function , represents the expectation under the strategy , γ represents the discount factor, T represents the total number of optimization periods, and θ i represents the strategy function parameters of the i-th device, represents the reward signal obtained at time step t;

[0101] In the upper layer cooperative optimization process, all energy storage devices are constructed as corresponding nodes in the graph structure, the energy storage state feature vector, the charging and discharging action and the reward signal are taken as node attributes, and the edge weight is defined by the coupling relationship between each pair of devices. The aggregated features are obtained through the graph attention mechanism, and the cooperative loss function is constructed based on the strategy distribution similarity:

[0102]

[0103] Wherein, L coop represents the cooperative loss function, N represents the number of energy storage devices, represents the adjacent device set of the i-th device, and α ijdenotes the attention weight between the i-th device and the j-th device calculated by the graph attention mechanism, denotes the Kullback-Leibler divergence between the i-th and j-th device strategy distribution;

[0104] The coupling relationship between the devices is defined by strategy similarity, specifically using Kullback-Leibler divergence, for measuring the difference in action distribution of different energy storage devices in similar states, and the higher the strategy similarity, the more consistent the strategy tendency between the devices, and the greater the edge weight;

[0105] The attention weight is obtained by linear mapping and splicing the node attributes between the energy storage devices, using a leaky rectified linear unit function to calculate the unnormalized attention score, and normalizing it by the Softmax function within the device adjacency domain, for measuring the weight proportion of neighbor devices in the strategy information transmission process;

[0106] In the fusion optimization phase, a joint objective function is constructed to comprehensively consider the local revenue optimality and the consistency of the strategies between the devices:

[0107]

[0108] wherein, L total denotes the joint objective function, and p denotes the weight coefficient of the coordination loss term, for adjusting the balance between local optimization and global consistency;

[0109] Based on the joint objective function, the joint objective function is minimized by the gradient descent method to synchronously update all strategy function parameters, generate a globally optimal set of collaborative charging and discharging strategies, and distribute the corresponding charging and discharging strategies to each energy storage device.

[0110] The present application constructs a hierarchical optimization structure combining upper and lower layers, realizes individual optimization of the strategy at the device layer, introduces a graph structure and a graph attention mechanism to model the strategy coupling relationship between the devices at the system layer, and realizes the collaborative improvement of strategy consistency and local revenue by joint optimization of the objective function. This method significantly enhances the overall control coordination in the multi-device system, improves the collaborative discharging efficiency and revenue integration capability, and has good scalability and cluster control adaptability.

[0111] Embodiment 1:

[0112] In order to verify the feasibility of the application in implementation, the application is applied to a certain integrated energy dispatching center with multiple energy storage devices, which faces the power demand response management of industrial and commercial load side, contains a plurality of energy storage devices with different capacities and types, is distributed in the power distribution system of multiple plant areas, has real-time electricity price access capability, short-term load prediction capability and unified energy storage dispatching interface, and realizes the closed-loop control logic from state perception, strategy generation, action execution to strategy update through the deployment of the profit maximization control system driven by the strategy gradient algorithm, effectively solving the problems of unstable strategy and insufficient optimization capability of the existing energy storage dispatching scheme in response to multiple device cooperation, dynamic price fluctuation and prediction error disturbance.

[0113] In this implementation environment, first, the data acquisition module is used to acquire the operation information of each energy storage device in a unified format in real time, such as voltage, current, state of charge, charging and discharging power, temperature and cycle life state, and at the same time, real-time electricity price data and local load prediction information released by the electricity market are accessed, and the data is time-aligned, abnormal data is excluded, missing data is completed and normalized through the preprocessing link, then, through the feature extraction module, the multi-dimensional time series data is input into the deep time sequence perception network, the state vector containing the energy storage evolution trend and the market environment characteristics is extracted, and the feedforward neural strategy network is initialized, and the initial charging and discharging control action is output. The action is calculated directly by the strategy network according to the state vector and the initialization parameters, and contains the size, action time and probability distribution of the charging and discharging power.

[0114] When the control action acts on the device, the system records the action execution effect and the economic benefit obtained in real time, and updates the gradient of the current strategy function according to the price change trend, the action deviation and the historical performance, the update process controls the exploration ability by introducing the entropy regularization term, and at the same time, the strategy loss function is constructed, so that the system gradually converges to the optimal solution of stable income, in order to improve the response ability of the strategy to the change of time scale, this embodiment introduces the sliding time window mechanism, uses the state, action and reward trajectory in multiple periods to construct the period self-evolution strategy, so that the strategy function has the predictability and adjustment ability to the future trend.

[0115] In the actual operation of multiple energy storage devices, this embodiment adopts a hierarchical optimization structure, models each energy storage device as an independent strategy unit, first completes local strategy optimization in the lower layer, then aggregates the strategy information between devices through the graph attention mechanism, realizes the upper layer cooperative optimization under the strategy distribution consistency constraint, the edge weight in the graph structure is automatically updated through the strategy similarity measurement function, the attention mechanism weights the node information, and ensures that the final fused strategy considers both local optimality and global coordination. According to the actual operation feedback, the stability of the system is significantly improved, the response speed of the strategy is faster, the income volatility is obviously reduced, the scheduling between devices tends to be consistent, and the overall energy utilization efficiency is improved.

[0116] To verify the performance of the application in implementation, comparison is made with the traditional method, and the results are shown in Table 1.

[0117] Table 1 Comparison of energy storage optimization method based on policy gradient with traditional method

[0118] Indicator The method of the present invention Conventional method Strategy stability index 0.92 0.67 Return standard deviation 6.8 12.5 Strategy convergence round number 105 180 Average return improvement rate 18.3％ 0％ Multi-device strategy consistency score 0.85 0.58 Load response accuracy rate 92.6％ 81.2％

[0119] In terms of policy stability index, the method of the application shows higher consistency in multiple rounds of update of charging and discharging strategy, with a value of 0.92, which is better than 0.67 of the traditional method. This improvement is mainly due to the periodic self-evolution mechanism and sliding time window processing mode introduced by the application, which effectively smooths the strategy fluctuation, so that a relatively stable strategy output can be maintained in the environment where the electricity price changes frequently, thereby enhancing the robustness of the energy storage control.

[0120] In terms of standard deviation of income, the income fluctuation value of the application after multiple rounds of operation is reduced to 6.8, which is obviously superior to 12.5 of the traditional method, which shows that the charging and discharging strategy generated by the application not only has higher income, but also can significantly reduce the short-term income uncertainty caused by market changes. The reason is that the reward signal construction of the application combines expected value comparison, entropy regularization control and multi-period feedback mechanism, which has stronger risk relief ability.

[0121] In terms of policy convergence rounds, the method of the application completes policy stable convergence within 105 rounds of training, while the traditional method needs more than 180 rounds. The improvement of convergence speed is due to the bidirectional time sequence modeling and gating mechanism used in the application, which makes the energy storage state and action feedback features form a clear gradient direction in the early stage of policy training, shortens the invalid attempts in the policy search process, and thus greatly improves the training efficiency.

[0122] In terms of average income growth rate, the income growth of the traditional method is basically zero, while the average growth rate of the method of the application is 18.3%. This growth is due to the introduction of advantage function estimation and multi-scale time sequence modeling mechanism in the policy objective function, which makes the charging and discharging behavior match the market fluctuation more closely, can capture better arbitrage opportunities, and achieve balanced maximum economic income in different periods.

[0123] In terms of multi-device strategy consistency score, the score of the application is 0.85, which is much higher than 0.58 of the traditional method. This advantage is due to the use of graph attention mechanism in the application to jointly optimize the strategy between multiple energy storage nodes, so that different devices can make complementary decisions when facing similar market situations, reduce strategy conflicts and resource competition, and thus realize the consistency and efficiency of device scheduling

[0124] In terms of load response accuracy, the method can reach 92.6%, which is obviously improved compared with 81.2% of the traditional method. The fundamental reason is that the dynamic adjustment mechanism based on load prediction is introduced in the strategy generation process of the application, combined with the real-time feedback of charging and discharging actions, which can compensate the prediction error in time, thereby guaranteeing the rapid and accurate response of the energy storage system to the load demand, and effectively improving the overall service quality of the system.

[0125] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical scheme and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for maximizing energy storage revenue based on a policy gradient algorithm, characterized in that, Includes the following steps: Collect energy storage device status parameters, real-time electricity market price data, and load forecast data; preprocess the collected data to generate preprocessed data. Feature extraction is performed on the preprocessed data to generate an energy storage state feature vector. The energy storage state feature vector is used as the system state input to initialize the charging and discharging strategy parameters and generate the initial charging and discharging strategy. Based on the initial charge and discharge strategy, the charge and discharge actions for this cycle are generated, and the charge and discharge actions are applied to the energy storage device to perform the charge and discharge operations. The results of the charge and discharge operations are collected, and charge and discharge feedback information is generated. By combining charging and discharging feedback information with charging and discharging actions, and calculating a reward signal based on the difference between immediate and expected benefits, the reward signal, energy storage state feature vector, and charging and discharging actions are used to update the charging and discharging strategy parameters to generate an updated charging and discharging strategy. By utilizing the time-series dependencies of energy storage state feature vectors, charging and discharging actions, and reward signal sequences over multiple cycles, feedback optimization is performed on future cycle returns to generate a cycle-evolving charging and discharging strategy. In multi-energy storage device scenarios, the energy storage state feature vectors, charging and discharging actions, and reward signals of each energy storage device are used for lower-level autonomous strategy optimization and upper-level device coordination to generate a globally optimal collaborative charging and discharging strategy.

2. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The energy storage device status parameters include state of charge, voltage, current, charging power and maximum charging power, discharging power and maximum discharging power, temperature and cycle life status. The preprocessing includes time synchronization, outlier detection, missing value completion, noise filtering and normalization.

3. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the initial charge / discharge strategy specifically includes: The preprocessed data is arranged in chronological order to construct a multidimensional data sequence with multiple consecutive time steps. Each time step includes energy storage device status parameters, real-time electricity market price data, and load forecast data. Feature extraction is performed on multidimensional data sequences. Short-term dynamic change features between time steps are extracted through a one-dimensional convolution structure. The convolution results are then activated and compressed to obtain the feature response within a local time segment. The feature response within a local time segment is input into a bidirectional long short-term memory neural network, and recursive feature encoding is performed simultaneously in both the forward and reverse directions of time to capture the trend pattern in the long time series. Finally, the corresponding energy storage state feature vector is output at the current time step. The energy storage state feature vector is used as the system state input, and the charging and discharging strategy parameters are initialized. The energy storage state feature vector and the initial charge / discharge strategy parameters are input into the strategy network. The initial charge / discharge strategy parameters are used to initialize the strategy network. The strategy network is a feedforward neural network with a two-layer fully connected structure. The first layer uses a modified linear unit as the activation function, and the second layer uses a hyperbolic tangent function as the activation function. The output is the initial charge / discharge strategy under the current system state.

4. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the charge / discharge feedback information specifically includes: The charging and discharging actions for the current cycle are generated based on the initial charging and discharging strategy; The generated charging and discharging actions are applied to the energy storage device, and the actual charging and discharging operations are executed according to the control parameters; After the charging and discharging operation is completed, the energy storage state feature vector at the end of the charging and discharging operation is collected; The immediate revenue for the current period is calculated by combining the charging and discharging actions with the real-time price of the electricity market. The immediate revenue is the economic return obtained by the energy storage device through charging and discharging during the operation, which is obtained by multiplying the electricity by the real-time electricity price. The energy storage state feature vector after charging and discharging is combined with the immediate benefits to form charging and discharging feedback information.

5. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the updated charge / discharge strategy specifically includes: The reward signal is calculated based on the difference between the immediate return and the expected return for the current period. A policy training input consisting of energy storage state feature vectors, charging and discharging actions, and reward signals is constructed. The charging and discharging policy parameters are optimized using a policy network, and an optimization objective function is constructed. The charging / discharging strategy parameters are updated using gradient ascent based on the optimization objective function; The updated charge / discharge strategy parameters are applied to the energy storage state feature vector of the current cycle to generate an updated charge / discharge strategy for the next cycle's strategy iteration.

6. The method for maximizing energy storage revenue based on a policy gradient algorithm according to claim 1, characterized in that, The generation of the periodic self-evolving charging and discharging strategy specifically includes: Construct a historical behavior trajectory sequence to obtain energy storage state feature vector sequences, charging and discharging action sequences, and reward signal sequences over multiple consecutive periods; The three types of sequences are input into the time series modeling structure, and a gated recurrent unit network is used to extract the temporal state representation under each period to form a joint state representation vector sequence. Based on the current period joint state representation vector, the charging and discharging policy probability distribution is generated using the policy function, and the policy loss function is constructed by combining the advantage function. The policy loss function is trained using gradient ascent, the parameters of the policy function are updated, and a sliding time window mechanism is introduced in each update to control the historical behavior trajectory within the most recent k cycles as the basis for optimization. The updated strategy function parameters are used in the strategy generation process of the next cycle. By combining the energy storage state feature vector of the latest cycle and the joint state representation vector corresponding to the current cycle, the cycle-evolving charging and discharging strategy of the current cycle is generated.

7. The method for maximizing energy storage revenue based on the policy gradient algorithm according to claim 1, characterized in that, The generation of the globally optimal cooperative charging and discharging strategy specifically includes: In an operation scenario involving multiple energy storage devices, the energy storage state feature vector, charging and discharging actions in the current cycle, and corresponding reward signals of each energy storage device are extracted. A hierarchical optimization structure consisting of autonomous optimization of lower-level devices and coordination between upper-level devices is constructed. Based on the fusion optimization results, the globally optimal collaborative charging and discharging strategy is output and fed back to each energy storage device for execution. In the lower-level autonomous optimization process, for the i-th energy storage device, a strategy function is constructed with the energy storage state feature vector, charging and discharging actions and reward signals as inputs, and a cumulative objective function is constructed by maximizing the cumulative discount reward; In the upper-level collaborative optimization process, all energy storage devices are constructed as corresponding nodes in the graph structure. The energy storage state feature vector, charging and discharging actions and reward signals are used as node attributes. The edge weights are defined using the coupling relationship between each pair of devices. Aggregated features are obtained through the graph attention mechanism, and a collaborative loss function is constructed based on the similarity of policy distribution. During the integration and optimization phase, a joint objective function is constructed to comprehensively consider both local benefit optimization and strategy consistency among devices. Based on the joint objective function, the parameters of all policy functions are updated synchronously by minimizing the joint objective function through the gradient descent method, generating a globally optimal set of coordinated charging and discharging policies, and distributing the corresponding charging and discharging policies to each energy storage device.

Images