Methods and systems for controlling renewable energy plant

Abstract

Methods and systems for controlling a renewable energy plant are described A method of controlling a renewable energy plant comprising a renewable energy generation system and an energy storage system comprises: receiving state data of the renewable energy plant, the state data comprising power output data of the renewable energy generation system and state of charge data of the energy storage system; generating a power output bid indicating a power output for the renewable energy plant over a subsequent dispatch period by inputting the state data into a deep reinforcement learning model which is trained to maximise a reward function over a generation cycle comprising a plurality of dispatch periods including the subsequent dispatch period, wherein the reward function is formulated from a positive reward proportional to a power delivered in each dispatch period of the generation cycle and a penalty when a power delivered falls short of a power output bid for a dispatch period of the generation cycle; and controlling the renewable energy plant to output power at the power output bid over the subsequent dispatch period.

Description

[0001] METHODS AND SYSTEMS FOR CONTROLLING RENEWABLE ENERGY PLANT

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to the field of renewable energy production and power management. In particular, the present disclosure relates to methods and systems for controlling a renewable energy plant based on power output bidding with a trained model.

[0004] BACKGROUND

[0005] In response to global climate change, many countries have set carbon emission reduction plans. For example, Singapore’s power sector has set targets of net-zero emissions by 2050. To achieve such targets, the Energy Market Authority (EMA) of Singapore has identified nine action strategies, where the first is to “pursue the adoption of electricity imports to access cleaner and cost-effective energy sources beyond Singapore’s borders”. In this context, several overseas photovoltaic (PV) plants have been planned for construction in neighbouring countries; these plants will export significant power to Singapore’s power grid through subsea cables.

[0006] To participate in the Singapore Wholesale Electricity Market (SWEM), electricity importer will be required to play a role similar to that of traditional generators. Specifically, the overseas PV plants must provide offers to the energy market and compete to secure dispatches to the Singapore network for each half-hour period. Once a dispatch for a particular quantity of electricity is secured, the importer will be scheduled to inject and sell this quantity of electricity to the market. In other words, the PV plants will need to maintain constant power output during each real-time operating interval (30 min); that is, once the PV plant submits the bid power amount, it must satisfy its commitment, which would otherwise lead to a penalty payment. However, the PV power output depends on solar irradiance, which is highly volatile. To maintain a constant power output, it would be necessary to deploy an energy storage system (ESS) in the PV plant to compensate for PV power fluctuations during each operation interval. For this particular large-scale renewable power integration and participation in the SWEM, one technical question needs to be answered: What is the optimal bidding power for the operation of a PV plant?

[0007] In recent years, there are several research works focusing on bidding optimization for PV-ESS plants. Because of the unpredicted nature of PV generations, such bidding problem is normally solved by model predictive control (MPC), robust optimization (RO) approaches, and stochastic programming (SP) approaches. Although those conventional approaches have been making significant advances, there are still barriers to apply these solutions to the real-world. For MPC, it may suffer from modelling complexity and parameter uncertainty. Besides, RO / SP approach highly depends on the assumptions made about the prior distribution of the random variables. For example, the assumptions about the distribution or the range of the random variables, and the convexity of the optimization problem, are indispensable in most cases. Furthermore, SP approaches are sometimes not realistic and practical, since they could be intractable and suffer from unclarified and high-dimensional frameworks. In practice, only the historical data of the random variables (PV / load) rather than the predefined distributions are available, and it is tricky to formulate the problem as a convex optimization problem.

[0008] Due to the significant development of artificial intelligence techniques in this decade, data-driven methods have been widely studied in the power system. As one of the emerging data-driven approaches, reinforcement learning (RL) has been demonstrated as an effective tool to solve practical problems, whose general principle is to use “trial and error” interaction with a dynamic system to find an optimal policy. In the process of RL, the agent receives a state from the environment and generates an action according to the state. This action will be sent to the environment and executed to obtain a newly-observed state. Meanwhile, a reward value is obtained based on the observation, which is utilized to update the policy of the agent. Through multiple iterations, an optimal policy can be achieved to maximize the reward during the process. On the other hand, conventional RL algorithms may suffer from highdimensional state / action, which leads to a significant limitation in the continuous domain. To handle this “curse of dimensionality”, deep reinforcement learning (DRL) is developed, which has attracted considerable attention from both academic and industrial fields. Currently, a number of research works have exploited DRL in power system field, such as for frequency control, energy management system, volt / var control, etc.

[0009] In this context, the present invention is directed to providing an optimal bidding strategy for PV-ESS plants based on DRL techniques. The invention aims to accurately and comprehensively characterize the ESS and offer efficient computational performance and enable straightforward implementation in practical deployment scenarios.

[0010] SUMMARY OF THE INVENTION

[0011] According to a first aspect of the present disclosure, a method of controlling a renewable energy plant comprising a renewable energy generation system and an energy storage system is provided. The method comprises receiving state data of the renewable energy plant, the state data comprising power output data of the renewable energy generation system and state of charge data of the energy storage system; generating a power output bid indicating a power output for the renewable energy plant over a subsequent dispatch period by inputting the state data into a deep reinforcement learning model which is trained to maximise a reward function over a generation cycle comprising a plurality of dispatch periods including the subsequent dispatch period, wherein the reward function is formulated from a positive reward proportional to a power delivered in each dispatch period of the generation cycle and a penalty when a power delivered falls short of a power output bid for a dispatch period of the generation cycle; and controlling the renewable energy plant to output power at the power output bid over the subsequent dispatch period.

[0012] The state data input to the deep reinforcement learning model includes both power output data and state of charge data, which enable the model to inherently capture the temporal dynamics and interdependencies of these variables. This implicitly embeds the functionality of power output and state of charge prediction within the learned policy of the model, thereby eliminating the need for a separate forecasting or prediction module. The deep reinforcement learning model is pre-trained, enabling resource-intensive learning to be conducted in advance. During online deployment, real-time decisionmaking involves only matrix operations on a current set of state data, resulting in fast and computationally efficient execution. This feature makes the present invention well- suited for embedded or real-time control systems.

[0013] Furthermore, the method leverages the capability of deep neural networks to recognize and learn temporal patterns within the data. This enables the model to generalize effectively across varying operating conditions and unseen scenarios, ensuring robust performance over time.

[0014] In an embodiment, the deep reinforcement learning model is trained with historic state data of the renewable energy plant.

[0015] Since the model may be trained entirely using historical state data, it avoids reliance on complex system modeling or additional predictive components. This simplifies implementation and deployment in practical systems, particularly in dynamic energy market environments.

[0016] In an embodiment, the state data comprises power output data from the renewable energy generation system and state of charge data from the energy storage system for a time interval preceding receipt of the state data.

[0017] In an embodiment, the state data comprises power output data from the renewable energy generation system for a time interval on previous day that corresponds to a time interval following receipt of the state data.

[0018] In an embodiment, the generation cycle is a daily cycle.

[0019] In an embodiment, the renewable energy plant is a photovoltaic plant.

[0020] The invention may leverage structured state data, which includes: state data for the current time interval, capturing real-time system conditions; and state data from the corresponding time interval on previous days, enabling the model to recognize and learn temporal patterns that recur over time.

[0021] This structure is particularly effective in renewable energy systems, such as photovoltaic plants, which typically exhibit characteristic daily generation cycles. By incorporating historical temporal context into the input data, the deep reinforcement learning model can identify and adapt to these recurring patterns, thereby improving the accuracy and reliability of power bidding decisions.

[0022] In an embodiment, the formulation of reward function further includes a penalty component based on a cost associated with restoring state of charge of the energy storage system.

[0023] When the reward function takes into consideration of the cost associated with restoring the state of charge of the energy storage system, the model discourages complete depletion of the state of charge and promotes more sustainable and practical operation of the energy storage system. This design reflects real-world operational constraints and enhances the model’s predictive performance and decision-making quality under practical conditions.

[0024] In an embodiment, proximal policy optimization is used in the deep reinforcement learning model.

[0025] Proximal policy optimization demonstrates good convergence performance under uncertain transitions in high-dimensional, continuous domains, such as the power bidding problem addressed in the present invention.

[0026] In an embodiment, the step of controlling the renewable energy plant comprises controlling the energy storage system to release electrical power when output of the renewable energy generation system is insufficient to satisfy the power output bid, or to absorb electrical power when output of the renewable energy generation system exceeds the power output bid. According to a second aspect of the present disclosure, a computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a method set out above is provided.

[0027] According to a third aspect of the present disclosure, a system for controlling a renewable energy plant comprising a renewable energy generation system and an energy storage system is provided. The system comprises a processor; data storage storing a deep reinforcement learning model which is trained to maximise a reward function over a generation cycle comprising a plurality of dispatch periods including a subsequent dispatch period, wherein the reward function is formulated from a positive reward proportional to a power delivered in each dispatch period of the generation cycle and a penalty when a power delivered falls short of a power output bid for a dispatch period of the generation cycle; and program storage storing computer program instructions operable to cause the processor to: receive state data of the renewable energy plant, the state data comprising power output data of the renewable energy generation system and state of charge data of the energy storage system; generate a power output bid indicating a power output for the renewable energy plant over the subsequent dispatch period by inputting the state data into the deep reinforcement learning model; and control the renewable energy plant to output power at the power output bid over the subsequent dispatch period.

[0028] In an embodiment, the deep reinforcement learning model is trained with historic state data of the renewable energy plant.

[0029] In an embodiment, the state data comprises power output data from the renewable energy generation system and state of charge data from the energy storage system for a time interval preceding receipt of the state data.

[0030] In an embodiment, wherein the state data comprises power output data from the renewable energy generation system for a time interval on previous day that corresponds to a time interval following receipt of the state data.

[0031] In an embodiment, the generation cycle is a daily cycle. In an embodiment, the renewable energy plant is a photovoltaic plant.

[0032] In an embodiment, the formulation of reward function further includes a penalty component based on a cost associated with restoring state of charge of the energy storage system.

[0033] In an embodiment, proximal policy optimization is used in the deep reinforcement learning model.

[0034] In an embodiment, the program storage further stores computer program instructions operable to cause the processor to control the energy storage system to release electrical power when output of the renewable energy generation system is insufficient to satisfy the power output bid, or to absorb electrical power when output of the renewable energy generation system exceeds the power output bid.

[0035] In an embodiment, the program storage further stores computer program instructions operable to cause the processor to train the deep reinforcement learning model using historic state data.

[0036] BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In the following, embodiments of the present invention will be described as non-limiting examples with reference to the accompanying drawings in which:

[0038] FIG.1 shows a schematic of an overseas PV-ESS power plant connected to the Singapore grid;

[0039] FIG.2 is a block diagram showing a renewable energy plant control system according to an embodiment of the present invention;

[0040] FIG.3 is a flowchart showing a method of controlling a renewable energy plant according to an embodiment of the present invention; FIG.4 shows an example of the overall structure of the invention for optimal bidding strategy according to an embodiment of the present invention;

[0041] FIG.5 shows one example piece of historic data for offline training, including PV generation data within seven days;

[0042] FIG.6 is a diagram of the deep neural network (DNN) based actor network (bidding agent) according to an embodiment of the present invention;

[0043] FIG.7 shows the episode rewards during offline learning according to an embodiment of the present invention;

[0044] FIG.8A to 8C show a set of test results for an example embodiment of the present invention; FIG. 8A shows PV generations and bidding power outputs by an example embodiment of the present invention; FIG.8B and FIG.80 show ESS outputs and SOO values during dispatch period respectively accordingly to an embodiment of the present invention;

[0045] FIG.9A to 90 show a set of test results for an example embodiment of the present invention; Fig.9A shows PV generations and bidding power outputs by an example embodiment of the present invention; FIG.9B and FIG.90 show ESS outputs and SOC values during dispatch period respectively accordingly to an embodiment of the present invention;

[0046] FIG.10 shows an example of bidding process of an overseas PV plant participating in Singapore Electricity market;

[0047] FIG.11 shows an equivalent circuit model of ESS according to an embodiment of the present invention;

[0048] FIG.12 shows an example of the general framework of the method according to an embodiment of the present invention;

[0049] FIG.13 shows an example of PV generation data for seven days; FIG.14A to 14D show curves of polynomial parameters of electric elements according an embodiment of the present invention; FIG.14A shows the polynomial curve of vQC; FIG.14B shows the polynomial curve of ?o; FIG.14C shows the polynomial curve of Ri ; FIG.14D shows the polynomial curve of Ci;

[0050] FIG.15 shows a plot of terminal voltage to State of Charge (SoC) under different currents according to an embodiment of the present invention;

[0051] FIG.16A and 16B are plots showing relation between of State of Power (SoP) and SoC of ESS according to an embodiment of the present invention. FIG.16A shows a SoP- SoC plot during ESS discharging. FIG.16B shows a SoP-SoC plot during ESS charging;

[0052] FIG.17 shows a plot of episode rewards during offline training according to an embodiment of the present invention;

[0053] FIG.18A to 18D show test results of a proposed method on one week of test data, according to an embodiment of the present invention; FIG.18A shows a plot of PV generations and bidding results; FIG.18B shows a plot of ESS’s output power; FIG.18C shows a plot of ESS’s SoC ; FIG.18D shows a plot of ESS’ SoH;

[0054] FIG.19A to 19C show comparison between a proposed method according to an embodiment of the present invention and a model (optimization) based method with real PV data; FIG.19A shows a plot of bidding results; FIG.19B shows a plot of ESS’s SoC; FIG.19C shows a plot of average profit;

[0055] FIG.20A to 20E show comparison between a proposed method according an embodiment of the present invention and a model (optimization) based method; FIG.20A shows a plot of measured and predicted PV data; FIG.20B shows a plot of bidding results; FIG.20C shows a plot of ESS’ SoC; FIG.20D shows a plot of average profits; and FIG.20E shows a plot accumulated profits. DETAILED DESCRIPTION

[0056] The present disclosure relates to systems and methods for controlling the output of renewable energy plants and managing power bidding processes. Examples of the disclosed systems and methods may be configured to comply with regulatory frameworks applicable to the Singapore power grid, but the application is not limited to Singapore grid. While embodiments of the present invention may be described with reference to photovoltaic (PV) power plants, the invention is not restricted to such applications. The present invention is applicable to a broad range of renewable energy power plants characterized by intermittent electricity generation, including but not limited to wind and tidal energy plants.

[0057] Singapore’s power sector has set targets of net-zero emissions by 2050. Given the limited land space of the country, a key strategy to decarbonize the power grid is to import clean power from renewable energy resources such as photovoltaic (PV) plants installed at overseas locations. The present electricity market rules require such overseas PV plants to maintain constant power generation during each bidding period. To meet such requirements, energy storage systems (ESSs) are to be deployed in the PV plants to compensate for the PV power fluctuation.

[0058] FIG.1 illustrates an example of a PV-ESS power plant connected to Singapore grid 130. The PV-ESS comprises a PV generation system 110 and ESS 120. The PV generation system 110, or alternatively referred to as the PV plant 1 10, utilizes the solar panel to transform solar energy into electricity, which can be stored into or be sold to the power grid directly through the inverter. Ppvis the generated power by PV plant 110. As an intermediate, ESS 120 interacts closely with the PV plant 1 10 via charging and discharging and PESSrepresents the charging or discharging power from ESS 120. PBi.dis the bidding power from PV-ESS power plant to Singapore grid 130.

[0059] FIG.2 is a block diagram 200 showing a renewable energy plant control system 210 according to an embodiment of the present invention. The renewable energy plant control system 210 controls a renewable energy plant 220 comprising a renewable energy generation system 221 and an energy storage system 222. The delivery or output of power from the renewable energy plant 220 to an energy grid 230 is controlled. The renewable energy plant control system 210 comprises a processor 240, program storage 270 and data storage 280. The system may further comprise a working memory 250 and network interface 260.

[0060] The processor 240 may be implemented as one or more central processing unit (CPU) chips. The program storage 270 is a non-volatile storage device such as a hard disk drive which stores computer program modules. The computer program modules are loaded into the working memory 250 for execution by the processor. The data storage 280 is a non-volatile storage device which stores data which is used by the renewable energy plant control system during processing. The network interface 260 allows the renewable energy plant control system to communicate with other devices, send control signals and receive data from other devices. In various embodiments, the control system 210 may receive real-time data related to energy generation, energy storage capacity, energy grid demand, and other operational parameters. It is appreciated that the boundaries separating the energy plant control system and the renewable energy plant are for illustration only. In certain embodiments, the renewable energy plant control system 210 may be implemented as an integrated control module within the renewable energy plant 220 and the control system may directly exchange data and execute control signals with the renewable energy generation system 221 , energy storage system 222 and the energy grid 230.

[0061] The program storage stores a state observation module 271 , a power bidding module 272 and an energy exchange control module 273. The computer program modules cause the processor to execute various processing methods which are described in more detail below. The program storage 270 may be referred to in some contexts as computer readable storage media and / or non-transitory computer readable media.

[0062] The state observation module 271 comprises program instructions operable to cause processor 240 to observe or receive state data 281 of the renewable energy plant. The state data 281 comprises power output data of the renewable energy generation system 221 and state of charge data of the energy storage system 222. The state data 281 generally represents the operational status of the environment or system at a given time. The state data 281 may comprise a combination of recent and earlier data within the same time frame or episode, thereby capturing temporal dependencies and dynamic context. In embodiments of the present invention, the state data 281 may comprise power output data from the renewable energy generation system 221 and state of charge data from the energy storage system 222 for a time interval preceding receipt of the state data. The state data 281 may also comprise power output data from the renewable energy generation system 221 for a time interval on previous day that corresponds to a time interval following receipt of the state data 281 .

[0063] The power bidding module 272 comprises program instructions operable to cause the processor 240 to generate a power output bid indicating a power output for the renewable energy plant 220 over a subsequent dispatch period by inputting the state data into a deep reinforcement learning model 284.

[0064] The energy exchange control module 273 comprises program instructions operable to cause the processor 240 to control the renewable energy plant 220 to output power at the power output bid over the subsequent dispatch period. Since the renewable energy plant 220 comprises a renewable energy generation system 221 and an energy storage system 222, the two components of the renewable energy plant may be independently controlled to output power at the power output bid. The energy exchange control module 273 may further stores computer program instructions operable to cause the processor 240 to control the energy storage system 222 to release electrical power when output of the renewable energy generation system 221 is insufficient to satisfy the power output bid, or to absorb electrical power when output of the renewable energy generation system 221 exceeds the power output bid.

[0065] In situations where both the energy storage system 222 and the renewable energy generation system 221 are unable to supply sufficient power to fulfil a power output bid, external power supply may be required to supplement the power output. In one embodiment, the energy exchange control module may further stores computer program instructions operable to initiate an import of external power in order to satisfy the required power output bid. Furthermore, when the state of charge (SoC) of the energy storage system 222 falls below a predetermined threshold, and no power exchange with the energy grid 230 is currently taking place, the program storage 270 may store additional computer program instructions configured to cause the processor 240 to restore the ESS SoC to a defined level. This predefined level may be determined based on criteria that promote the health, performance, or operational strategy of the energy storage system. For example, the SoC may be restored to approximately 50%, or another suitable level, depending on the implementation.

[0066] The deep reinforcement learning model 284 used in the power bidding module 272 may be trained during an offline phase, wherein historical operational data is used to optimize the model parameters. Once trained, the model may be deployed during online operation to generate power output bids in real time. The system is configured to ensure computational efficiency during the online phase, enabling fast and responsive bid generation suitable for real-time operation. As a result, the program storage 270 may further store computer program instructions operable to cause the processor 240 to train the deep reinforcement learning model using historic state data 281. The historic state data 281 may represent system conditions corresponding to specific patterns or characteristic periodicities associated with the renewable energy generation profile. By learning these temporal patterns, the deep reinforcement learning model 284 can enhance its predictive accuracy and improve the alignment of power output bids with actual generation. The incorporation of historic state data 281 also implicitly integrates the forecasting of power output from the renewable energy generation system into the method executed by an embodiment of the present control system and there is no extra power output forecasting modules required for the system.

[0067] As depicted in FIG.2, the computer program modules are distinct modules which perform respective functions implemented by renewable energy plant control system 210. It will be appreciated that the boundaries between these modules are exemplary only, and that alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. The data storage 280 of the renewable energy plant control system stores the deep reinforcement learning model 284, which is trained to maximise a reward function over a generation cycle comprising a plurality of dispatch periods including a subsequent dispatch period, wherein the reward function is formulated from a positive reward proportional to a power delivered in each dispatch period of the generation cycle and a penalty when a power delivered falls short of a power output bid for a dispatch period of the generation cycle. In relation to the reward function, the positive reward is indicative of desirable outcomes, such as energy-efficient operation and maximization of power output from the renewable energy plant. The penalty represents undesirable conditions, including energy insufficiency and potential operational costs associated with drawing supplemental power from external sources to compensate for a deficiency in meeting the committed power output bid.

[0068] The reward function may also be formulated as a cumulative reward accumulated over a period of time. Formulating the reward in this manner encourages the model to optimize performance over the long term rather than reacting to short-term gains.

[0069] In one embodiment, the formulation of reward function may further include a penalty component based on a cost associated with restoring state of charge of the energy storage system 222. This penalty is designed to account for the long-term operational efficiency of the energy storage system 222 and to discourage complete depletion of the state of charge during operation, which is generally undesirable. The inclusion of this component in the reward function also reflects the impact of degradation on the energy storage system 222, thereby promoting operational strategies that preserve the health and longevity of the storage system.

[0070] As discussed above, the deep reinforcement learning model 284 may be trained with historic state data 281 of the renewable energy plant 220. The historic state data 281 as training data implicitly incorporates forecasting function into the deep reinforcement learning model 284.

[0071] Given that PV energy generation corresponds to a 24-hour pattern, the generation cycle may be a daily cycle. Alternatively, in the case of other types of renewable energy sources, the generation cycle may be configured to align with the specific temporal characteristics or periodicity associated with the respective renewable energy generation.

[0072] In one embodiment, proximal policy optimization (PPO) is used in the deep reinforcement learning model 284. Proximal policy optimization is a policy-gradient method for reinforcement learning that enables stable and efficient training by restricting the magnitude of policy updates between iterations. This is achieved through a clipped loss function, which updates policy in a trusted region and avoids excessively large policy changes and improves convergence behaviour. PPO algorithm is an algorithm well-suited for solving practical problems under unclarified transitions in a high-dimensional and continuous domain, such as power bidding in the present case.

[0073] In one embodiment, the deep reinforcement learning model 284 may be trained by formulating the power bidding process as a Markov Decision Process (MDP). A MDP would require formulation of state, action and reward for implementation. A deep neural network (DNN) may be initially configured as the bidding agent, which may also be referred to as a policy function that maps system states to corresponding actions. The actions are the power output bids generated by the bidding agent. A separate critic network may evaluate the performance of these actions based on a formulated reward function. The formulation of reward function, as discussed above, may incorporate State of Charge (SoC) limitations and the restoration of SoC to reflect real- world operational requirements. Through iterative training, the bidding agent learns an optimal bidding policy that maximizes expected cumulative reward under practical system dynamics.

[0074] The data storage module may further store action data 283 comprising the power output bid values, which indicate the power to be offered by the renewable energy plant in each dispatch period. The action data 283 may be formulated as the sum of power output from the renewable energy generation system 221 and the energy storage system 222. The date storage 280 may also store the corresponding state data 281 and the reward data 282. The storage of state data 281 , reward data 282 and action data 283 may be alternatively referred to as a replay buffer and these data in the replay buffer are used to continuously train and improve the policy learned by the deep reinforcement learning model 284.

[0075] FIG.3 is a flow chart 300 showing a method of controlling a renewable energy plant 220 according to an embodiment of the present invention. The method shown in FIG.3 is carried out by the renewable energy plant control system 210 shown in FIG.2.

[0076] In step 310, the state observation module 271 receives state data 281 of the renewable energy plant, the state data 281 comprising power output data of the renewable energy generation system 221 and state of charge data of the energy storage system 222. The state data 281 may comprise power output data from the renewable energy generation system 222 and state of charge data from the energy storage system 222 for a time interval preceding receipt of the state data. The state data may also comprise power output data from the renewable energy generation system 221 for a time interval on previous day that corresponds to a time interval following receipt of the state data.

[0077] The integration of state data 281 from a time interval preceding the current moment offers insight into the recent operational dynamics of the system. This historical window may capture transient behaviours, such as fluctuations in power output or changes in the state of charge, that may indicate trends or instabilities. The inclusion of power output data from the corresponding time interval on the previous day provides a temporal reference for understanding recurring patterns in the operation of the renewable energy plant control system. The inclusion of recent and earlier state data 281 may improve the accuracy of forecasting and allow for better planning and coordination of energy.

[0078] In step 320, the processor of the renewable energy plant control system executes the power bidding module 272 to generate a power output bid indicating a power output for the renewable energy plant over a subsequent dispatch period by inputting the state data into a deep reinforcement learning model 284. The model 284 is trained to maximise a reward function over a generation cycle comprising a plurality of dispatch periods including the subsequent dispatch period, wherein the reward function is formulated from a positive reward proportional to a power delivered in each dispatch period of the generation cycle and a penalty when a power delivered falls short of a power output bid for a dispatch period of the generation cycle.

[0079] The execution of step 320 may be referring to the deployment of the deep reinforcement learning model 284 in an online operational environment, wherein the model is utilized during live operation of the renewable energy plant to generate power output bids in real time. The real-time operation allows the control system to dynamically adapt to current state conditions of the plant 220 and respond promptly to energy grid 230 signals requiring a power output commitment for the upcoming dispatch period. The deep reinforcement learning model 284 may be pre-trained offline prior to deployment in the online environment. The offline training enhances the operational efficiency of the model when deployed online, as the model is able to generate optimal or near-optimal power output bids with reduced computational overhead during real-time execution. In one embodiment, the deep reinforcement learning model 284 may be trained with historic state data of the renewable energy plant. Incorporation of historic state data into the training process allows the model to learn temporal correlations and operational patterns and incorporate forecasting functionality with enhanced predictive accuracy in generating power output bids that are more closely aligned with actual power generation capacity.

[0080] As discussed above, the formulation of reward function may further include a penalty component based on a cost associated with restoring state of charge of the energy storage system, thereby discouraging complete depletion of the energy storage system. The reward function may also be formulated as a cumulative reward accumulated over a period of time to optimize long term performance.

[0081] Furthermore, proximal policy optimization may be used in the deep reinforcement learning model 284 in order to achieve stable convergence in optimization.

[0082] As discussed above, the generation cycle may be a daily cycle. This may correspond to the case where the renewable energy plant 220 is a photovoltaic plant. The generation cycle may be configurable to align with the specific temporal characteristics, periodicity, or operational profile associated with the particular type of renewable energy plant. In step 330, the processor 240 of the renewable energy plant control system 210 executes the energy exchange control module 273 to control the renewable energy plant 220 to output power at the power output bid over the subsequent dispatch period.

[0083] As discussed above, the step of controlling the renewable energy plant 330 may comprise controlling the energy storage system 222 to release electrical power when output of the renewable energy generation system 221 is insufficient to satisfy the power output bid, or to absorb electrical power when output of the renewable energy generation system 221 exceeds the power output bid.

[0084] The step of controlling the renewable energy plant 330 may further comprise initiating an import of external power in order to satisfy the required power output bid when both the energy storage system 222 and the renewable energy generation system 221 are unable to supply sufficient power to fulfil the power output bid.

[0085] An embodiment of the present invention provides a data-driven method for optimizing the bidding strategy of overseas PV-ESS power plants in the context of power import for Singapore power grid. The embodiment relates to a bidding method for a renewable power plant, which is coupled with an energy storage system. The method may comprise: for each bidding time period, (a) receiving a state of the renewable power plant, the state comprising historical data of the renewable power plant and historical data of state-of-charge of the energy storage system, and (b) determining a bidding power of the renewable power plant based on the received state of the renewable power plant using a trained deep reinforcement learning model, where the deep reinforcement learning model is trained based on an action-reward tuple and the reward comprises revenue of the renewable power plant, penalty cost of the renewable power plant for not satisfying the bidding power and restoration cost for restoring the state-of-charge of the energy storage system. This example embodiment aims to maximize the revenue of such PV-ESS power plants while complying with the electricity market rules of Singapore.

[0086] Another embodiment of the present invention provides a data-driven optimal bidding strategy for PV-ESS plants, which aims to maximize the profit of power plants. The approach may be divided into two stages: offline development and online application. For offline development, a deep neural network (DNN) may be firstly initialized as the bid agent. Then, historic PV data and ESS’s State of Charge (SoC) value may be observed as the state vector st, which are utilized as the input of the agent. Based on that, the agent may calculate a new bid power as the action at, which PV-ESS plant will provide for the electric grid in next 30 minutes. During this period, if PV output exceeds the bid power (Ppv> PBid), the exceeding PV power will be used to charge ESS; if PV power cannot reach the bid power Ppv< PBid), ESS will be discharged to compensate the shortage. If ESS needs to compensate the shortage but does not have enough storage, the power plant may have to buy power from outside to complement the bid power. Therefore, after the action atis determined and executed, the overall revenue of the power plant in next 30 minutes may be calculated as the reward rt, and another system state st+1can be observed as the input for the next time step. Such transitions {st, at,rt} may be stored into a replay buffer for policy learning, which optimizes the policy of the agent towards maximizing the reward. In this example approach, an effective DRL algorithm, proximal policy optimization (PPO), is applied to realize policy learning. By introducing various PV conditions into offline training, the agent can be trained to be self-adaptive and robust in practice. After iterative learning episodes, the trained agent may be applied for online application. At online stage, the agent will not be trained during operation, which may only conducts simple matrix calculation with lower computational burdens. Moreover, since the input may contain historic PV data, it implicitly integrates the forecasting of PV into the invention. As an advantage, the bidding strategy does not require PV prediction module as a prior and shows an effective computational mechanism.

[0087] Reference to the maximization of "revenue" or "profit" in the description of embodiments of the present invention is not intended to imply a focus on monetary or commercial gain. Rather, such terminology is used to refer to the optimization of power output and the efficient operation of the renewable energy plant. In particular, the objective is to maximize delivered power while minimizing instances where the plant is unable to fulfill a committed power output bid to the energy grid. Failure to meet such bids introduces inefficiencies and may require supplemental power sourcing, which is operationally undesirable. Accordingly, references to "revenue" or "profit" are intended to indicate improvements in power generation and energy management efficiency, rather than financial return.

[0088] As the increasing penetration of renewable energy resources (RESs), especially PV, ESS devices, the present invention has good prospects in a wide range of industry systems. By integrating numerous PV panels into an entity, PV power plant has been an emerging generation unit in the modern power system. On the other hand, PV outputs highly rely on weather, which is uncertain and hard to predict. Therefore, PV plant is normally accompanied by ESS units, which can regulate and manage the output by releasing or absorbing the power. By utilizing an appropriate bidding strategy, such PV-ESS plants can take an active participation into the grid to realize an economic and environmental-friendly operation.

[0089] In various embodiments of the present invention, the overall bidding problem may be modelled by taking SoC limitation and restoration into consideration. The violation of SoC constraints and the restoration of SoC may both be modelled into the reward as economic issues, which is more accordant with practical situations. By formulating the bidding process into MDP, this invention learns an optimal bidding policy based on DRL techniques, i.e., PPO algorithm, which can solve practical problems under unclarified transitions in a high-dimensional and continuous domain.

[0090] Compared with existing MPC, RO, SP-based approaches, embodiments of present invention show advantages owing to the data-driven nature. First, the present invention may only require historic dataset without complex mathematical modelling, which shows a simple implementation process. Besides, since the input of the bid agent comprises historic data of PV / SoC, embodiments of present invention implicitly include the functionality of PV / SoC prediction, which would not need an extra prediction module as conventional RO / SP-based approaches do. Furthermore, once the offline training is implemented, the trained agent only conducts matrix calculation to obtain the bidding power, which only requires historic data as the input. Compared with iterative optimization algorithms, such data-driven approach shows a more effective computational mechanism. Embodiments of the present invention also perform a good generalization ability, which can self-adapt to various operation conditions. In summary, systems and methods of present invention show a reliable and effective performance, which would have good prospects in the industrial application.

[0091] A detailed description of an embodiment of the present invention with mathematical formulations is provided in the following.

[0092] The PV generation system utilizes the solar panel to transform solar energy into electricity, which can be stored into the battery or be sold to the power grid directly through the PV inverter. As an intermediate, ESS interacts closely with the grid and PV generation system via charging / discharging, which can be modelled as: where SoC(t) indicates SoC value at t moment, Pdis(t), PCh(t) represent discharging, charging powers of ESS at t moment, t]dis, rjchrepresent the efficiency rates of discharging and charging of ESS, At indicates the time interval, and Eraterefers to the rated capacity of ESS.

[0093] In the example embodiment of the present invention, a data-driven method is designed to optimize the bidding strategy of the PV-ESS power plant, and the bidding strategy may be represented by the sum of PV and ESS outputs (PBid- Ppv+ PESS) in each bidding cycle. To learn an optimal bidding strategy, the bidding problem may be formulated as a Markov Decision Process (MDP). By observing the system state stas the input, the bid agent generates the action ataccording to policy at= ?r(st), which changes the state of the environment to st+1. Then, a reward value rtis returned to evaluate the performance under the pair of (st, at) . In this invention, based on preliminary data analytics, the state that represents the input of the bid agent may be selected as: The state may comprise three parts: historic PV data in the same period of the day before, historic PV data of last 30 minutes, SoC values of last 10 minutes. As the state contains historic data, it implicitly integrates the forecasting into the agent’s policy. The action ataims to determine the total output of the power plant, which may also be represented by the bidding power. The ESS output command may be determined by the residual between PV generation and bidding power, which can be expressed as: at=PBid(t) (3)

[0094] PESS,com(.t)=PBid(t) ~ Ppv(t) (4)

[0095] In this example embodiment, the time-resolution of the action at(bidding power) may be defined as 30 minutes to comply with the electricity market rules in Singapore.

[0096] The reward rtmay comprise three components rB,d, rPen, rtRes, which indicate revenue of power plant, ESS penalty cost, SoC restoration cost respectively. The detailed definition may be formulated as follows: rt —rtld+rtSS+ ERes, t = 1,2, ... , T (5) rtBid= YBid ■ Peid' t = 1'2, - , T (6)

[0097] As the first item, rtBldindicates the profits of PV-ESS plant, where yBidrefers to the electricity price for selling power to the electricity grid. Second, rtESSindicates the penalty cost if the PV-ESS plant cannot satisfy the bided power, a penalty cost will be placed in proportional to the deficient power, where refers to the maximum power that ESS can provide, yESSrefers to the penalty rate to compensate the power deficiency, and SoCminrepresents the lower bound of ESS’s SoC. The last item rtResindicates the payment cost of ESS for restoring its SoC to 50% during nighttime, yfiesrefers to the price for SoC restoration. The goal of the bidding strategy in this example embodiment is to learn the optimal policy n by maximizing its accumulated rewards.

[0098] To solve such MDP above, a PPO based approach may be adopted. Based on an actor-critic structure, it consists of an actor network for action generation and a critic network for evaluation. The actor network is actually the bid agent, at- n(st\3TT) , which is parameterized by 0T The policy n is initialized as a probability distribution in a stochastic manner. It can be developed based on reparametrized Gaussian with a squashing function: indicate two parameterized DNNs to generate mean and covariance, and is a random variable from a standard normal distribution that ft~N(0, I). It should be noted that both state and action spaces are in the continuous space.

[0099] The actor is updated towards maximizing the reward, as follows: where E(-) indicates the expectation function, p07r(at|st) indicates the probability of taking the action atunder state stbased on the policy with parameters , VZogpe7r(at|st) indicates the direction for selecting action atunder state st, R(t) refers to the cumulative reward that the agent obtains after st, and VR9n indicates the direction for selecting atwith larger reward R(t).

[0100] To approximate the cumulative reward and achieve a better learning efficiency, a critic network may be formulated as VOtl#7), where 6Vrefers to network’s parameters. This critic network aims to approximate the expected cumulative reward that the policy achieves under observed state st. Based on this critic network, an advantage function 4(st, at) may be designed based on the temporal-difference error, as follows: yl(st, at) = rt(st,at) + ydK(st+1) - 7(st) (13) where ydrefers to a discounting factor. Such advantage function reflects the advantage of action atunder stover the expected reward of all previous actions. Besides, the loss function of the critic network may be given by:

[0101] / (C> - E[(7(st) - yf)2] (14) yt= rt(st,at) + ydV(st+1) (15) where ytindicates the target value of To update the policy within an appropriate trust region, a clipped loss function is designed as: where E refers to the clipping rate that restricts the updated policy into a trusted region, 0n' indicates the parameters of the “old” policy, clipQ) indicates the clipped function that achieves a trade-off among simplicity, sampling complexity, and wall-time.

[0102] The overall training process of the example embodiment is illustrated in Algorithm 1 , as follows:

[0103] Algorithm 1 : PPO for Offline Training

[0104] 1 : Initialize actor network n~6n, critic network V~6v, and “old” actor network 7r'~07r'

[0105] 2: for episode = 1 to M do

[0106] 3: Randomly determine an initial state set sx.

[0107] 4: for t = 6:00 to 18:30 do

[0108] 5: Generate action atby observed state st(historic PV and SoC value).

[0109] 6: Calculate ESS output command by: 8: PEss,com(t) = Pdi?x, the shortage is compensated by purchasing power outside, the penalized reward rtESSis calculated,

[0110] 9: end if

[0111] 10: if t= 18:30 do

[0112] 11 : Calculate ESS restoration reward rEes(restore SoC to 50% at night time)

[0113] 12: end if

[0114] 13: Calculate the overall reward rt, and observe next state st+1.

[0115] 14: Store the transition {st, at, rt, st+1} into the replay buffer R.

[0116] 15: end for

[0117] 16: A mini-batch including m instances is randomly sampled from R.

[0118] 17: Compute advantage function value A(st, at).

[0119] 18: Update the actor network’s parameters:

[0120] 19: Update the critic network’s parameters:

[0121] 20: Update “old” actor network:

[0122] 21 : end for

[0123] FIG.4 shows an example of the overall operational and training structure of the invention for optimal bidding strategy. The diagram illustrates the MDP formulation 410 for the power bidding process of a PV-ESS plant. In this framework, the action generated by the actor network 454 is executed within the environment 420, representing the physical PV plant. The reward is then computed based on the outcome observed from the environment 420. The state includes the State of Charge (SoC) of the energy storage system 430 and historical PV generation data 440, both of which are obtained from the environment 420. The policy optimization using the PPO algorithm 450 is achieved through the formulation of loss functions 451 , 453, which guides the simultaneous updating of both the actor 454 and critic networks 452, enabling the learning of an optimal bidding policy.

[0124] To verify the performance of the present invention, several tests are conducted based on the example embodiment described in the section above. Detailed parameters of the test model are presented in Table I. A PV power generation dataset for an island near Singapore is used here, which has a 1 -min resolution from 10 / 22 / 2020 6:00 to 12 / 31 / 2020 18:59.

[0125] Table I. Detailed parameters of test model.

[0126] Parameter Value

[0127] Size of DC-DC converter 120 MW

[0128] Capacity of ESS 120 MWh

[0129] Upper, lower bounds of SoC 95%, 5%

[0130] ESS Discharging, charging efficiency 95%, 95%

[0131] Size of PV generation 600 MW

[0132] FIG.5 shows an example of PV data, including 1 -week period of PV generation. A characteristic daily cycle in PV generation is shown.

[0133] First 40-day (10 / 22 / 2020 ~ 11 / 30 / 2020) datasets are utilized for offline training, and remaining 31 -day (12 / 01 / 2020 ~ 12 / 31 / 2020) datasets are used for testing. It should be noted that training and testing data are not overlapped. In the beginning of each learning episode, 1 -day training dataset is randomly selected. Besides, according to the setting of state / action in Equation (2) (3), the actor network (i.e., bid agent) is built as 70x512x128x1 as presented in FIG.6, and the critic network is built as 70x512x1 .

[0134] FIG. 6 illustrates a diagram of a deep neural network (DNN)-based actor network, also referred to as a bidding agent, in accordance with an embodiment of the present invention. The diagram includes an input layer 610 that receives PV data from the previous day, along with PV data and state of charge (SoC) data from the current day. The network further comprises one or more hidden layers 620 utilizing ReLU (Rectified Linear Unit) activation functions, and an output layer 630 employing a sigmoid activation function to generate the bidding action.

[0135] FIG.7 shows the episode rewards during offline learning according to an embodiment of the present invention. According to the episode rewards shown, the algorithm is capable of searching for the optimal policy with a stable learning procedure and a fast convergence speed.

[0136] FIG.8A to 8C show a set of test results for the implemented data-driven bidding strategy as an example embodiment of the present invention.

[0137] FIG. 8A shows PV generations and bidding power outputs by the example embodiment. FIG.8B and FIG.8C show ESS outputs and SOC values during dispatch period respectively. The bidding power is relatively high at the beginning by discharging ESS, and then bidding power will follow the trend of PV generation, which is capable to maximize the overall profit during the day. In the end of the day time, ESS will also discharge to make full use of its SoC; but SoC will not reach the lower bound considering the restoration cost.

[0138] FIG.9A to 90 show another set of test results for the example embodiment described herein. Fig.9A shows PV generations with a more unstable output curve and bidding power outputs by the implemented example embodiment. FIG.9B and FIG.90 show ESS outputs and SOC values during dispatch period respectively.

[0139] The present invention is shown to perform a reliable bidding work to maximize the profit and satisfy SoC constraints at the same time.

[0140] The following provides another detailed description of an embodiment of the present invention with mathematical formulations.

[0141] In this embodiment, the bidding problem is formulated in the context of the Singapore electricity market. In the Singapore electricity market, overseas PV plants are required to provide its electricity offer for sale in every dispatch period (30 min), and the submission window for the bidding will be closed 65 min before the dispatch commencement. Besides, to take communication and processing time into consideration, it is normal to reserve 10 min for the bidding submission. As a result, for conventional optimization-based bidding methods, the lead time of PV forecast shall be at least 75 min. This bidding process is illustrated in FIG.10. FIG.10 illustrates an example of bidding process of an overseas PV plant participating in Singapore Electricity market. The various time periods defined in the figure are used in the mathematical formulation of the bidding problem for the embodiment described herein.

[0142] Considering the relatively small capacity of the overseas PV plants at this stage, embodiments of the invention described herein assume the PV plants to be a pricetaker, so only power quantity is bided.

[0143] After the bidding submission, the amount of offered power shall be consistent during the 30-min dispatch period. To smooth PV fluctuation, ESS will release / absorb the power to maintain the total power output at the offered amount. Once the offered amount cannot be satisfied, the PV plant will be charged with high penalty payment for the shortfall. In this study, the objective of optimal bidding strategy is to maximize the profit of PV plant, which can be expressed as: subject to:

[0144] SoPDis(0 = fr(SoC(ty) , SoPai(0 = f2(SoC(t>j) (22) where PBid(t) is the bidding power from PV power plant at time slot t, Ppv(f) is PV generation power at time slot t, Poistt), Pch(t) are discharging, charging power of ESS, Psho(t) is the deficient power when the bidding power cannot be satisfied by PV plant, y refers to the electricity price that PV plant sells power to the grid, yprefers to the electricity price paid for the deficient power, CESS(0 is the ESS operation cost at time slot t. Equations (19)-(21 ) give the constraints of ESS output powers, where SoPois(t) and SoPch(t) denote the SoP (State of Power) for discharging and charging. Besides, SoPs can be modeled as functions of SoC as shown in (22). Equations (23) and (24) define the constraints of SoC, where qd, qcare discharging, charging efficiency, E is the capacity of ESS, SoCmax, SoCmin denote the upper, lower bounds of SoC. Further, the operation cost of the ESS, CESS(0, is modeled as a function of SoC and SoH.

[0145] To better analyze its discharging / charging characteristic, ESS is modeled based on an equivalent circuit model (ECM) as illustrated in FIG.11 , which is comprised by N parallel strings and M series-connected cells per string.

[0146] For each cell in this ECM, the first-order Resistor-Capacitor (RC) model is used to emulate the battery’s behavior, which includes a voltage source Voc'm,i)representing open-circuit voltage (OCV), a resistor Ro(m ,,)representing the internal resistance, and a parallel RC circuit Ci(m n)representing the ohmic effect and polarization effect, m=1 , 2, ... , M, n=1 , 2, ... , N. Based on this ECM, the terminal voltage of a battery cell Vh(m n!' can be obtained as: where vi(m n)indicates the polarization voltage of RC circuit, in indicates the current through nth string. Besides, vocand the electric elements (Ro, Ri, Ci) are modeled as polynomial functions of SoC, which are expressed as: where ^ik, ^2k, ^3k, ^4k are polynomial coefficients. Further, considering a discrete time interval [t, t+At], vi can be written as: where r is the time constant that T(SOC) - Ri(SoC) C-i(SoC). Therefore, the terminal voltage can be rewritten as a function of SoC and current:

[0147] Based on the models above, SoP of ESS can be estimated as the maximum available power at the certain time horizon. For one battery cell, its power can be calculated by P = vt, in. To find the maximum output power, current constraint of the battery cell is first considered. According to the ratings of battery cell, the maximum charging / discharging currents can be obtained as ic,max, id, max. Besides, the terminal voltage v& can be expressed as a function of SoC and current inaccording to (33). Therefore, assume that the current reaches the upper / lower bound, the discharging / charging power range can be found as:

[0148] Similar, voltage constraint is also considered to calculate the available discharging / charging range. The upper and lower bound of terminal voltage can be obtained from battery ratings as Vb.max, vt>,min. Assuming that the terminal voltage reaches the bounds, the available power range is determined by the string current. According to (32), the string current is calculated as:

[0149] Besides, since voc, Ro, Ri, rare the functions of SoC, (36) can be rewritten as:

[0150] Therefore, according to the terminal voltage constraint, another set of available discharging / charging power range can be obtained as:

[0151] Apart from current and voltage constraints, SoC constraint is also considered as:

[0152] SoC:< SoC(t) < SoC (40)

[0153] By integrating all power ranges given in (35) (39) (41 ) and considering the rated maximum power output, the feasible region of State of Power (SoP) is determined as: where Pois.max, Pch.max indicate the rated maximum power when the battery discharges and charges. In this embodiment, SoP is firstly calculated based on one battery cell, and the overall SoP of the whole ESS is obtained by multiplying the number of battery cells.

[0154] To evaluate the health status of ESS, State of Health (SoH) may also be taken into consideration, which, in this embodiment, is defined by the remaining capacity of ESS:

[0155] SoH (?) = £(?) / Eratex 100% (44) where E(t) indicates the remaining capacity at time slot t, Erate indicates the rated capacity. The degradation of SoH, 8SoH(t) can be calculated as following: where S denotes the degradation factor, 6DOD denotes the factor of depth of discharging (DoD), 8cai denotes the calendar factor. These two factors SDOD, Scaiare calculated as: where KDI, KDZ, KDJ, AZ- denote empirical coefficients. Based on this SoH definition, the cost of ESS degradation can be calculated as: where by indicates ESS capacity degradation cost per kWh.

[0156] The embodiment described herein provides a learning-based bidding strategy for PV- ESS power plant. First, the overall bidding problem is modeled into a Markov Decision Process (MDP). Then, the general framework of the proposed methodology is explained. After that, policy learning process based on Proximal Policy Optimization (PPO) algorithm is introduced. The optimization problem (17)-(25) are highly nonlinear and non-convex. As an alternative of conventional optimization-based methods, DRL-based methods can solve nonlinear / non-convex problems through the interaction between environment and agent. To apply DRL in this bidding problem, the most difficult part is to model the problem into a Markov Decision Process (MDP), i.e., how to design state, action, reward, according to the specific system (Singapore Electricity Market), which is introduced as follows.

[0157] At the time slot t, the bidding agent generates the action a(f) (the command of bidding power) based on the observed state s(f) as a = TT(S|9), where 6 indicates the parameters of TT. After action a(t) is executed in the environment, a reward r(t) can be obtained to evaluate the performance under the pair of {s(t), a(f)}. Based on this process, the overall bidding problem is formulated as an MDP:

[0158] State: To deal with the stochastic natures of PV generations, PV data are considered in the state formulation, which may include PV data in the same period of the day before, historic PV dataset. According to the bidding process shown in FIG.10, the dataset before f-75min is available for utilization. Hence, by combining historic SoC and PV data, the state may be formulated as:

[0159] Since the state may contain historic PV power information, it implicitly integrates the forecasting of PV power output into the proposed method. Hence, the proposed bidding strategy in the embodiment described herein does not require extra PV forecasting modules.

[0160] Action: The action may be defined as the bidding power of the PV-ESS plant:a( ^Bid ) (52)

[0161] The time-resolution of the action may be defined as 30 minutes to comply with the electricity market rules in Singapore.

[0162] Reward: To evaluate the performance of an action-state pair {s(f), a(f)}, the reward may be designed based on the objective defined in (17): where the last item CRes(t) denotes the charging cost of ESS for restoring its SoC to 50% during nighttime, YR denotes the electricity price during SoC restoration. It should be noted that this restoration cost is only added at the end of the day (f = T). Besides, SoH may be considered in the reward as one of objectives, i.e. C SS. The goal of the optimal bidding strategy of the embodiment described herein is to maximize the accumulated reward.

[0163] To solve this MDP-based bidding problem, a learning-based strategy is developed to find an optimal bidding policy of the agent. The proposed method may be divided into two stages: offline learning and online application, as illustrated in FIG.12.

[0164] FIG. 12 shows an example of the general framework of the method according to the embodiment described herein. While the MDP problem formulation remains consistent with that shown in FIG. 4, this figure highlights the distinction between offline training 1210 and online deployment of the trained DNN-based bidding agent 1220.

[0165] For initialization, the bidding agent may be formulated based on a deep neural network (DNN). At offline learning stage, the state, consisting of historic SoC and PV data, may be observed from the environment (PV plant) as the agent’s input. Then, the action (bidding power command) is generated based on the agent’s policy and then executed in the environment (PV plant). Meanwhile, the reward is obtained to evaluate the performance of the agent’s policy. The transition, including state, action, reward, is stored in a replay buffer, which is used to optimize the policy during the learning process. After iterative learning, the trained bidding agent can be applied for online application stage. Based on real-time collected state, the bidding power command is calculated and provided for power plant, which does not require extra PV forecasting modules.

[0166] Noted that the proposed method is not only limited to Singapore market, rather, it can be used in different scenarios where PV-ESS power plants need to bid in the wholesale market. Singapore electricity market is chosen to demonstrate a use case of the present embodiment. For other electricity markets, the formats of state, action, reward may be adjusted according to the new market rules. Rather than the unique market characteristics, the present invention generally develop a data-driven bidding strategy for a PV-ESS plant while considering detailed ESS models.

[0167] For the embodiment of the invention described herein, an actor-critic-based algorithm, Policy Learning by Proximal Policy Optimization (PPO), may be used to search for the optimal bidding policy, which consists of two types of DNN: “actor” and “critic”. The actor indicates the policy function m(s|9) mapping the state to the action. The critic indicates the value function V^slSy) mapping the state to a scalar that evaluates the state’s quality. The actor is built based on a standard normal distribution for better exploration: where , 09 are two parameterized DNNs that generate mean and covariance, f is a random variable from a standard normal distribution. During iterations, the algorithm aims to maximize the expectation of rewards Re. where pg is the probability of taking action a(f) under state s(t). The gradient for the expected reward is calculated as:

[0168] VRe can update the policy in the direction that increases the probability of action with a larger reward. To better describe this expected reward, an advantage function A is designed as: where A refers to the discount factor. This advantage function indicates the advantage of performing a(t) under state s(t) over the expected reward of all actions. The parameters 6v of the value function V (critic) are optimized by minimizing the loss function Jv(9v) as: where y(f) is the training target of the value function, Av is the learning rate. Different from existing off-policy algorithms, such as soft actor-critic, deep deterministic policy gradient, PPO updates the actor in an on-policy manner, which can make better use of samples and provide a better convergence guarantee. Therefore, the loss function for actor training JTAO) is defined as: where 9' indicates the old parameters of policy function before updating the actor, he(f) is the ratio between the updated policy and the old policy. During the on-policy learning, the sampled data from 9' can be reused to train 9 many times. On the other hand, if 9 and 9' have a large difference or a limit number of samples are collected, they would have a large variance and the learning performance may degrade. To prevent this case, a clipped probability ratio is used to ensure the actor parameters update within a trust region. The loss function Jn(9) can be written as: where c is the range of the trusted region for updating, clip( ) denotes the clipped function to prevent unsafe policy updates. Based on this loss function Jn(9), the parameters of policy function are updated based on gradient descent: where An denotes the learning rate.

[0169] The overall algorithm of the example embodiment is shown in Algorithm 2 as follows:

[0170] Algorithm 2: Policy Learning by PPO Algorithm

[0171] 1 : Initialize actor network TT~9, critic network V~9v

[0172] 2: for episode = 1 to epimax do

[0173] 3: Randomly determine an initial state s(1).

[0174] 4: for t= 1 to T do

[0175] 5: Generate action aft) with probability pft) based on state sft).

[0176] 6: Execute action aft), get reward rft), observe new state s(t+ 1 ). 7: Store the transition {s(t), a(t), p(t), r(t), s(t+1 )} into a memory cell Ce.

[0177] 8: end for

[0178] 9: Store Ce into a replay buffer R.

[0179] 10: L cells {Ce1, Ce2, ... , Ce'-} are selected from R.

[0180] 11 : for I = 1 to L do

[0181] 12: Calculate advantage functions for {s, a} in Ce'.

[0182] 13: Update the critic’s parameters based on (46).

[0183] 14: Update the actor’s parameters based on (50).

[0184] 15: end for

[0185] 16: end for

[0186] Several case studies are carried out in this section in relation to the embodiment described herein. The PV generation data are obtained during the time range from 22- Oct-2017 0:00 to 31 -Dec-2017 23:30, with the time resolution of 1 minute. To exemplify, the PV generation from first seven days are plotted in FIG.13.

[0187] Besides, the parameters of the PV-ESS power plant are summarized in Table II. All tests are conducted by Matlab R2020b software on a laptop with AMD Ryzen 74800H 2.9 GHz CPU and 16 GB RAM. Parameters of PV-ESS Plant.

[0188] Parameter Value

[0189] Maximum PV generation: Ppv.max 600 MW

[0190] Capacity of ESS: E 120 MWh

[0191] 120MW,

[0192] Rated power of dc-dc converter: Pois.max, Pch.max

[0193] -120 MW

[0194] Upper, lower bounds of SoC: SoCmax, SoCmin 95%, 5% ESS discharging, charging efficiency: r]d, Qc 0.95, 0.97

[0195] Electricity price that PV-ESS plant sells: y $200 / MWh

[0196] Electricity price paid for the deficient power: yp$400 / MWh

[0197] ESS capacity degradation cost: XESS $115 / kWh

[0198] ESS discharging, charging efficiency: rjd, r)c0.95, 0.97

[0199] The ESS model is firstly validated based on the equivalent circuit model in FIG.11 , which contains / V=180 parallel-connected strings and / W=1500 series-connected cells per string. The polynomial parameters of electric elements (vOc, Ro, Ri, Ci) in (28)-(31 ) are from previous studies, and the curves of these polynomial functions are presented in FIG.14A to 14D. FIG.14A shows the polynomial curve of voc. FIG.14B shows the polynomial curve of Ro. FIG.14C shows the polynomial curve of Ri . FIG.14D shows the polynomial curve of Ci.

[0200] Besides, the parameters of the ESS model are listed in Table III.

[0201] TABLE III Parameters of ESS Modeling

[0202] Parameter Value

[0203] Maximum charging / discharging currents 140A, ic,max, l'd, max ~ 120A

[0204] Maximum, minimum terminal voltage

[0205] 3.6V, 2.5V

[0206] Vb,min, Vb,max

[0207] Coefficient of the DoD factor: KDI 1 .4ex105

[0208] Coefficient of the DoD factor: KD2 -0.503

[0209] Coefficient of the DoD factor: KD3 -1 .2x105

[0210] Coefficient of the calendar factor: KC 4.14X 10-10

[0211] Based on (33), the terminal voltage Vb can be calculated according to different values of SoC and charging / discharging current in, which is plotted in FIG.15.

[0212] If a different type of battery is used, its nameplate parameters are needed, which can be obtained by battery manuals.

[0213] Based on (42), (43), the SoP-SoC relationship is obtained and shown in FIG.16A and 16B. FIG.16A shows a SoP-SoC plot during ESS discharging. FIG.16B shows a SoP- SoC plot during ESS charging.

[0214] As presented in FIG.16A, discharging SoP will be determined by SoC constraint if SoC is low, since ESS does not have enough stored power to discharge. When SoC increases, discharging SoP will be determined by voltage constraint to ensure that the terminal cell voltage may go beyond the allowable range, i.e. SoP = SoPvb - Vb,max-in(SoC(t), Vb.max). Besides, there is a certain range (SoC is around 12~20%) that the current through strings may go beyond the maximum allowable range. So, in such range, discharging SoP is determined by the current constraint, i.e. SoP = SoPin = Vb(SoC(t), id, max)' id, max-

[0215] As shown in FIG.16B, ESS’s charging SoP will be low if SoC is high, because the store energy almost reaches upper bound. When SoC decreases, charging SoP will be determined by current constraint to make sure that the current through strings shall be lower than the maximum allowable current, i.e. SoP - SoPin - Vb(SoC(t), ic, max)- io, max. When SoC further decreases (lower than around 7~9%), the voltage constraint will become dominant and charging SoP will equal to SoPvb = Vb,min in(SoC(f), Vb,min).

[0216] After system modeling, offline training is conducted based on PPG algorithm, searching for an optimal bidding strategy. According to MDP defined above, the actor network (bidding agent) is built by two DNNs with the structure of 90x800*1 , for mean and covariance respectively in (55). Besides, the critic network (state function) is also built with the structure of 90x800x1. For initialization, these DNNs are assigned with random weight and bias. The activation functions of hidden layers are set by “relu” function in both actor and critic. The activation functions of output layers are set by “relu” function in the critic and “tanh” function in the actor. At the beginning of each learning episode, PV historic data are randomly selected for policy exploration. During the learning episode, the bidding agent runs for multiple iterations and the transition {s(f), a(t), (f), r(t), s(f+1 )} is stored into a memory cell, which is implemented in Matlab interface. After several episodes, a number of cells can construct as a replay buffer. To update the policy, several memory cells are selected from the replay buffer to update the agent’s parameters. Through numerous learning episodes, the bidding agent can be well-trained under various PV generations and is applied for online application. The parameters for offline training are listed in Table IV.

[0217] TABLE IV Parameters of the learning algorithm.

[0218] Parameter Value

[0219] Structure of actor network 90x800x1

[0220] Structure of critic network 90x800x1

[0221] Actor learning rate An1 e-4

[0222] Critic learning rate 1 e-3

[0223] Optimizer Adam

[0224] Range of the trusted region e 0.1

[0225] Size of replay buffer NR 100

[0226] Size of minibatch L 20

[0227] The episode rewards are plotted in FIG.17.

[0228] Since the initial state is randomly selected in each episode, the reward fluctuates during the learning. To show the clear trend of episode rewards, average rewards per 50 episodes are calculated and illustrated in FIG.17 as trend lines.

[0229] For comparison, Soft Actor-Critic (SAC) and Deep Deterministic Policy Gradient (DDPG) algorithms are also tested, which reflects the better performance of the adopted PPG. Although SAC has a good capability of policy exploration as an entropybased algorithm, the learning process is unstable, which may not converge. As a result, PPG algorithm shows a better learning process compared to existing DRL algorithms in the bidding problem formulated for the embodiment described herein.

[0230] After multiple iterations of training, the bidding agent can be applied for online stage. To validate the performance of the bidding strategy described in the embodiment, several case studies are given in this section. It should be noted that the overall dataset contains PV data of 71 days, where the data of first 40 days are used as training dataset and the remaining 31 -day data (day 41 -71 ) are used for testing. Therefore, training and testing dataset are not overlapped.

[0231] FIG.18A to 18D presents the test case within one week (from day 50 to day 56). FIG.18A shows a plot of PV generations and bidding results. FIG.18B shows a plot of ESS’s output power, wherein the proposed method according to the embodiment is compared with a method excluding consideration of SoP. FIG.18C shows a plot of ESS’s SoC, wherein the proposed method according to an embodiment of the present invention is compared with a method excluding consideration of SoP. FIG.18D shows a plot of ESS’ SoH, wherein the proposed method according to the embodiment is compared with a method excluding consideration of SoH.

[0232] In FIG.18A, the proposed agent according to the embodiment can follow variable PV generations and generate the bidding power command by historic information, which does not need PV power forecasting. A comparative analysis is provided to explain the necessity for SoP and SoH modeling. Since the real SoP does not match the rated maximum power according to the discussion above, the ESS output may exceed realtime SoPs if not considering SoP, as shown in FIG.18B. Taking SoP into consideration, the real output power of ESS shows a relatively lower level. Besides, the variation of SoC also reflects a smoother operation of ESS when considering SoP in ESS modeling. During the operation, SoC can be kept within the constraints [5%, 95%]. In the end of daytime (19:00 ~ 20:00), ESS’s SoC is restored to 50%. For SoH modeling, the initial SoH is defined as 99%, indicating the ratio of the remaining ESS capacity. By taking the cost of SoH capacity into the reward given in (50), the degradation can be improved during operation, as presented in FIG.18D.

[0233] To further explain the effectiveness of the method presented in the embodiment of the present invention, it is compared with classic model (optimization)-based methods. The optimization problem is formulated as (17), where the time-resolution is also defined as 30 minutes. To ensure the long-term profit, SoC is required to restore to 50% in each time horizon (30 minute). Noted that it is reasonable that SoC is required to restore to 50% in each time horizon. First, if SoC is not required to restore to 50% in each time horizon, ESS cannot be timely charged by the optimization-based method for real-time bidding. Second, to restore SoC to 50% in each time horizon (30 minute) still can fully utilize PV generation and will not sacrifice any potential profits.

[0234] FIG.19A to 19C show comparison between a proposed method according to an embodiment of the present invention and an optimization-based method with real PV data. FIG.19A shows a plot of bidding results. FIG.19B shows a plot of ESS’s SoC. FIG.19C shows a plot of average profit.

[0235] In the tests, historic data of day 42 to 48 are used. First, the bidding results of the proposed and model-based method in the embodiment described herein are presented in FIG.19A where the optimization-based method utilizes real PV data as the input. As shown, these two curves have a similar trend as PV variation. SoC curve is also given in FIG.19B. Compared to the model-based method which makes a limited use of ESS, the proposed method is capable to utilize ESS within a larger range of SoC. Moreover, the average profits per 30 minutes are plotted in FIG.19C. The proposed method as according to the described embodiment can achieve similar profits with the optimization-based method during the operation. It should be noted that in this case, true PV data is used for the optimization-based method.

[0236] FIG.20A to 20E show comparison between a proposed method according an embodiment of the present invention and an optimization-based method. FIG.20A shows a plot of measured and predicted PV data with 10% RMSE. FIG.20B shows a plot of bidding results of proposed method and optimization-based method (predicted PV of 0% and 10% RMSE). FIG.20C shows a plot of ESS’ SoC with proposed method and optimization-based method (predicted PV with 0% and 10% RMSE). FIG.20D shows a plot of average profits with proposed method and optimization-based method (predicted PV with 5%, 10% and 15% RMSE). FIG.20E shows a plot of accumulated profits with proposed method and optimization-based method (predicted PV with 0%, 5%, 10%, 15% and 20% RMSE). However, true PV power outputs are unavailable at real-time stage in practice, which shall be predicted based on historic information. In this regard, the prediction errors could affect the bidding results. In this study, a normally distributed random signal is added to the real PV power for simulating prediction errors. First, an illustration of measured PV and predicted PV with 10% root mean square error (RMSE) are given in FIG.20A. Based on PV outputs with predicted errors, the performance of the optimization-based method will be degraded, as shown in FIG.20B. Besides, SoC variations are given in FIG.20C, where the proposed method according to the embodiment described herein performs a better utilization of ESS compared to the optimization-based method. Furthermore, to compare the performance under different levels of prediction errors, the average profits per 30 minutes are presented in FIG.20D. As shown in the figure, when the optimization-based method uses predicted data with small errors (5%), the proposed method shows a similar profit curve as the modelbased method. As the prediction error increases (from 5% to 10% RMSE), the average profit decreases significantly. In FIG.20E the accumulated profits are also shown. The method proposed in the embodiment described herein shows a reliable bidding performance with a high accumulated profit, close to the optimization-based method using true PV. When the forecasting error reaches 10% or a higher level, the bidding strategy in the present embodiment can outperform the optimization-based approach.

[0237] Whilst the foregoing description has described exemplary embodiments, it will be understood by those skilled in the art that many variations of the embodiments can be made within the scope and spirit of the present invention.

Claims

CLAIMS1. A method of controlling a renewable energy plant comprising a renewable energy generation system and an energy storage system, the method comprising: receiving state data of the renewable energy plant, the state data comprising power output data of the renewable energy generation system and state of charge data of the energy storage system; generating a power output bid indicating a power output for the renewable energy plant over a subsequent dispatch period by inputting the state data into a deep reinforcement learning model which is trained to maximise a reward function over a generation cycle comprising a plurality of dispatch periods including the subsequent dispatch period, wherein the reward function is formulated from a positive reward proportional to a power delivered in each dispatch period of the generation cycle and a penalty when a power delivered falls short of a power output bid for a dispatch period of the generation cycle; and controlling the renewable energy plant to output power at the power output bid over the subsequent dispatch period.

2. The method of claim 1 , wherein the deep reinforcement learning model is trained with historic state data of the renewable energy plant.

3. The method of any preceding claim, wherein the state data comprises power output data from the renewable energy generation system and state of charge data from the energy storage system for a time interval preceding receipt of the state data.

4. The method of any preceding claim, wherein the state data comprises power output data from the renewable energy generation system for a time interval on previous day that corresponds to a time interval following receipt of the state data.

5. The method of any preceding claim, wherein the generation cycle is a daily cycle.

6. The method of any preceding claim, wherein the renewable energy plant is a photovoltaic plant.

7. The method of any preceding claim, wherein the formulation of reward function further includes a penalty component based on a cost associated with restoring state of charge of the energy storage system.

8. The method of any preceding claim, wherein proximal policy optimization is used in the deep reinforcement learning model.

9. The method of any preceding claim, wherein the step of controlling the renewable energy plant comprises controlling the energy storage system to release electrical power when output of the renewable energy generation system is insufficient to satisfy the power output bid, or to absorb electrical power when output of the renewable energy generation system exceeds the power output bid.

10. A computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a method according to any one of the preceding claims.

11. A system for controlling a renewable energy plant comprising a renewable energy generation system and an energy storage system, the system comprising: a processor; data storage storing a deep reinforcement learning model which is trained to maximise a reward function over a generation cycle comprising a plurality of dispatch periods including a subsequent dispatch period, wherein the reward function is formulated from a positive reward proportional to a power delivered in each dispatch period of the generation cycle and a penalty when a power delivered falls short of a power output bid for a dispatch period of the generation cycle; and program storage storing computer program instructions operable to cause the processor to: receive state data of the renewable energy plant, the state data comprising power output data of the renewable energy generation system and state of charge data of the energy storage system; generate a power output bid indicating a power output for the renewable energy plant over the subsequent dispatch period by inputting the state data into the deep reinforcement learning model; andcontrol the renewable energy plant to output power at the power output bid over the subsequent dispatch period.

12. The system of claim 11 , wherein the deep reinforcement learning model is trained with historic state data of the renewable energy plant.

13. The system of any one of claims 11 to 12, wherein the state data comprises power output data from the renewable energy generation system and state of charge data from the energy storage system for a time interval preceding receipt of the state data.

14. The system of any one of claims 11 to 13, wherein the state data comprises power output data from the renewable energy generation system for a time interval on previous day that corresponds to a time interval following receipt of the state data.

15. The system of any one of claims 11 to 14, wherein the generation cycle is a daily cycle.

16. The system of any one of claims 11 to 15, wherein the renewable energy plant is a photovoltaic plant.

17. The system of any one of claims 11 to 16, wherein the formulation of reward function further includes a penalty component based on a cost associated with restoring state of charge of the energy storage system.

18. The system of any one of claims 11 to 17, wherein proximal policy optimization is used in the deep reinforcement learning model.

19. The system of any one of claims 11 to 18, wherein the program storage further stores computer program instructions operable to cause the processor to control the energy storage system to release electrical power when output of the renewable energy generation system is insufficient to satisfy the power output bid, or to absorb electrical power when output of the renewable energy generation system exceeds the power output bid.

20. The system of any one of claims 11 to 19, wherein the program storage further stores computer program instructions operable to cause the processor to train the deep reinforcement learning model using historic state data.

Images