CHP MCS optimization scheduling method and device based on deep reinforcement learning

Abstract

The present application relates to a kind of CHPMCS optimization scheduling method and equipment based on deep reinforcement learning, the method includes the following steps: obtaining the relevant data of cascade water storage wind-solar complementary system;Based on the relevant data, the multi-constraint short-term optimization operation model of cascade water storage wind-solar complementary system is established;The multi-constraint short-term optimization operation model is converted into unconstrained reinforcement learning task;Deep deterministic policy gradient algorithm is used to solve the unconstrained reinforcement learning task, and the optimal output is obtained, to realize the optimal scheduling of cascade water storage wind-solar complementary system.Compared with prior art, the present application has the advantages of improving power system reliability, resource utilization and flexibility, and economic benefits.

Description

Technical Field

[0001] This invention relates to the field of power systems, and in particular to a CHPMCS optimization scheduling method and device based on deep reinforcement learning. Background Technology

[0002] Currently, the space for hydropower development is gradually narrowing. By the end of 2022, the installed capacity of conventional hydropower reached 414 million kilowatts, accounting for 61.25% of the theoretical total potential. Furthermore, the difficulty of subsequent development is gradually increasing. Therefore, reshaping conventional hydropower stations by converting them into pumped storage systems to form a "Cascade Hydropower-Pumping-Storage-Wind-Photovoltaic Multi-energy Complementary System (CHPMCS)" has become an inevitable choice to meet the large-scale flexibility resource demands of the "new power system." Thus, by reshaping the role of hydropower, it is necessary to gradually shift from a "power supplier" to a "power supplier + 'battery' regulator," in order to better leverage the regulation and storage functions of hydropower.

[0003] As a new trend and innovative model for the development of future clean energy bases, CHPMCS (Hybrid Pumped Storage and Coal Mine) has yielded limited results in terms of optimized operation. Theoretical exploration and technological innovation can primarily draw upon the experience of cascade wind-solar-hydro complementary systems and hybrid pumped storage power stations in their operation and management. Regarding the operation of hybrid pumped storage power stations, the redesign of the "pumped storage" function leads to more complex operating conditions and a sharp increase in control difficulty. Coordinating pumped storage with conventional units to improve overall efficiency while responding to grid flexibility requirements becomes a major challenge. The optimized operation of CHPMCS is essentially a sequential decision-making problem with multiple stochastic factors, centered on cascade hydropower stations / hybrid pumped storage power stations. Commonly used methods include quadratic programming, dynamic programming and their improved versions, as well as intelligent algorithms such as genetic algorithms and particle swarm optimization. With increasingly diverse optimization objectives and a surge in complex and coupled operational constraints, dynamic programming methods face a severe "curse of dimensionality," and intelligent algorithms are prone to getting trapped in local optimization, making it difficult to meet the requirements of accuracy and timeliness. Artificial intelligence reinforcement learning, through continuous exploration and trial and error in dynamic environments, possesses learning and memory capabilities, exhibiting better adaptability to complex sequential decision-making problems. It can effectively address the challenges of solving CHPMCS operational models and has become a mainstream development direction in the context of digital transformation of energy and power systems. However, few publications employ deep reinforcement learning algorithms to analyze the optimal scheduling problem of wind-solar-hydro complementary systems, particularly the intelligent optimal scheduling problem of hybrid cascade hydropower stations containing pumped-storage units and complementary power generation systems of new energy sources.

[0004] Therefore, it is necessary to design a control method for a cascade hydro-storage-wind-solar hybrid system based on deep reinforcement learning to address how CHPMCS can improve the resource utilization, flexibility, and economic benefits of the power system. Summary of the Invention

[0005] The purpose of this invention is to provide a CHPMCS optimization scheduling method and device based on deep reinforcement learning to improve system reliability.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A CHPMCS optimization scheduling method based on deep reinforcement learning includes the following steps:

[0008] Obtain relevant data on the cascade hydro-storage-wind-solar hybrid system;

[0009] Based on the aforementioned data, a multi-constraint short-term optimization operation model for CHPMCS is established.

[0010] The multi-constraint short-term optimization model is converted into an unconstraint reinforcement learning task.

[0011] The unconstrained reinforcement learning task is solved using a deep deterministic policy gradient algorithm to obtain the optimal output, thereby achieving optimal scheduling of CHPMCS.

[0012] Furthermore, the relevant data includes operating cycles, total number of hydropower stations, total number of wind power stations, electricity sales, total number of photovoltaic power stations, electricity sales price, and abandoned electricity.

[0013] Furthermore, the expression for the multi-constraint short-term optimization model is:

[0014]

[0015] The constraints on transmission capacity are as follows:

[0016]

[0017] The constraints for wind power plants are:

[0018]

[0019] The constraints for photovoltaic power plants are:

[0020]

[0021] The constraints for the power curtailment penalty are:

[0022] 0≤P ab (t)≤P ab,max

[0023]

[0024] The constraints for the operation of a hydropower station are:

[0025]

[0026]

[0027] The constraints for pumped storage power stations are:

[0028]

[0029]

[0030]

[0031] The constraints for cascade hydropower stations are:

[0032]

[0033]

[0034] The constraints on the reservoir capacity are:

[0035]

[0036] In the formula, ER is the daily power generation benefit of the CHMCS system, T is the operating cycle of 24 hours; J is the total number of hydropower stations, K is the total number of wind power stations, and M is the total number of photovoltaic power stations. Let j be the electricity sold by hydropower station j at time t. and Let k be the electricity sold by wind power and m by photovoltaic power at time t, respectively. The electricity price of the system at time t; and These represent the amount of electricity purchased from the grid at time t and the corresponding price, respectively. and Let λ be the amount of electricity discarded by the system at time t. abandon The price is the penalty for abandoning electricity; This represents the power generation of hydropower station i during time period t. This represents the power generation of wind farm i during time period t. p represents the power generation of the photovoltaic power station during time period t. t,max This represents the maximum transmission capacity of the system during time period t; This represents the total power generation of wind power station i during time period t. This represents the amount of electricity generated by wind power station i in time period t. This represents the amount of electricity consumed by the pumped storage power station i during time period t. This represents the total power generation of photovoltaic power station i during time period t. This represents the amount of electricity generated by photovoltaic power station i in time period t. P represents the amount of electricity consumed by the photovoltaic power station i during time period t; ab,max P represents the upper bound of the power curtailment during the scheduling cycle; ab,sum This serves as the upper bound of the total amount of power wasted during the entire scheduling cycle. This represents the total power generation of hydropower station i during time period t. This represents the amount of electricity generated by hydropower station i in time period t. This represents the amount of electricity generated by hydropower station i during time period t that is consumed by the pumped-storage power station. and Let i represent the minimum and maximum power generation of hydropower station i, respectively. This indicates the total electricity consumed by the pumped storage power station. This represents the amount of electricity consumed by the pumped-storage power station i during time period t. This represents the amount of electricity consumed by the pumped-storage power station i during time period t. This represents the amount of electricity generated by hydropower station i during time period t that is consumed by the pumped-storage power station. This represents the amount of electricity a pumped storage power station purchases from the grid during time period t; and This represents the generating power and pumping power of the unit during time period t, where T is the operating cycle; f represents the pumping power of the l-th pumped storage unit during time period t; l (·) represents the relationship function between the pumping power, head, and flow rate of the l-th pumped storage unit; H represents the power generation flow of the l-th pumped storage unit during time period t; l,t This represents the generating head of the l-th pumped storage unit during time period t; v t+1 v represents the reservoir capacity at time t+1. t I represents the reservoir capacity at time t. t This represents the inflow rate of the reservoir during time period t. This represents the amount of water consumed by the pumped storage power station during time period t. This represents the power generation flow of the reservoir during time period t. This represents the discharge flow rate of the reservoir during time period t; This represents the minimum water storage capacity of reservoir i. This represents the maximum water storage capacity of reservoir i.

[0037] Furthermore, the specific steps for converting to an unconstrained reinforcement learning task include:

[0038] The state space and action space are selected based on the aforementioned multi-constraint short-term optimization operation model;

[0039] The output increments of wind power, photovoltaic power, hydropower stations, and pumped storage power stations are selected as the action space;

[0040] Set the objective function as the reward function for the unconstrained reinforcement learning task;

[0041] Based on the reward function, the action value function is defined according to policy π, thereby realizing the transformation from a multi-constraint short-term optimization model to an unconstrained reinforcement learning task.

[0042] Furthermore, reservoir water level, power grid load, electricity price, wind speed, and solar irradiance are selected as the state space, and the expression for this state space is:

[0043] s t ={s level ,s load ,s price ,s w ,s pv}

[0044] In the formula, S t Let S be the state space. level S represents the reservoir water level. load For the power grid load, S price For electricity price, S w For wind speed, S pv Light intensity.

[0045] Furthermore, the output increments of wind power, photovoltaic power, hydropower stations, and pumped storage power stations are selected as the action space, and the expression for this action space is:

[0046] a t ={a w ,a pv ,a h ,a pump}

[0047] In the formula, a t Let a be the state space. w For the increase in wind power output, a pv To increase photovoltaic power output, a h To increase the power output of the hydropower station, a pump This is to increase the output of pumped storage power stations.

[0048] Furthermore, the expression for the reward function is:

[0049]

[0050] In the formula, R tS represents the cumulative reward obtained by the agent after obtaining the corresponding scheduling plan based on the system's external state variables; γ is a discount factor, indicating the relative importance of future rewards compared to the current reward. When γ = 0, only the current immediate reward is considered, without considering future long-term rewards; when γ = 1, the future long-term reward is equally important as the current immediate reward; t For state space; a t This is the state space.

[0051] Furthermore, the expression for the action value function is:

[0052] Q π (s,a)=E π [R t |s=s t a=a t ]

[0053] In the formula, Q π (s, a) is the action value function, E π R is the expected cumulative return calculated according to strategy π. t S is the cumulative reward obtained by the agent after obtaining the corresponding scheduling plan based on the external state variables of the system. t Let a be the state space. t This is the state space.

[0054] Furthermore, the reinforcement learning algorithm is a dual-delay deep deterministic policy gradient algorithm.

[0055] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs stored in the memory, said one or more programs including instructions for executing the CHPMCS optimization scheduling method based on deep reinforcement learning as described above.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) This invention establishes a multi-constraint short-term optimization operation model with the goal of maximizing the power generation benefits of CHPMCS. Considering the characteristic that CHPMCS output is continuously adjustable, the optimization scheduling problem is transformed into a Markov decision process, thereby transforming the multi-constraint optimization problem into an unconstrained reinforcement learning task. The reinforcement learning algorithm is used to solve the problem and obtain the optimal output of each unit, thereby achieving the optimal scheduling of the system and significantly improving the reliability of the system.

[0058] (2) The multi-constraint short-term optimization operation model of the present invention includes balance constraints such as transmission capacity, wind power station, photovoltaic power station, hydropower station operation and cascade hydropower station operation, as well as unbalance constraints such as curtailment penalty, pumped storage power station operation and reservoir capacity. By considering these constraints, CHPMCS can consume excess electricity from hydropower, wind power and photovoltaic power for pumped storage, improve the utilization rate of water resources and the absorption of clean energy. At the same time, CHPMCS has better flexibility, ensuring the balance of energy supply and demand and the rational use of resources in the system, while also improving the power generation efficiency of the system.

[0059] (3) The present invention uses a dual-delay deep deterministic policy gradient algorithm to solve the problem, which can avoid the algorithm getting stuck in local optima, improve the performance and stability of reinforcement learning in the continuous action space, and thus improve the stability and reliability of the system operation. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0061] Figure 2 This is a flowchart of the CHPMCS optimization solution according to an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the cascade hydro-storage-wind-solar hybrid system according to an embodiment of the present invention;

[0063] Figure 4 The diagram shows the power output of each unit under different schemes on a typical day during the high water season according to an embodiment of the present invention. Among them, (a) is the typical daily power output of the upstream hydropower station of CHMCS during the high water season, (b) is the typical daily power output of the upstream hydropower station of CHPMCS during the high water season, (c) is the typical daily power output of the downstream hydropower station of CHMCS during the high water season, and (d) is the typical daily power output of the downstream hydropower station of CHPMCS during the high water season.

[0064] Figure 5 The following are the power output diagrams of each unit under different schemes on a typical day during the dry season according to the embodiments of the present invention: (a) is the typical daily power output of the upstream hydropower station of CHMCS during the dry season; (b) is the typical daily power output of the upstream hydropower station of CHPMCS during the dry season; (c) is the typical daily power output of the downstream hydropower station of CHMCS during the dry season; and (d) is the typical daily power output of the downstream hydropower station of CHPMCS during the dry season. Detailed Implementation

[0065] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0066] Example 1

[0067] This embodiment provides a CHPMCS optimization scheduling method based on deep reinforcement learning, such as... Figure 1 As shown, the method includes the following steps:

[0068] S1. Obtain relevant data on the cascade hydro-storage-wind-solar hybrid system.

[0069] Relevant data includes the CHPMCS operating cycle, total number of hydropower stations, electricity sales, electricity price, and abandoned electricity.

[0070] S2. Based on the relevant data, establish a multi-constraint short-term optimization operation model for CHPMCS.

[0071] The specific steps for establishing a short-term optimal operation model with the goal of maximizing system power generation efficiency are as follows:

[0072] S21. Establish a mathematical model of the CHPMCS system to simulate the operation optimization process of multiple energy sources, and add a power curtailment penalty to the objective function. The optimization objective of the CHPMCS system is to maximize the power generation efficiency of the entire system, and its optimization model is as follows:

[0073]

[0074] In the formula, ER is the daily power generation benefit of the CHMCS system, T is the operating cycle of 24 hours; J is the total number of hydropower stations, K is the total number of wind power stations, and M is the total number of photovoltaic power stations. Let j be the electricity sold by hydropower station j at time t. and Let k be the electricity sold by wind power and m be the electricity sold by photovoltaic power at time t, respectively, in MWh. The electricity price at time t is expressed in yuan / MWh. and These represent the amount of electricity purchased from the grid at time t and the corresponding price, respectively. and Let λ be the amount of electricity discarded by the system at time t. abandon The price is the penalty for abandoning electricity;

[0075] S22. Constrain the system transmission capacity. The constraint conditions are as follows:

[0076]

[0077] In the formula, This represents the power generation of hydropower station i during time period t. This represents the power generation of wind farm i during time period t. p represents the power generation of the photovoltaic power station during time period t. t,max This represents the maximum transmission capacity of the system during time period t.

[0078] S23. Constraints are imposed on the wind power station, with the following conditions:

[0079]

[0080] In the formula, This represents the total power generation of wind power station i during time period t. This represents the amount of electricity generated by wind power station i in time period t. This represents the amount of electricity consumed by the pumped storage power station i during time period t.

[0081] S24. Constraints are imposed on the photovoltaic power station, and the constraint conditions are as follows:

[0082]

[0083] In the formula, This represents the total power generation of photovoltaic power station i during time period t. This represents the amount of electricity generated by photovoltaic power station i in time period t. This represents the amount of electricity consumed by the photovoltaic power station i and the pumped storage power station during time period t.

[0084] S25. Constraints are imposed on the power curtailment penalty, with the following constraints:

[0085] 0≤P ab (t)≤P ab,max

[0086]

[0087] In the formula: P ab,max P represents the upper bound of the power curtailment during the scheduling cycle; ab,sum This serves as the upper bound of the total amount of power wasted during the entire scheduling cycle.

[0088] S26. Constraints are imposed on the operation of the hydropower station, with the following conditions:

[0089]

[0090]

[0091] In the formula, This represents the total power generation of hydropower station i during time period t. This represents the amount of electricity generated by hydropower station i in time period t. This represents the amount of electricity generated by hydropower station i during time period t that is consumed by the pumped-storage power station. and These represent the minimum and maximum power generation of hydropower station i, respectively.

[0092] S27. Operational constraints for pumped storage power stations, the constraints are as follows:

[0093]

[0094] In the formula, This indicates the total electricity consumed by the pumped storage power station. This represents the amount of electricity consumed by the pumped-storage power station i during time period t. This represents the amount of electricity consumed by the pumped-storage power station i during time period t. This represents the amount of electricity generated by hydropower station i during time period t that is consumed by the pumped-storage power station. This represents the amount of electricity a pumped storage power station purchases from the grid during time period t.

[0095]

[0096] In the formula: and This represents the generating power and pumping power of the unit during time period t, in MW; T is the operating cycle of 24 hours.

[0097]

[0098] In the formula: f represents the pumping power of the l-th pumped storage unit during time period t; l (·) represents the relationship function between the pumping power, head, and flow rate of the l-th pumped storage unit; H represents the power generation flow of the l-th pumped storage unit during time period t; l,t This represents the generating head of the l-th pumped storage unit during time period t, in meters.

[0099] S28. Constraints on cascade hydropower stations, the constraint conditions are as follows:

[0100] Upstream hydropower station:

[0101]

[0102] Downstream hydropower station:

[0103]

[0104] In the formula, v t+1 v represents the reservoir capacity at time t+1. t I represents the reservoir capacity at time t. t This represents the inflow rate of the reservoir during time period t. This represents the amount of water consumed by the pumped storage power station during time period t. This represents the power generation flow of the reservoir during time period t. This represents the discharge flow rate of the reservoir during time period t.

[0105] Reservoir capacity limits:

[0106]

[0107] In the formula, This represents the minimum water storage capacity of reservoir i. This represents the maximum water storage capacity of reservoir i.

[0108] S3. Convert the multi-constraint short-term optimization model into an unconstrained reinforcement learning task.

[0109] This step transforms the multi-constraint optimization problem into an unconstraint reinforcement learning task. The specific implementation process is as follows:

[0110] S31. In a cascade hydro-storage-wind-solar hybrid system, the reservoir water level, grid load, electricity price, wind speed, and solar irradiance are selected as the state space, i.e.:

[0111] s t ={s level ,s load ,s price ,s w ,s pv}

[0112] S32. Select the output increment of wind power, photovoltaic power, hydropower stations, and pumped storage power stations as the action space, that is:

[0113] a t ={a w ,a pv ,a h ,a pump};

[0114] S33. Set the objective function as the reward function, thereby encouraging the agent to learn the optimal scheduling plan and obtain an immediate reward r. t The formula for calculation is:

[0115] r(a t / s t ) = ER t

[0116] In the formula: r t For an agent in a certain state [s] level ,s load ,s price ,s w ,s pv [Select action below] w ,a pv ,a h ,a pump The instant reward received.

[0117] For the entire scheduling period T, in order to minimize the objective function over the entire scheduling period, there exists a cumulative reward function R. t for:

[0118]

[0119] In the formula: R t γ is the cumulative reward obtained by the agent after obtaining the corresponding scheduling plan based on the external state variables of the system; γ is the discount factor, which represents the importance of future reports relative to the current ones. When γ = 0, it means that only the current immediate report is considered and the future long-term reward is not considered. When γ = 1, it means that the future long-term reward is as important as the current immediate reward.

[0120] S34. For policy π, the cumulative reward can be described by the action value function:

[0121] Q π (s,a)=E π [R t |s=s t a=a t ].

[0122] Among them, E π The expected cumulative return is calculated according to strategy π.

[0123] S4. The unconstrained reinforcement learning task is solved using a deep deterministic policy gradient algorithm to obtain the optimal output, thereby achieving optimal scheduling of the cascade hydro-storage-wind-solar hybrid system.

[0124] like Figure 2 As shown, the solution process for this step is as follows:

[0125] S41. During training, the Actor network parameters are optimized using a gradient descent strategy, i.e.:

[0126]

[0127] S42. The Critic network is updated by minimizing the loss function, i.e.:

[0128]

[0129] y i =r i +γQ i ′(s′,a′|θ q′ )

[0130] S43. After updating the Actor and Critic network parameters, the corresponding updates to the two target network parameters are performed using a soft update method, i.e.:

[0131]

[0132] In the formula, θμ′ For the Actor target network parameters; θ q′ Here are the target network parameters for Critic; τ is the soft update coefficient (0 < τ ≤ 1).

[0133] S44. The improved DDPG algorithm TD3 is used to solve the problem. When calculating the target value, the smaller of the two values ​​is used to estimate the value of the next state-action pair (s′, a′).

[0134]

[0135] The optimal output of each unit is obtained by solving the TD3 algorithm. Under the condition of the optimal output, the maximum power generation efficiency of the system can be achieved.

[0136] To verify the effectiveness of the above method, this embodiment uses a cascade hydro-storage-wind-solar hybrid system structure as an application scenario, such as... Figure 3 As shown. This embodiment includes two examples (for comparative experiments). Example 1 uses the proposed cascade hydro-storage-wind-solar hybrid system (CHPMCS) to optimize and obtain the system's operation plan. Example 2 uses the traditional cascade hydro-storage-wind-solar hybrid system (CHMCS) without pumped storage power station to optimize and obtain the joint system's operation plan.

[0137] The analysis steps in this example include:

[0138] a) Analysis of power generation flexibility of different systems

[0139] b) Economic analysis of power generation in different systems

[0140] Analysis of power generation flexibility of different systems:

[0141] a01) During the high-water season, the reservoir has sufficient water on typical days. Its main goal is to consume the surplus hydropower and minimize water wastage. Therefore, the upstream and downstream hydropower stations of CHPMCS and CHMCS are almost all operating at full capacity, which can easily reach the upper limit of the system's transmission capacity.

[0142] a02) as Figure 4 As shown in (c), during a typical day in the high-water season, downstream hydropower stations of the CHMCS system experience power curtailment between 12:00 and 14:00 due to sufficient wind and solar power output and the constraint of minimum power generation of hydropower stations, resulting in resource waste. At the same time, due to the existence of power curtailment penalties, the overall power generation efficiency of the CHMCS system decreases.

[0143] a03) such as Figure 4As shown in (b) and (d), the CHPMCS system also experiences situations where the system's power generation exceeds the system's transmission capacity limit on typical days during the high-water season, resulting in water, wind, and solar power curtailment. At this time, due to the presence of pumped storage units, CHPMCS can reduce the occurrence of curtailment by consuming excess electricity to pump water and store energy, thereby improving the utilization rate of water resources and the absorption of clean energy.

[0144] (a04) During the dry season, the reservoir has less water on typical days, and the output of wind power and photovoltaic power is also significantly different from that on typical days during the wet season, resulting in lower overall power generation of CHPMCS and CHMCS, which does not reach the limit of the system transmission capacity.

[0145] (a05) At this time, CHPMCS with pumped storage units exhibits better flexibility. On the one hand, CHPMCS can pump water and store energy during periods when the reservoir needs to store large amounts of water, thus redistributing water resources and ensuring that the reservoir has more hydropower available on typical days during the dry season compared to CHMCS. Figure 5 As shown in (c) and (d).

[0146] (a06) On the other hand, CHPMCS, due to the presence of pumped storage units, significantly enhances system flexibility. During periods of low load and low electricity prices, it can purchase electricity from the grid and pump it to upstream reservoirs for storage (e.g., Figure 4 (b) Improve the head efficiency of the system, which is a function that traditional cascade hydro-wind-solar hybrid systems do not have.

[0147] a07) as Figure 5 As shown in Figure (b), during peak load and high electricity price periods, pumped storage units generate electricity using stored power, regulate grid load through power substitution, optimize grid resource allocation, and achieve price arbitrage. While achieving "low storage and high generation", they also play a positive role in "peak shaving and valley filling".

[0148] The economic analysis of power generation for different systems is shown in Table 1:

[0149] Table 1. Economic Comparison of Different Options

[0150]

[0151] (b01) Although there are significant differences in the typical daily wind and solar power output and water resource availability between the wet and dry seasons, Scheme 1, which incorporates pumped storage units, has higher power generation efficiency while reducing system curtailment.

[0152] b02) The main reason is that in Scheme 1, CHMCS can use pumped storage units to consume the excess power of the system to pump water for energy storage when there is power curtailment in the system, thereby reducing power loss.

[0153] b03) On the other hand, during periods of low load and low electricity prices, pumped storage units can purchase electricity from the grid and pump water for energy storage, generating electricity during periods of high electricity prices, thereby increasing the system's power generation efficiency through "low-load storage and high-load generation".

[0154] (b04) In addition, the improvement in power generation efficiency during the dry season is more obvious than that during the wet season. The main reason is that there is more water during the wet season, and the reservoir is more likely to reach the flood limit level. However, the flood limit level is lower than the normal storage level. At this time, the output of pumped storage units is limited, and they can only pump and store energy to a very small extent to reduce power curtailment.

[0155] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] Example 2

[0157] This embodiment provides an electronic device, including: one or more processors; a memory; and one or more programs stored in the memory, the one or more programs including instructions for executing the CHPMCS optimization scheduling method based on deep reinforcement learning as described in Embodiment 1.

[0158] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0159] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0163] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A CHPMCS optimization scheduling method based on deep reinforcement learning, characterized in that, Includes the following steps: Obtain relevant data on the cascade hydro-storage-wind-solar hybrid system; Based on the aforementioned data, a multi-constraint short-term optimization model for CHPMCS is established. The expression for the multi-constraint short-term optimization model is as follows: The constraints for wind power stations are as follows: ， The constraints for photovoltaic power plants are: ， The constraints for the operation of a hydropower station are: ， The constraints for pumped storage power stations are: ， The constraints for cascade hydropower stations are: ， In the formula, To improve the daily power generation efficiency of the CHMCS system, The operating cycle is 24 hours. The total number of hydropower stations, This represents the total number of wind power stations. This represents the total number of photovoltaic power stations; For hydroelectric power station In the Electricity sales volume at that time and Wind power and photovoltaic In the Electricity sales volume at that time; For the system in the first The electricity price at that time; and Representing the first The amount of electricity purchased from the power grid and the corresponding price; For the system in the first The amount of electricity wasted at that time The price is the penalty for abandoning electricity; Indicates wind power station During the period Total power generation Indicates wind power station During the period Internet usage power consumption Indicates wind power station During the period The amount of electricity consumed by a pumped-storage hydroelectric power station; Indicates photovoltaic power station During the period Total power generation Indicates photovoltaic power station During the period Internet usage power consumption Indicates photovoltaic power station During the period The amount of electricity consumed by a pumped-storage hydroelectric power station; Indicates hydroelectric power station During the period Total power generation Indicates hydroelectric power station During the period Internet usage power consumption Indicates hydroelectric power station During the period The electricity generated is consumed by pumped-storage hydroelectric power stations. and They represent hydroelectric power stations. The minimum and maximum power generation; Indicates the pumped storage power station during the time period Electricity purchased from the power grid; and Indicates the time period of the unit Power generation and pumping capacity; Indicates the first Taiwan pumped storage units during the period Pumping power; For the first The relationship function between pumping power, head, and flow rate of a pumped storage unit; Indicates the first Taiwan pumped storage units during the period The power generation flow rate; Indicates the first Taiwan pumped storage units during the period Hydropower head; express The reservoir capacity at any given time express The reservoir capacity at any given time Indicates time period Inflow rate of the reservoir, Indicates time period The amount of water consumed by pumped storage power stations Indicates the time period The power generation flow of the reservoir, Indicates the time period The discharge flow of the reservoir; The multi-constraint short-term optimization model is converted into an unconstraint reinforcement learning task. The unconstrained reinforcement learning task is solved using a reinforcement learning algorithm to obtain the optimal output, thereby achieving optimal scheduling of CHPMCS.

2. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The relevant data includes operating cycle, total number of hydropower stations, total number of wind power stations, electricity sales, total number of photovoltaic power stations, electricity sales price, and abandoned electricity.

3. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The multi-constraint short-term optimization model also satisfies the following constraints: The constraints on transmission capacity are: ， The constraints for the power curtailment penalty are: ， The constraints on the reservoir capacity are: ， In the formula, Indicates system time period Maximum capacity transmission; This represents the upper bound of the power curtailment within the scheduling cycle; This serves as the upper bound of the total amount of power wasted during the entire scheduling cycle. Reservoir Minimum water storage capacity, Reservoir The maximum water storage capacity.

4. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The specific steps for converting to an unconstrained reinforcement learning task include: The state space and action space are selected based on the aforementioned multi-constraint short-term optimization operation model; The output increments of wind power, photovoltaic power, hydropower stations, and pumped storage power stations are selected as the action space; Set the objective function as the reward function for the unconstrained reinforcement learning task; Based on the reward function, according to the strategy π Define an action value function to realize the transformation from a multi-constrained short-term optimization model to an unconstrained reinforcement learning task.

5. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 4, characterized in that, The reservoir water level, power grid load, electricity price, wind speed, and solar irradiance are selected as the state space, and the expression for this state space is: In the formula, For state space, For the reservoir water level, For grid load, For electricity price, For wind speed, Light intensity.

6. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 4, characterized in that, The output increments of wind power, photovoltaic power, hydropower stations, and pumped storage power stations are selected as the action space, and the expression for this action space is: In the formula, For state space, To increase wind power output, To increase the output of photovoltaic power, To increase the power output of the hydropower station This is to increase the output of pumped storage power stations.

7. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 4, characterized in that, The expression for the reward function is: In the formula, The cumulative reward obtained by the agent after obtaining the corresponding scheduling plan based on the external state variables of the system; The discount factor represents the importance of future returns relative to current returns. =0 indicates that only current immediate returns are considered, without considering future long-term returns. A value of 1 indicates that long-term future returns are as important as immediate current returns; For state space; This is the action space.

8. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 4, characterized in that, The expression for the action value function is: In the formula, Let the action value function be... In accordance with the strategy Calculated expected cumulative return, The cumulative reward obtained by the agent after obtaining the corresponding scheduling plan based on the external state variables of the system. For state space, This is the state space.

9. The CHPMCS optimization scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The reinforcement learning algorithm is a dual-delay deep deterministic policy gradient algorithm.

10. An electronic device, characterized in that, include: One or more processors; Memory; and One or more programs stored in memory, the one or more programs including instructions for executing the CHPMCS optimization scheduling method based on deep reinforcement learning as described in any one of claims 1-9.

Images