Method and device for obtaining energy management optimization model of wind-solar electric heating system

Abstract

The invention discloses a method for obtaining an energy management optimization model of a wind-solar electric heating system. The method comprises the following steps: establishing a Markov chain of a real-time optimization scheduling model of the wind-solar electric heating system containing solid heat and energy storage under a reinforcement learning framework; and based on a Markov chain, a near-end strategy optimization algorithm is adopted to train the wind-light electric heating system real-time optimization scheduling model containing solid heat storage and energy storage, and an energy management optimization model of the wind-light electric heating system is obtained. According to the method, the problems of dimension disaster, low solving efficiency and the like caused by excessive decision variables are avoided, and offline training and online application can be realized by utilizing the real-time decision capability of the deep neural network. The method can be widely applied to the energy management and scheduling system of the wind and light electric heating system in different regions and different scenes, an integrated scheduling plan of the wind and light electric heating system can be quickly obtained based on the SOC of the wind and light electric heating load output and energy storage, wind and light curtailment is reduced, and economy, safety and feasibility of the scheduling plan are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of wind-solar-thermal-electricity systems, and particularly to a method and device for obtaining an energy management optimization model of a wind-solar-thermal-electricity system. Background Art

[0002] Traditional primary fossil energy represented by coal and oil is non-renewable and difficult to provide stable and economical energy security for the future sustainable development energy society. Therefore, improving energy utilization efficiency, developing new energy, and strengthening the comprehensive utilization of renewable energy are inevitable choices for energy development. Against this background, the integrated electric-thermal energy system emerges as the times require. The integrated electric-thermal energy system couples the power and thermal energy systems, enabling two-way conversion between electricity and heat. Through the complementary advantages of electric and thermal energy, the accommodation ratio of wind and light can be greatly improved, promoting the economic and safe operation of the integrated electric-thermal energy system.

[0003] In an integrated electric-thermal energy system with a high proportion of wind and light, how to flexibly control the proportion of wind and light for heating and power generation and how to achieve the coordinated optimization of wind and light power stations and the electric-thermal system have always been research hotspots in the academic community. To reduce the curtailment rate of wind and light new energy power stations and meet the heat load demand at the same time, installing solid heat storage energy storage in new energy power stations has become a new option. When the new energy output is at a peak and the power load cannot fully accommodate the new energy, the new energy can be converted into heat and stored in the heat storage energy storage. When the thermal load is at a peak, the heat can be released to meet the heat load demand. At the same time, an electric-thermal conversion device is installed in the new energy power station, enabling the new energy power station to supply heat and power to the system simultaneously, making the system power conversion more flexible.

[0004] However, due to the high randomness and uncertainty of wind and light, it becomes extremely complex to achieve the coordinated operation of the electric-thermal system. Currently, traditional mathematical algorithms and intelligent algorithms have poor timeliness and low solution efficiency, making it difficult to achieve real-time control of wind-solar-storage new energy power stations. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for obtaining an energy management optimization model of a wind-solar-thermal-electricity system, using the real-time decision-making ability of the proximal policy algorithm and applying it to the wind-solar-thermal-electricity system to achieve real-time optimal operation of the wind-solar-thermal-electricity system and promote the accommodation of a high proportion of wind and light.

[0006] To achieve the above purpose, the present invention provides a method for obtaining an energy management optimization model of a wind-solar-thermal-electricity system, including: establishing a Markov chain of a real-time optimal scheduling model of a wind-solar-thermal-electricity system with solid heat storage energy storage under a reinforcement learning framework; based on the Markov chain, training the real-time optimal scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage using the proximal policy optimization algorithm to obtain an energy management optimization model of the wind-solar-thermal-electricity system.

[0007] The present invention also provides a device for obtaining an energy management optimization model of a wind-solar-electric-thermal system, including: a building unit, configured to build a Markov chain of a real-time optimization scheduling model of a wind-solar-electric-thermal system with solid heat storage energy storage under a reinforcement learning framework; an obtaining unit, configured to train the real-time optimization scheduling model of the wind-solar-electric-thermal system with solid heat storage energy storage based on the Markov chain by using a proximal policy optimization algorithm to obtain an energy management optimization model of the wind-solar-electric-thermal system.

[0008] The present invention also provides an electronic device, including: a processor, the processor being coupled with a memory; the processor, configured to read and execute a computer program stored in the memory to implement the method as described above.

[0009] The present invention also provides a computer-readable storage medium, in which a program or an instruction is stored, and when the program or the instruction is executed by a processor, the method as described above is implemented.

[0010] The technical effects and advantages of the present invention:

[0011] The present invention builds a real-time optimal scheduling model of a wind-solar-electric-thermal system that fully considers system electric power balance constraints, system electric power balance constraints, electro-thermal coupling constraints, wind-solar output constraints, solid heat storage energy storage heat storage and heat release constraints, and SOC constraints, defines a Markov chain of the model under a reinforcement learning framework, determines corresponding state space, action space, reward function, etc., and based on the self-exploration learning mechanism of the proximal policy optimization algorithm, through interaction with the wind-solar-electric-thermal system environment, trains a scheduling network that can ensure high wind-solar accommodation and safe and economic operation of the system under random changes of wind-solar-electric-thermal load.

[0012] Compared with traditional mathematical solution algorithms and heuristic intelligent algorithms, this method avoids problems such as the curse of dimensionality and low solution efficiency caused by too many decision variables, and at the same time, using the real-time decision-making ability of a deep neural network, it can achieve offline training and online application. The present invention can be widely applied to energy management and scheduling systems of wind-solar-electric-thermal systems in different regions and different scenarios, can quickly obtain an integrated scheduling plan of the wind-solar-electric-thermal system based on the output of wind-solar-electric-thermal load and the SOC of energy storage, reduce wind and light curtailment, and ensure the economy, safety and feasibility of the scheduling plan.

[0013] Other features and advantages of the present invention will be described in the subsequent description, and, in part, will be obvious from the description, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained by the structures pointed out in the description and the drawings. Description of the Drawings

[0014] Figure 1Flow chart of the method for obtaining the energy management optimization model of the wind-solar-thermal-electricity system;

[0015] Figure 2 For the wind power output curve on a typical day;

[0016] Figure 3 For the photovoltaic output curve on a typical day;

[0017] Figure 4 For the electric load output curve on a typical day;

[0018] Figure 5 For the heat load curve on a typical day;

[0019] Figure 6 For the output of each device in the electric system on a typical day;

[0020] Figure 7 For the output of each device in the heat system on a typical day. Detailed implementation manner

[0021] Next, in combination with the attached drawings provided by the present invention, the technical solutions of the present invention will be clearly and completely described. Moreover, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0022] It should be noted that the structures, ratios, sizes, etc. shown in the drawings of this specification are only used to cooperate with the content disclosed in the specification for those who are familiar with this technology to understand and read, and are not used to limit the limited conditions under which the present invention can be implemented. Therefore, they do not have technical substance significance. Any modification of the structure, change of the proportional relationship or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope that can be covered by the technical content disclosed in the present invention. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope that the present invention can be implemented. The change or adjustment of their relative relationships, without substantial change in the technical content, should also be regarded as the scope that the present invention can be implemented.

[0023] The present invention provides a method for obtaining an energy management optimization model of a wind-solar-thermal-electricity system. The following is a detailed introduction to this method in combination with Figure 1 The steps are as follows:

[0024] 1. Establish a Markov chain of the real-time optimal scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage under the framework of reinforcement learning.

[0025] Specifically include: (1) Establish a real-time optimal scheduling model for a wind-solar-thermal power system with solid heat storage energy storage.

[0026] Establish a real-time optimal scheduling model for a wind-solar-thermal power system with solid heat storage energy storage with the goal of minimizing the curtailment of wind and solar power. The objective function of the optimal scheduling model is as follows:

[0027] min P wt,real +P pv,real -P wt,grid -P pv,grid -P wt,heat -P pv,heat +P bt,heat (1)

[0028] In the formula, P wt,real and P pv,,real are the output powers of wind power and photovoltaic power at each moment, as shown in Figure 2 and Figure 3 ; P wt,grid and P pv,,grid are the actual power supply powers of wind power and photovoltaic power to the electrical load at each moment; P wt,heat and P pv,,heat are the actual power supply powers of wind power and photovoltaic power to the heat load at each moment; P bt,heat is the output power of the solid heat storage energy storage. When P bt is greater than zero, it means that the solid heat storage energy storage releases heat. When P bt is less than zero, it means that the solid heat storage energy storage stores heat.

[0029] The constraint conditions of the real-time optimal scheduling model for the wind-solar-thermal power system with solid heat storage energy storage include system electrical power balance constraint, system electrical power balance constraint, electro-thermal coupling constraint, wind-solar output constraint, solid heat storage energy storage heat storage and release constraint, and solid heat storage energy storage SOC constraint;

[0030] Among them, the system electrical power balance constraint is as follows:

[0031] P wt,grid +P pv,grid =P load (2)

[0032] In the formula, P load is the electrical load of the wind-solar-storage power station, as shown in Figure 4 ;

[0033] The system thermal power balance constraint is as follows:

[0034] Q wt,heat +Q pv,heat +Q bt,heat =Q load (3)

[0035] Wherein, Q wt,heat and Q pv,heat are the heat production of wind power and photovoltaic power, Q bt,heat is the stored and released heat of solid heat storage energy storage, Q load is the heat load demand, as shown in Figure 5 shown.

[0036] The wind and light output constraints are as follows:[[]]

[0037]

[0038] Wherein, P wt,real and P pv,,real are the outputs of wind power and photovoltaic power at each moment, as shown in Figure 2 and Figure 3 shown; P wt，grid and P bv，，grid are the actual power supply powers of wind power and photovoltaic power to the electrical load at each moment; P wt，heat and P pv，，heat are the actual power supply powers of wind power and photovoltaic power to the heat load at each moment.

[0039] The electro-thermal coupling constraints are as follows:[[]]

[0040] Q wt，heat = η eh P wt，heat (5)

[0041] Q pv，heat = η eh P pv，heat (6)

[0042] Q bt，heat = η eh P bt，heat (7)

[0043] Wherein, η eh is the electro-thermal conversion efficiency, and its value is 0.7.

[0044] The stored and released heat constraints of solid heat storage energy storage are as follows:[[]]

[0045] -P bt，heat，max ≤ P bt，heat ≤ P bt，heat，max (8)

[0046] Wherein, P bt，heat，max is the maximum heat storage and heat release power of the solid heat storage energy storage device, and its value is 5MW.

[0047] The SOC constraint of solid heat storage energy storage is as follows:[[]]

[0048] SOC bt，heat = (1 - ρ)SOC bt，heat，last-ΔSOC bt，heat (9)

[0049]

[0050] SOC bt，heat，min ≤SOC bt，heat ≤SOC bt，heat，max (11)

[0051] Wherein, SOCb t，heat，max and SOC bt，heat，min are respectively the upper and lower limits of the state of charge (SOC) of the solid heat storage energy storage, and their values are 0.1 and 0.9 respectively. ρ is the self-discharge rate of the energy storage battery, and its value is 0.9; ΔSOC bt，heat is the change in the state of charge of the solid heat storage energy storage device, which depends on the heat storage state of the energy storage device and the heat storage and release power E bt，heat，max is the maximum capacity of the solid heat storage energy storage device; η c is the charging efficiency of the solid heat storage energy storage device, and its value is 0.95; η d is the discharge efficiency of the solid heat storage energy storage device, and its value is 0.95.

[0052] (2) Establish a Markov chain of the real-time optimal scheduling model of the wind-solar-thermal power system with solid heat storage energy storage under the reinforcement learning framework.

[0053] The Markov chain includes a four-tuple {S, A, γ, R}; S, A, γ, and R represent the state space, action space, discount factor, and reward function respectively.

[0054] The state space S is as follows:

[0055] S = {S wt,real , S pv.real , S soc,heat,last , S pload , S qload} (12)

[0056] Wherein, S wt,real and S pv,real respectively represent the real-time output sets of wind power and photovoltaic power, S soc,heat,last is the SOC of the solid heat storage energy storage at the previous moment, S pload is the real-time output set of the electrical load of the wind-solar-thermal power system, and S qload is the real-time output set of the thermal load of the wind-solar-thermal power system;

[0057] The action space A is as follows:

[0058] A = {A wt,grid , A pv,grid , Awt,heat , A pv,heat A bt,heat} (13)

[0059] Wherein, A wt,grid , A pv,grid , A wt,heat and A pv,heat represent the electrical output and heat conversion output of wind power and photovoltaic power connected to the grid, and A bt,heat is the real-time output of the thermal energy storage;

[0060] The reward function R is shown as follows:

[0061]

[0062] Wherein, γ is the discount factor, and its value is 0.99;

[0063] Among them, the immediate reward r t is shown as follows:

[0064] r t = -(P wt,real + P pv,real - P wt,grid - P pv,grid - P wt,heat - P pv,heat + P bt ) (15)

[0065] 2. Based on the Markov chain, use the proximal policy optimization algorithm to train the real-time optimization scheduling model of the wind-solar-thermal-electric system with solid thermal energy storage to obtain the energy management optimization model of the wind-solar-thermal-electric system.

[0066] Specifically, it includes: (1) Randomly initialize the current value network parameter θ Q and the policy network parameter θ π , determine the training period K = 1000000, the scheduling period T = 24, and the target network update frequency C = 100, and let k = 0;

[0067] (2) Initialize the initial state s t of the wind-solar-thermal-electric system from the state space, and let t = 0;

[0068] (3) Output the action a t based on the state s t in the current value network;

[0069] (4) Execute the action a t in the energy management system of the wind-solar-thermal-electric system, and obtain the next state s t+1 , the reward r t, let \(t = t + 1\), and go to step 3); where, the wind-solar-thermal system with solid thermal energy storage is a physical system, and the energy management system of the wind-solar-thermal system is a system that manages the energy of this physical system;

[0070] (5) Store the learned time-series samples \(\{A\) T , S T , R T , S T+1}\) in the sample experience pool as the data set for training the network;

[0071] (6) Randomly collect \(m\) time-series samples \(\{A\) T , S T , R T , S T+1}\) from the experience pool, and calculate the loss function \(L(\theta\) Q ) of the value network;

[0072] \(L(\theta\) Q ) = E(y\) t - Q(s\) t , a\) t |\theta\) Q )) 2 (16)

[0073] y\) t = r\) t + \gamma Q'(s\) t+1 , \pi'(s\) t+1 |\theta\) π′ )|\theta\) Q′ ) (17)

[0074] In the formula, \(Q(s\) t , a\) t |\theta\) Q ) is the Q value output by the current network at time \(t\); E represents the mean function; \(r\) t is the immediate reward at time \(t\) extracted from the experience pool; \(\pi'(s\) t+1 |\theta\) π′ ) is the action variable output by the target policy network when the input state variable is \(s\) π′ under the parameter \(\theta\); \(Q'(s\) t+1 , \pi'(s\) t+1 |\theta\) t+1 )|\theta\) π′ ) is the input Q value of the target network when the input state is \(s\) Q′ and the action variable is \(\pi'(s\) Q′ |\theta\) t+1 ) under the parameter \(\theta\) t+1 |\theta\) π′ ).

[0075] (7) Obtain the updated value network parameter \(\theta\) according to the following formulaQ′ :

[0076]

[0077] In the formula, θ Q is the current value network parameter, and μ Q is the learning rate of the current value network. is the gradient of the loss function L(θ Q ) with respect to the parameter θ Q ; π θ (a t |s t ) and π θold (a t |s t ) are two different old and new policies of the policy network respectively; A t is the advantage function; ε is the boundary value of the loss function, and its value is 0.1; E t is the mean function.

[0078] (8) Update the parameter θ π' of the policy network, that is:

[0079]

[0080] In the formula, θ π is the current policy network parameter, and μ π is the learning rate of the policy network, and its value is 0.00001; is the gradient of the Q-value Q(s, a|θ Q ) with respect to the action a; is the policy gradient.

[0081] (9) Judge whether k > K holds. If so, output the energy management optimization model of the wind-solar-thermal-electricity system; otherwise, let k = k + 1 and return to step (2).

[0082] To verify the adaptability and superiority of the method proposed in this paper, the following cases are set in this paper:

[0083] Control group: The particle swarm optimization algorithm is used to solve the above-mentioned optimization scheduling model of the wind-solar-thermal-electricity system. Implementation example: The real-time scheduling network of the wind-solar-thermal-electricity system is trained by the method proposed in this paper.

[0084] According to the proposed real-time scheduling network of the wind-solar-thermal-electricity system, the scheduling scheme of the thermal-electricity system can be obtained as Figure 6 and Figure 7 shown. At the same time, the results under the two different cases are shown in Table 1 below. It can be seen that the curtailment of wind and solar power and the calculation time of the method proposed in this paper are both less than those of the particle swarm optimization algorithm:

[0085] Table 1 Performance indicators under different schemes

[0086]

[0087] As can be seen from Table 1 above, the method proposed in this paper has very good performance and rapidity. The overall abandoned wind and light power is reduced by 43.55% compared with the particle swarm algorithm. The time required to give the scheduling plan is only 0.98 s, which is reduced by 96.95% compared with the conventional particle swarm algorithm. Moreover, the network has good applicability and can adapt to different inputs of wind, light, electricity, heat loads and energy storage.

[0088] The present invention provides a device for obtaining an energy management optimization model of a wind-solar-thermal-electricity system, including: a building unit, configured to build a Markov chain of a real-time optimization scheduling model of a wind-solar-thermal-electricity system with solid heat storage energy storage under a reinforcement learning framework; an obtaining unit, configured to train the real-time optimization scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage based on the Markov chain by using a proximal policy optimization algorithm to obtain an energy management optimization model of the wind-solar-thermal-electricity system.

[0089] Since the content protected by this device is similar to the content protected by the above method, no more introduction will be made here. For details, please refer to the discussion part of the above method.

[0090] The present invention also provides a device. The electronic device includes: at least one processor, at least one communication interface, at least one memory, and at least one communication bus; optionally, the communication interface may be an interface of a communication module, such as an interface of a GSM module; the processor may be a processor CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention. The memory may include a high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory. Among them, the memory stores a program, and the processor calls the program stored in the memory to execute the method provided in the above embodiments of the present application.

[0091] Corresponding to the above method of the present application, the present application also provides a computer storage medium. The computer storage medium stores a computer program, and the computer program is run by a processor to execute the method provided in the above embodiments of the present application.

[0092] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for obtaining an energy management optimization model of a wind-solar-thermal-electricity system, characterized in that, it includes: Establish a Markov chain of the real-time optimization scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage under the reinforcement learning framework; Based on the Markov chain, use the proximal policy optimization algorithm to train the real-time optimization scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage to obtain an energy management optimization model of the wind-solar-thermal-electricity system.

2. The method according to claim 1, characterized in that, the objective function of the real-time optimization scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage is as follows: minP wt,real +P pv,real -P wt,grid -P pv,grid -P wt,heat -P pv,heat +P bt,heat (1) Where, P wt,real and P pv,,real are the outputs of wind power and photovoltaic power at each moment; P wt,grid and P pv,,grid are the actual power supply powers of wind power and photovoltaic power to the electrical load at each moment; P wt,heat and P pv,,heat are the actual power supply powers of wind power and photovoltaic power to the heat load at each moment; P bt,heat is the output of the solid heat storage energy storage. When P bt is greater than zero, it means that the solid heat storage energy storage releases heat. When P bt is less than zero, it means that the solid heat storage energy storage stores heat.

3. The method according to claim 2, characterized in that, the constraint conditions of the real-time optimization scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage include: system electric power balance constraint, system heat power balance constraint, wind-solar output constraint, electro-thermal coupling constraint, solid heat storage energy storage heat storage and heat release constraint, solid heat storage energy storage SOC constraint.

4. The method according to claim 3, characterized in that, the formula of the system electric power balance constraint is as follows: P wt,grid +P pv,grid =P load (2) where P load is the electrical load of the wind-solar-storage power station.

5. The method according to claim 3, characterized in that, the formula of the system heat power balance constraint is as follows: Q wt,heat +Q pv,heat +Q bt,heat =Q load (3) Where, Q wt,heat and Q pv,heat are the heat production of wind power and photovoltaic, Q bt,heat is the stored and released heat of solid heat storage energy storage, and Q load is the heat load demand.

6. The method according to claim 3, characterized in that, the formula of the wind-solar output constraint is as follows: where, P wt,real and P pv,,real are the output powers of wind power and photovoltaic power at each moment; P wt,grid and P pv,,grid are the actual power supply powers of wind power and photovoltaic power to the electrical load at each moment; P wt,heat and P pv,,heat are the actual power supply powers of wind power and photovoltaic power to the heat load at each moment.

7. The method according to claim 3, characterized in that, the formula of the electro-thermal coupling constraint is as follows: Q wt,heat = η eh P wt,heat (5) Q pv,heat = η eh P pv,heat (6) Q bt,heat = η eh P bt,heat (7) where η eh is the electro-thermal conversion efficiency.

8. The method according to claim 3, characterized in that, the formula of the solid heat storage energy storage heat storage and heat release constraint is as follows: -P bt,heat,max ≤P bt,heat ≤P bt,heat,max (8) Where P bt,heat,max is the maximum heat storage and release power of the solid heat storage energy storage device.

9. The method according to claim 3, characterized in that, the formula of the solid heat storage energy storage SOC constraint is as follows: SOC bt,heat = (1 - ρ)SOC bt,heat,last - ΔSOC bt,heat (9) SOC bt,heat,min ≤ SOC bt,heat ≤ SOC bt,heat,max (11) Wherein, SOC bt,heat,max and SOC bt,heat,min are respectively the upper and lower limits of the State of Charge (SOC) of the solid heat storage energy storage, ρ is the self-discharge rate of the energy storage battery; ΔSOC bt,heat is the change in the charge of the solid heat storage energy storage device, which depends on the heat storage state of the energy storage device and the heat storage and release power E bt,heat,max is the maximum capacity of the solid heat storage energy storage device; η c is the charging efficiency of the solid heat storage energy storage device; η d is the discharging efficiency of the solid heat storage energy storage device.

10. The method according to claim 1, characterized in that, the Markov chain includes a four-tuple {S, A, γ, R}; wherein, S, A, γ, and R respectively represent the state space, action space, discount factor, and reward function.

11. The method according to claim 10, characterized in that, the state space S is as follows: S = {S wt,real , S pv.real , S soc,heat,last , S pload , S qload} (12) Wherein, S wt,real and S pv,real respectively represent the real-time output sets of wind power and photovoltaic power, S soc,heat,last is the SOC of the solid heat storage energy storage at the previous moment, S pload is the real-time output set of the electrical load of the wind-solar-thermal-electricity system, and S qload is the real-time output set of the thermal load of the wind-solar-thermal-electricity system.

12. The method according to claim 10, characterized in that, the action space A is as follows: A = {A wt,grid , A pv,grid , A wt,heat , A pv,heat A bt,heat} (13) Where, A wt,grid , A pv,grid , A wt,heat and A pv,heat represent the electrical output and heat conversion output of wind power and photovoltaic power connected to the grid, and A bt,heat is the real-time output of thermal energy storage.

13. The method according to claim 10, characterized in that, the reward function R is as follows: In the formula, γ is the discount factor.

14. The method according to claim 10, characterized in that, The immediate reward r t is as follows: r t = -(P wt,real + P pv,real - P wt,grid - P pv,grid - P wt,heat - P pv,heat + P bt ) (15).

15. The method according to claim 10, characterized in that, the method of training the real-time optimization scheduling model of the wind-solar-thermal-electricity system with solid heat storage energy storage based on the Markov chain using the proximal policy optimization algorithm to obtain an energy management optimization model of the wind-solar-thermal-electricity system includes: Randomly initialize the parameters θ of the current value network Q and the parameters θ of the policy network π , determine the training period K, the scheduling period T, and the target network update frequency C, and set the number of training times k = 0; Based on the Markov chain, determine the data set for training the network; Obtain the updated value network parameter θ according to the dataset for training the network Q′ and the updated policy network parameter θ π′ ; Judge whether the number of training times k > K holds. If so, obtain an energy management optimization model of the wind-solar-thermal-electricity system.

16. The method according to claim 15, characterized in that, determining the data set for training the network based on the Markov chain includes: Initialize the initial state s of the wind-solar-thermal-electric system from the state space t , and let t = 0; Output action a based on the initial state s in the current value network t Output action a t ; Execute action a in the energy management system of the wind-solar-thermal system t and obtain the next state s t+1 and reward r t ; Let \(t=t + 1\), repeat the above steps until the time series samples \(\{A T ,S T ,R T ,S T+1 \} are stored in the sample experience pool as the data set for training the network.

17. The method according to claim 16, characterized in that, Obtain the updated value network parameter θ according to the dataset for training the network Q′ and the updated policy network parameter θ π′ , including: Randomly collect m time series samples {A T , S T , R T , S T+1} from the dataset, and calculate the loss function L(θ Q ) of the value network; According to the loss function \(L(\theta\) Q ) of the value network, obtain the updated value network parameter \(\theta\) Q′ and the updated policy network parameter \(\theta\) π′ ; if k % C = 1, then update based on the update formula Among them, the loss function \(L(\theta Q ) is given by the following formula: L(θ Q ) = E(y t - Q(s t , a t |θ Q )) 2 (16) y t = r t + γQ′(s t+1 , π′(s t+1 |θ π′ )|θ Q′ ) (17) Where, Q(s t , a t | θ Q ) is the Q - value output by the current network at time t; E represents the mean function; r t is the immediate reward at time t extracted from the experience pool; π′(s t+1 | θ π′ ) is the action variable output by the target policy network when the input state variable is s π′ under the parameter θ t+1 ; Q′(s t+1 , π′(s t+1 | θ π′ )| θ Q′ ) is the input Q - value of the target network under the parameter θ Q′ when the input state is s t+1 and the action variable π′(s t+1 | θ π′ ); Among them, the formula for updating the value network parameter θ Q′ is as follows: where, θ Q is the current value network parameter, μ Q is the learning rate of the current value network, is the gradient of the loss function L(θ Q ) with respect to the parameter θ Q ; π θ (a t |s t ) and π θold (a t |s t ) are respectively the old and new policies of the policy network; A t is the advantage function; ε is the boundary value of the loss function; E t is the mean function. Among them, the formula for updating the parameters θ of the policy network π′ is as follows: where θ π is the current policy network parameter, and μ π is the learning rate of the policy network; is the gradient of the Q-value Q(s,a|θ Q ) with respect to the action a; is the policy gradient.

18. An apparatus for obtaining an energy management optimization model of a wind-solar-thermal-electricity system, Characterized in that, Including: A building unit, configured to build a Markov chain of a real-time optimal scheduling model of a wind-solar-thermal-electric system with solid heat storage energy storage under a reinforcement learning framework; An obtaining unit, configured to train the real-time optimal scheduling model of the wind-solar-thermal-electric system with solid heat storage energy storage by using a proximal policy optimization algorithm based on the Markov chain, so as to obtain an energy management optimization model of the wind-solar-thermal-electric system.

19. An electronic device, Including: A processor, the processor being coupled to a memory; The processor is configured to read and execute a computer program stored in the memory to implement any one of the methods of claims 1-17.

20. A computer-readable storage medium, in which a program or instruction is stored, and when the program or instruction is executed by a processor, any one of the methods of claims 1-17 is implemented.

Images