Power dispatching method, device and storage medium based on muzero and deep reinforcement learning

Abstract

The present invention discloses a power dispatching method, device and storage medium based on muzero and deep reinforcement learning. The method obtains real-time power data from the power system, inputs the real-time power data into the deep reinforcement learning model based on muzero, outputs the power of each distributed power resource, and the communication power between the power system and the upper power grid; the deep reinforcement learning model based on muzero is trained according to the power system environment model, and the power system environment model is used to feedback decision rewards to the deep reinforcement learning model based on muzero; the distributed power resources of the power system are dispatched according to the power of each distributed power resource, and the power system meets the communication power requirement. The technical solution of the present invention improves the dispatching efficiency of the power system for distributed power resources and reduces the waste of power resources.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power dispatching, and in particular to an electric power dispatching method, device and storage medium based on Muzero and deep reinforcement learning. Background Art

[0002] As a device that meets the needs of energy conservation and emission reduction and adapts to the consumption of renewable energy, the planning and operation of energy storage will introduce time coupling constraints. The modeling of energy storage state transfer and the solution of real-time output must also consider the power generation and charge state of adjacent time periods. Therefore, the use of energy storage as the main consumption carrier makes the new energy consumption business a complex multi-stage optimization decision-making problem. The accurate modeling and solution of the problem are difficult, and at the same time, it requires quick decision-making and the consumption of wind and solar power as much as possible to meet the real-time scheduling needs, which puts high demands on the model for solving the problem.

[0003] Traditional decision-making schemes, such as stochastic programming and robust optimization, have certain advantages in reducing system operating costs and considering feasible decisions in extremely harsh scenarios. However, when solving random multi-scenario problems caused by renewable energy and electrical loads, the above traditional methods have the disadvantages of large amount of calculation, low calculation accuracy, and overly conservative decision-making schemes. Therefore, in order to solve complex operation scheduling problems and meet the requirements for the absorption of renewable energy, researchers urgently need to introduce new decision-making methods.

[0004] With the rise and development of artificial intelligence technology, reinforcement learning (RL), as an important technology for solving sequential decision-making problems, has become an effective technology for solving sequential decision-making problems by interacting with the environment through trial and error, updating value judgments and strategy choices in continuous learning, especially the deep reinforcement learning model (DRL) that combines deep neural networks with reinforcement learning. It has better adaptive learning ability and optimal decision-making ability to solve non-convex nonlinear problems, which provides a new idea for dealing with complex power system operation and scheduling problems. However, many DRL methods based on a single neural network currently have shortcomings such as low accuracy in extracting system states and output strategies, and have high requirements for model performance. It is difficult to accurately grasp the system uncertainty state including source and load fluctuations, and it is difficult to accurately absorb new energy sources such as wind and solar. Summary of the invention

[0005] The present invention provides a power dispatching method, device and storage medium based on Muzero and deep reinforcement learning, which improves the dispatching efficiency of the power system for distributed power resources and reduces the waste of power resources.

[0006] An embodiment of the present invention provides a power dispatching method based on muzero and deep reinforcement learning, comprising the following steps:

[0007] Acquire real-time power data from the power system, input the real-time power data into the deep reinforcement learning model based on muzero, and output the power of each distributed power resource and the communication power between the power system and the upper power grid; the deep reinforcement learning model based on muzero is trained according to the power system environment model, and the power system environment model is used to feed back decision rewards to the deep reinforcement learning model based on muzero;

[0008] Distributed power resources of the power system are dispatched according to the power of each distributed power resource, so that the power system meets the connection power requirement.

[0009] Furthermore, the power system environmental model is established specifically as follows: the objective function of the power system environmental model is obtained according to the full-time operation cost of the power system, the interconnection benefit of the upper power grid and the carbon emission cost, and the power system environmental model is made to meet the constraints of safe operation of the power system.

[0010] Further, the muzero-based deep reinforcement learning model is trained according to the following steps:

[0011] Inputting historical power data of the power system into the deep reinforcement learning model based on muzero, the deep reinforcement learning model based on muzero determines the power of each distributed power resource and the connection power between the power system and the upper power grid;

[0012] The power system environment model calculates the target cost of the power system according to the power of each distributed power resource and the communication power between the power system and the upper power grid, and feeds the target cost back to the deep reinforcement learning model based on muzero as a decision reward;

[0013] The muzero-based deep reinforcement learning model judges the decision action according to the target cost, and adjusts the next decision action according to the target cost until the muzero-based deep reinforcement learning model converges.

[0014] Furthermore, the constraints for satisfying the safe operation of the power system are specifically: satisfying the power flow equation constraints, power balance constraints, voltage safety and stability constraints, and energy storage operation safety constraints of the power system.

[0015] Furthermore, the power of each distributed power resource is output, specifically including: wind power output power, photovoltaic output power, traditional unit output power, energy storage input power and energy storage output power.

[0016] Further, constructing the muzero-based deep reinforcement learning model according to the first neural network, the second neural network and the third neural network;

[0017] The first neural network is a characterization network, which is used to encode power data of the power system input into the deep reinforcement learning model;

[0018] The second neural network is a dynamic network, which is used to decide the first action according to the power system state at the current moment, and obtain the power system state at the next moment and obtain the immediate reward after taking the first action according to the first action;

[0019] The third neural network is a prediction network, which is used to obtain the control strategy and value function of the current state at the current moment.

[0020] Furthermore, the objective function of the power system environmental model is specifically:

[0021]

[0022] In the formula, F system, t k Indicates that at t k The operating cost of the power system at the moment, F trans, t k Indicates that at t k The connection efficiency between the power system and the upper power grid at the moment, F carbon, t k Indicates that at t k The carbon emission cost of the power system at the time, T represents the scheduling period.

[0023] Furthermore, the power data includes power characteristics of electric load, upper limit of wind power output, upper limit of photovoltaic output, output of traditional units, energy storage soc, time-of-use electricity price and time section.

[0024] Another embodiment of the present invention provides a power dispatching device based on muzero and deep reinforcement learning, including a power resource power decision module and a power resource dispatching module;

[0025] The electric power resource power decision module is used to obtain real-time power data from the power system, input the real-time power data into the deep reinforcement learning model based on muzero, and output the power of each distributed power resource and the communication power between the power system and the upper power grid; the deep reinforcement learning model based on muzero is trained according to the power system environment model, and the power system environment model is used to feed back decision rewards to the deep reinforcement learning model based on muzero;

[0026] The power resource scheduling module is used to schedule the distributed power resources of the power system according to the power of each distributed power resource, and make the power system meet the connection power requirement.

[0027] Another embodiment of the present invention provides a readable storage medium, which includes a stored computer program. When the computer program is executed, the device where the readable storage medium is located is controlled to execute the power scheduling method based on muzero and deep reinforcement learning described in any method embodiment of the present invention.

[0028] The embodiments of the present invention have the following beneficial effects:

[0029] The present invention provides a power dispatching method, device and storage medium based on muzero and deep reinforcement learning. The method establishes a deep reinforcement learning model based on muzero and a power system environment model, and uses the power system environment model to train the deep reinforcement learning model based on muzero to convergence, thereby improving the decision-making level and accuracy of the deep reinforcement learning model based on muzero, thereby improving the dispatching efficiency of the power system for distributed power resources, reducing the waste of power resources, and solving the problem of insufficient absorption of new energy in real-time dispatching business. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a power dispatching method based on muzero and deep reinforcement learning provided by an embodiment of the present invention;

[0031] Figure 2 It is a structural schematic diagram of a power dispatching device based on muzero and deep reinforcement learning provided by an embodiment of the present invention;

[0032] Figure 3 It is a structural schematic diagram of a power system of a power dispatching method based on muzero and deep reinforcement learning provided by an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram of the training process of a deep reinforcement learning model based on muzero provided by an embodiment of the present invention;

[0034] Figure 5 It is a structural diagram of a deep reinforcement learning model based on muzero provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] like Figure 1 As shown, an electric power dispatching method based on muzero and deep reinforcement learning provided by an embodiment of the present invention includes the following steps:

[0037] Step S101: Acquire real-time power data from the power system, input the real-time power data into the deep reinforcement learning model based on muzero, output the power of each distributed power resource, and the communication power between the power system and the upper power grid; the deep reinforcement learning model based on muzero is trained according to the power system environment model, and the power system environment model is used to feedback decision rewards to the deep reinforcement learning model based on muzero. Figure 3 As shown, the power system includes a wind power station, a photovoltaic power station, a traditional unit and an energy storage power station. Real-time power data is obtained from the power system, and the real-time power data includes power load power characteristics, wind power output upper limit, photovoltaic output upper limit, traditional unit output, energy storage soc, time-of-use electricity price and time section, and the power data reflects the status information of the power system. The output power of each distributed power resource specifically includes: wind power output power, photovoltaic output power, traditional unit output power, energy storage input power and energy storage output power.

[0038] When establishing the power system environmental model, the objective function of the power system environmental model is obtained according to the full-time operation cost of the power system, the upper power grid connection benefit and the carbon emission cost, and the power system environmental model is made to meet the constraints of safe operation of the power system, and the constraints of meeting the safe operation of the power system include meeting the power system's power flow equation constraints, power balance constraints, voltage safety and stability constraints and energy storage operation safety constraints. Specifically, the present invention considers the dynamic energy management of the power system, takes into account the time coupling of different scheduling times, and obtains the objective function of the power system environmental model according to the full-time operation cost of the power system, the upper power grid connection benefit and the carbon emission cost under the constraint of meeting the safe operation of the system:

[0039]

[0040] In the formula, F system， t k Indicates that at t k The operating cost of the power system at the moment, F trans， tk Indicates that at t k The connection benefit between the power system and the upper power grid at the moment, F carbon， t k Indicates that at t k The carbon emission cost of the power system at the time, T represents the scheduling period.

[0041] As one embodiment, the operating cost F of the power system is calculated according to the following formula: system :

[0042]

[0043] In the formula, a i , b i , c i is the consumption characteristic curve parameter of the traditional unit; δ is the cost coefficient of wind and solar power abandonment. In order to fully absorb new energy, the coefficient δ is much larger than the cost characteristic coefficient of other power sources; Ce(t k ) is t k The time-of-use electricity price coefficient of energy storage output at the moment, C(t k ) represents the loss cost coefficient, Δt is the dispatching time section interval (the time section refers to the action time of the power system for power dispatching, such as: 8 am, 8.15 am; the time section interval refers to the period between two action times, such as: 15 minutes); P mt，it The node i generator is at t k Active power at the moment, P pv，i，t is the node i at t k The active power of photovoltaic at the moment, P ess，i，t is the node i at t k The active power stored at all times, is line i at t k Line loss at any time.

[0044] As one embodiment, the connection benefit F between the power system and the upper power grid is calculated according to the following formula: trans :

[0045]

[0046] In the formula, Ce1(t k ) and Ce2(t k ) represent the real-time purchase and sale prices of electricity of the upper power grid, P trans， t k is the transmission power to the upper grid, when P trans， t k When the trend is positive, the power system purchases electricity from the upper-level power grid, and sells electricity when the trend is negative.

[0047] As one embodiment, the carbon emission cost F of the power system is calculated according to the following formula: carbon :

[0048]

[0049] In the formula, C carbon is the carbon emission cost coefficient.

[0050] The reward fed back by the power system environment model to the muzero-based deep reinforcement learning model is the negative value of the objective function:

[0051]

[0052] The muzero-based deep reinforcement learning model continuously learns the optimal strategy in a way that maximizes the cumulative reward over the entire period. Figure 5 As shown, during the training phase, the muzero-based deep reinforcement learning model continuously interacts with the power system environment model to obtain the power system state at each scheduling moment (i.e. Figure 5 state perception in) and generate decision actions (corresponding to Figure 5 The power system environment model calculates the reward value brought by the decision action of the deep reinforcement learning model based on muzero (corresponding to Figure 5 The feedback reward in the algorithm is fed back to the deep reinforcement learning model based on muzero, which uses the gradient descent method to update the model parameters to maximize the round reward. In the application stage, for each real-time scheduling task, the deep reinforcement learning model based on muzero only needs to obtain the power system status information (i.e., power data) at the current moment and directly output the optimal scheduling decision.

[0053] As one of the embodiments, the constraint conditions for satisfying the safe operation of the power system are specifically: satisfying the power system's power flow equation constraints, power balance constraints, voltage safety and stability constraints, and energy storage operation safety constraints.

[0054] As one embodiment, the power flow constraint of the power system is specifically:

[0055]

[0056]

[0057] Where P mt，i，t and Q mt，i，t are the generators at node i at t k Active power and reactive power at the moment; P load，i，t and Q load，i，t are node i at tk Active load and reactive load at the moment; P wt，i，t , P pv，i，t and P ess，i，t are node i at t k Active power of wind power, photovoltaic power and energy storage at the moment; U i，t is the voltage modulus of node i; θ ij，t is the phase angle difference between the two nodes; G ij and B ij are the conductance and susceptance between nodes i and j, respectively.

[0058] As one embodiment, the power balance constraint is specifically:

[0059]

[0060] In the formula, is line i at t k Line loss at the time; For k The transmission power (i.e. communication power) between the upper power grid and the upper power grid at any time.

[0061] As one embodiment, the voltage safety and stability constraint is specifically:

[0062] v i，mi n<v i <v i，max

[0063] In the formula, v i，max , v i，min They represent the upper and lower limits of the safe and stable voltage at node i, which are generally set to 0.95v N With 1.05v N , v N is the rated voltage.

[0064] As one embodiment, the energy storage operation safety constraint is specifically:

[0065]

[0066] Where: E ess，i is the energy storage capacity at node i, S ess，i，max , P ess，i，max , Qe ss，i，max are the upper limit of energy storage capacity, active power and reactive power at node i, respectively. ess，i，max and Soc ess，i，min are the upper and lower limits of the energy storage charge state, η c and η d They are the charging efficiency and discharging efficiency of energy storage respectively.

[0067] As one example, Figure 4 As shown, the muzero-based deep reinforcement learning model is constructed according to the first neural network, the second neural network and the third neural network;

[0068] The first neural network is a representation network (corresponding to Figure 4 A representation network in (a), used to encode power data of the power system input into the deep reinforcement learning model;

[0069] The second neural network is a dynamic network, which is used to decide the first action according to the power system state at the current moment, and obtain the power system state at the next moment and obtain the immediate reward after taking the first action according to the first action;

[0070] The third neural network is a prediction network, which is used to generate control strategies and values ​​according to the current state at the current moment.

[0071] Preferably, the characterization network h θ , Dynamic Network α and prediction network f ω The expression is:

[0072]

[0073]

[0074]

[0075] In the formula, θ represents the weight of the representation network; α represents the weight of the dynamic network; ω represents the weight of the prediction network, (s1, ...s t ) represents the historical and real-time power data of the power system, where s1 is the historical power data, s t is the real-time power data. The superscript k represents the variable of the kth training round. represents an internal state, the internal state is encoded according to historical and real-time power data, can be adapted to the input of a neural network, and the internal state is obtained through the representation network. P represents a control strategy, v represents a value, and r represents an immediate reward.

[0076] When the muzero-based deep reinforcement learning model makes a decision, the prediction network creates a root node according to the internal state encoded by the representation network, and constructs a search tree by iteratively performing N simulations. Finally, the optimal output power of the action that needs to be decided can be determined according to the access frequency of the child nodes of the root node, and the state transfer is realized by using the dynamic network in the next time step, and the above optimization process is repeated. In addition, the dynamic network can directly calculate the next internal state according to the current state and the action performed.

[0077] The deep reinforcement learning model based on muzero is a model that combines the Monte Carlo Tree Search (MCTS) planning method of muzero with the self-learning neural network model. It improves the performance of MCTS by introducing a learning model composed of three groups of neural networks, and continuously explores feasible solutions and five-tuples in the training process when making decisions.<S,A,P,r,γ> The loss function is calculated to update the model parameters in reverse order to ultimately find the optimal solution (i.e., the optimal decision action).

[0078] As one embodiment, the state space of the deep reinforcement learning model based on muzero is:

[0079] S={P load,|load| ,P pv,|pv|,max ,P wt,|wt|,max ,P mt,|mt| ,Soc ess,|ess| ,a,t} (14)

[0080] Where P load,|load| , P pv,|pv|,max , P wt,|wt|,max , P mt,|mt| ,、Soc ess,|ess| , a and t are respectively the power characteristics of electric load, the upper limit of wind power output, the upper limit of photovoltaic output, the output of traditional units, energy storage soc, time-of-use electricity price, and time section.

[0081] The action space of the deep reinforcement learning model based on muzero is:

[0082] A={a mt,|mt| ,a ess,|ess| ,a pv,|pv| ,a wt,|wt|} (15)

[0083] In the formula, a mt,|mt| 、a ess,|ess| 、a pv,|pv| and a wt,|wt| They represent the real-time traditional unit output increment, energy storage output increment, wind power output and photovoltaic output output of the model, respectively. mt,|mt| 、a ess,|ess| 、apv,|pv| and a wt,|wt| The value range is [-1, 1]. When setting up the deep reinforcement learning model, the neural network output is specified to be a value of [-1, 1], and then the value output by the neural network is converted into actual output according to the characteristics of each power resource.

[0084] According to the different ramp constraints of each power source, the model innovatively proposes to determine the output change or real-time output of different power sources. For traditional units and energy storage SOCs, time coupling constraints need to be considered, and the output change is determined as follows:

[0085]

[0086] In the formula, k represents the traditional unit or energy storage soc, U k,|k|,tk and D k,|k|,tk Respectively at t k At any given moment, the traditional unit or energy storage SOC is allowed to climb to the upper and lower limits.

[0087] Since the real-time output of renewable energy needs to be strictly limited to the upper limit of ultra-short-term accurate prediction output, the deep reinforcement learning model based on muzero is designed to decide the output of new energy sources as described in the sandwich function specification. The real-time output of renewable energy is:

[0088] P l,|l|,t =a l,|l|,t P l,|l|,max (17)

[0089] In the formula, l represents renewable energy, such as wind power or photovoltaic power, P l,|l|,t Indicates the real-time output of renewable energy.

[0090] Considering the upper and lower limits of the output of each power resource, the constraint condition of the action amplitude of the deep reinforcement learning model based on muzero is:

[0091] P *,|*|min ≤P *,|*|,t ≤P *,|*|,max (18)

[0092] Where P *,|*|,max and P *,|*|,min Indicates the upper and lower limits of active power for each decision object.

[0093] As one embodiment, the muzero-based deep reinforcement learning model is trained according to the following steps:

[0094] Step S1011: inputting the historical power data of the power system into the deep reinforcement learning model based on muzero, and the deep reinforcement learning model based on muzero determines the power of each distributed power resource and the connection power between the power system and the upper power grid;

[0095] Step S1012: the power system environment model calculates the target cost of the power system according to the power of each distributed power resource and the connection power between the power system and the upper power grid, and feeds the target cost back to the muzero-based deep reinforcement learning model as a decision reward;

[0096] Step S1013: The muzero-based deep reinforcement learning model judges the decision action according to the target cost, and adjusts the next decision action according to the target cost until the muzero-based deep reinforcement learning model converges.

[0097] Step S102: Dispatching the distributed power resources of the power system according to the power of each distributed power resource, and making the power system meet the connection power requirement. Specifically, dispatching is performed according to the output wind power output power, photovoltaic output power, traditional unit output power and energy storage input and output power, and making the power system meet the connection power requirement.

[0098] The present invention aims at the problem of insufficient absorption of new energy in real-time dispatching business. By combining deep learning with the construction of search trees, the present invention can ensure the full absorption of high-penetration new energy under the constraints of system safety operation and power flow balance, and absorb wind and solar power through energy storage, communication with the upper power grid and other means, while ensuring the stable operation of the system. The present invention trains a deep reinforcement learning model based on muzero offline. The model explores the output of traditional units, energy storage and communication with the upper power grid at each dispatching moment through continuous "trial and error" learning, meets the demand for new energy absorption at each moment, and can make the optimal decision for each dispatching period as time goes forward according to the current state of the environment. The present invention realizes the absorption of new energy in a way with the greatest economic benefit, improves the coupling and complementary capabilities with the upper power grid and the new energy absorption capacity of the power system.

[0099] Based on the above-mentioned embodiments of the invention, the present invention provides corresponding device embodiments, such as Figure 2 As shown;

[0100] Another embodiment of the present invention provides a power dispatching device based on muzero and deep reinforcement learning, including a power resource power decision module and a power resource dispatching module;

[0101] The electric power resource power decision module is used to obtain real-time power data from the power system, input the real-time power data into the deep reinforcement learning model based on muzero, and output the power of each distributed power resource and the communication power between the power system and the upper power grid; the deep reinforcement learning model based on muzero is trained according to the power system environment model, and the power system environment model is used to feed back decision rewards to the deep reinforcement learning model based on muzero;

[0102] The power resource scheduling module is used to schedule the distributed power resources of the power system according to the power of each distributed power resource, and make the power system meet the connection power requirement.

[0103] For the convenience and brevity of description, the device item embodiment of the present invention includes all the implementation methods in the above-mentioned power dispatching method embodiment based on muzero and deep reinforcement learning, which will not be repeated here.

[0104] Based on the above-mentioned invention items, the present invention provides a corresponding readable storage medium item embodiment; another embodiment of the present invention provides a readable storage medium, the readable storage medium includes a stored computer program, and when the computer program is executed, the device where the readable storage medium is located is controlled to execute the power dispatching method based on muzero and deep reinforcement learning as described in any method item embodiment of the present invention.

[0105] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0106] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0107] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0108] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0109] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium (i.e., the above-mentioned readable storage medium). Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0110] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0111] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

[0112] Those skilled in the art can understand that all or part of the processes in the above embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

Claims

1. A power dispatching method based on muzero and deep reinforcement learning, characterized in that: The following steps are involved: Acquire real-time power data from the power system, input the real-time power data into the deep reinforcement learning model based on muzero, and output the power of each distributed power resource and the communication power between the power system and the upper power grid; the deep reinforcement learning model based on muzero is trained according to the power system environment model, and the power system environment model is used to feed back decision rewards to the deep reinforcement learning model based on muzero; Dispatching each distributed power resource of the power system according to the power of each distributed power resource, and making the power system meet the connection power requirement; The deep reinforcement learning model based on muzero is trained according to the following steps: Inputting historical power data of the power system into the deep reinforcement learning model based on muzero, the deep reinforcement learning model based on muzero determines the power of each distributed power resource and the connection power between the power system and the upper power grid; The power system environment model calculates the target cost of the power system according to the power of each distributed power resource and the communication power between the power system and the upper power grid, and feeds the target cost back to the deep reinforcement learning model based on muzero as a decision reward; The muzero-based deep reinforcement learning model judges the decision action according to the target cost, and adjusts the next decision action according to the target cost until the muzero-based deep reinforcement learning model converges.

2. The power dispatching method based on muzero and deep reinforcement learning according to claim 1, characterized in that: The specific steps of establishing the power system environmental model are as follows: obtaining the objective function of the power system environmental model according to the full-time operation cost of the power system, the interconnection benefit of the upper power grid and the carbon emission cost, and making the power system environmental model meet the constraints of safe operation of the power system.

3. The power dispatching method based on muzero and deep reinforcement learning according to claim 2, characterized in that: The constraints for satisfying the safe operation of the power system specifically include: satisfying the power system's power flow equation constraints, power balance constraints, voltage safety and stability constraints, and energy storage operation safety constraints.

4. The power dispatching method based on muzero and deep reinforcement learning according to claim 3 is characterized in that: Output the power of each distributed power resource, specifically including: wind power output power, photovoltaic output power, traditional unit output power, energy storage input power and energy storage output power.

5. The power dispatching method based on muzero and deep reinforcement learning according to claim 4, characterized in that: Constructing the muzero-based deep reinforcement learning model according to the first neural network, the second neural network and the third neural network; The first neural network is a characterization network, which is used to encode power data of the power system input into the deep reinforcement learning model; The second neural network is a dynamic network, which is used to decide the first action according to the power system state at the current moment, and obtain the power system state at the next moment and obtain the immediate reward after taking the first action according to the first action; The third neural network is a prediction network, which is used to generate control strategies and values ​​according to the current state at the current moment.

6. The power dispatching method based on muzero and deep reinforcement learning according to claim 5, characterized in that: The objective function of the power system environmental model is specifically: In the formula, F system ,t k Indicates that at t k The operating cost of the power system at the moment, F trans ,t k Indicates that at t k The connection benefit between the power system and the upper power grid at the moment, F carbon ,t k Indicates that at t k The carbon emission cost of the power system at the time, T represents the scheduling period.

7. The power dispatching method based on muzero and deep reinforcement learning according to any one of claims 1 to 6, characterized in that: The power data includes power characteristics of electric load, upper limit of wind power output, upper limit of photovoltaic output, output of traditional units, energy storage soc, time-of-use electricity price and time section.

8. A power dispatching device based on muzero and deep reinforcement learning, characterized in that: It includes an electric power resource power decision module and an electric power resource scheduling module; The electric power resource power decision module is used to obtain real-time power data from the power system, input the real-time power data into the deep reinforcement learning model based on muzero, and output the power of each distributed power resource and the communication power between the power system and the upper power grid; the deep reinforcement learning model based on muzero is trained according to the power system environment model, and the power system environment model is used to feed back decision rewards to the deep reinforcement learning model based on muzero; The power resource scheduling module is used to schedule the distributed power resources of the power system according to the power of each distributed power resource, and make the power system meet the connection power requirement; The process of training the muzero-based deep reinforcement learning model includes: Inputting historical power data of the power system into the deep reinforcement learning model based on muzero, the deep reinforcement learning model based on muzero determines the power of each distributed power resource and the connection power between the power system and the upper power grid; The power system environment model calculates the target cost of the power system according to the power of each distributed power resource and the communication power between the power system and the upper power grid, and feeds the target cost back to the deep reinforcement learning model based on muzero as a decision reward; The muzero-based deep reinforcement learning model judges the decision action according to the target cost, and adjusts the next decision action according to the target cost until the muzero-based deep reinforcement learning model converges.

9. A readable storage medium, characterized in that: The readable storage medium includes a stored computer program. When the computer program is executed, the device where the readable storage medium is located is controlled to execute the power dispatching method based on muzero and deep reinforcement learning as described in any one of claims 1 to 7.

Images