Comprehensive energy system economic dispatching model method based on deep reinforcement learning

Abstract

The invention discloses an integrated energy system economic dispatching model method based on deep reinforcement learning, and the method comprises the steps: constructing a park equipment environment model, and carrying out the mathematical description of a park integrated energy system low-carbon economic dispatching problem through taking economy and carbon emission as targets; secondly, the scheduling problem is expressed as a Markov decision process framework, and a state space, an action space and a reward mechanism of the system are reasonably designed; then, a near-end strategy optimization algorithm is adopted to carry out low-carbon economic dispatching decision making, and an action noise dynamic updating strategy is provided to gradually improve the exploration capability and learning capability of a network model; according to the method, the action space, the state space, the reward mechanism, the neural network structure and the like of the intelligent agent are designed in detail, the minimization of the system operation cost is considered on the basis, and a deep reinforcement learning method is adopted for scheduling simulation. The method provided by the invention can perform optimal scheduling according to the load demand of the system, and has good scheduling performance and self-learning ability.

Description

Technical Field

[0001] The present invention relates to the field of energy technology, and more specifically to an economic dispatch model method for an integrated energy system based on deep reinforcement learning. Background Art

[0002] In traditional energy supply systems, the separate operation of subsystems such as the power system, natural gas system, and thermal system results in low energy utilization. The regional integrated energy system (RIES) uses advanced communication and control technologies to realize the interconnection of power generation equipment, energy conversion equipment, and energy storage systems to achieve the interconnection of electricity, heat, and gas networks, which can effectively improve energy utilization. Promoting the development of RIES has become an important way to achieve the "dual carbon" goal. However, the large-scale integration of renewable resources has increased the complexity of energy system scheduling problems. In regional integrated energy systems, the instability of renewable energy and the coupling of multiple energy sources make the physical model of the energy system too complex. At the same time, the access of intelligent information equipment makes the system contain a large amount of high-dimensional data. At present, the research on RIES optimization scheduling and operation mainly focuses on two aspects: model construction theory and optimization solution method. In the absence of a clear model and involving a huge amount of data, the solution methods for the optimal scheduling of regional integrated energy systems mainly include analytical methods and heuristic algorithms. At present, model predictive control (MPC) is used to adjust the output of each unit of the system, optimize the operation of system equipment, introduce particle concentration evaluation operators to improve the particle swarm algorithm to construct the IES multi-objective optimization model, etc., but the computational cost increases, and they are mostly limited to fixed scheduling plans or precise physical models, and it is difficult to respond dynamically to random changes. It will be difficult to apply to the increasingly complex systems.

[0003] In recent years, deep reinforcement learning (DRL) has been widely used and developed by researchers as an effective means to deal with sequential decision-making problems. It combines the advantages of DL and RL and has strong adaptability to dynamic and uncertain environments. DRL learns strategies by interacting with the environment to find the optimal action to achieve the maximum reward, which is consistent with the goal of RIES optimization scheduling problem, that is, adjusting the output of equipment to achieve the minimum cost. At present, there are many studies on the optimization scheduling of integrated energy systems based on reinforcement learning. End-to-end learning is achieved by directly interacting with the environment. The learned intelligent agent can make scheduling decisions immediately based on the environmental information of the period, and has efficient problem-solving capabilities for data and complex models. Many researchers have tried to apply DRL to this field. The literature uses deep Q-learning (deepQ-network, DQN) to solve the energy system scheduling problem. However, DQN is a value function method based on discretized action space, and the algorithm optimization accuracy is low. It is not suitable for continuous action scenes with strong randomness. The paper uses Deep Deterministic Policy Gradient (DDPG) to solve the problem of dynamic economic dispatch of integrated energy systems. The algorithm approximates the state value function of reinforcement learning through deep neural networks, so that continuous actions and state spaces in the dispatch model can be directly processed. However, this method has the problems of overly conservative agent exploration and low exploration efficiency, which directly affects the speed and accuracy of agent training. Based on the Soft Actor-Critic algorithm, the paper proposes an IES multi-objective optimization dispatch model that considers economy and power supply reliability. Due to the existence of action noise, there will be problems such as insufficient exploration efficiency to a certain extent. It is urgent to study how to use the powerful learning ability of DRL to effectively solve the optimization operation of RIES. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides an economic dispatch model method for an integrated energy system based on deep reinforcement learning.

[0005] The technical solution of the present invention is as follows:

[0006] The economic dispatch model method of the integrated energy system based on deep reinforcement learning includes:

[0007] Firstly, the park equipment environment model is constructed, and the low-carbon economic dispatch problem of the park's integrated energy system is mathematically described with economy and carbon emissions as the goals.

[0008] Secondly, the scheduling problem is expressed as a Markov decision process framework, and the state space, action space and reward mechanism of the system are reasonably designed;

[0009] Then, the proximal strategy optimization algorithm is used to make low-carbon economic dispatch decisions, and the action noise dynamic update strategy is proposed to gradually improve the exploration and learning capabilities of the network model.

[0010] The environmental model includes the flow of electrical energy, thermal energy, cold energy and the coupling links between energy sources in the integrated energy system. The environmental model is built around various energy equipment in the system and the balance conditions of supply and demand of the system energy load.

[0011] The environmental model is an integrated energy system environment that interacts with the intelligent agent. It calculates the output status of the energy equipment under the action strategy given by the deep reinforcement learning intelligent agent and feeds back the reward function. The environmental model includes the electric energy flow, thermal energy flow, cold energy flow and coupling links between energy sources in the integrated energy system, and constructs the environmental model around various energy equipment in the system and the supply and demand balance conditions of the system energy load.

[0012] The integrated energy system mainly includes three heterogeneous energy forms: electricity, heat, and gas, and meets the diversified energy needs within the system through a variety of energy conversion equipment. According to the different functions of the equipment during the operation of the system, the integrated energy system includes three types of renewable energy equipment, energy conversion equipment, and energy storage equipment. The overall framework of the integrated energy system is as follows Figure 1 As shown in the figure, the renewable energy equipment of the system includes photovoltaic (PV) and wind turbine (WT); the energy conversion equipment includes electric heat boiler (EHB), gas boiler (GB), gas turbine (GT), waste heat boiler, and combined heat and power (CHP); the energy storage equipment mainly considers battery energy storage (BES). The load side mainly includes electric load, gas load and thermal load. The goal of economic dispatch is to minimize the total economic cost while maintaining the normal operation of the system. Among them, renewable energy power generation and energy storage devices are connected to the grid bus in the form of nodes, realizing the coordinated work of multi-source power generation to supply the load power demand.

[0013] Model building of energy conversion equipment includes:

[0014] (1) Wind turbine device model: The power of a wind turbine unit is completely controlled by the wind speed. The power of wind turbine generation is expressed as follows:

[0015]

[0016] In the above formula, v is the current operating wind speed of the fan, v i is the set cut-in wind speed, vo is the set cut-out wind speed, v r is the set rated wind speed. It can be seen from the above formula that the fan cannot work normally when it is below the cut-in wind speed or above the cut-out wind speed, because it cannot drive the fan to run or is easy to damage the fan; when the wind speed is below the rated wind speed and above the cut-in wind speed, the fan is in a low-power operation state; when the wind speed is above the rated wind speed and below the cut-out wind speed, the fan runs at rated power and is also running near the optimal tip speed ratio;

[0017] (2) Photovoltaic device model: The power output of the photovoltaic power generation unit mainly depends on the intensity of the light source and the system parameters of the device itself. The expression of the photovoltaic generator power is generated as follows:

[0018] P PV =λ P,t cosθκ m A pv η pv (2)

[0019] In the formula, λ P,t represents the light intensity at time t, k m ,A pv ,η pv They represent the efficiency of the controller, the area of ​​the photovoltaic panel, and the rated power generation efficiency of the photovoltaic panel respectively;

[0020] (3) Electric boiler model:

[0021] Electric Boiler (EB) is an electric-thermal coupling device in the integrated energy system, which uses electric heating components to convert electrical energy into thermal energy. The operation of electric boilers is less affected by the ambient temperature and can still operate normally in extremely cold weather, so they are widely used in heating. The process of converting electricity into thermal energy is as follows:

[0022]

[0023] In the formula, represents the thermal power of the electric boiler at time t, represents the power consumption at time t, η EHB and κ l They represent the electrical-to-heat conversion efficiency and the thermal utilization efficiency respectively.

[0024] (4) Gas boiler model:

[0025] Gas Boiler (GB) is a device that uses natural gas as energy input and heats low-temperature water through high-temperature flue gas to achieve gas-heat conversion.

[0026]

[0027] in, are the heating power of the gas boiler during period t and the energy consumed by natural gas respectively; For the operating efficiency of gas boilers.

[0028] (5) Gas turbine model:

[0029] A gas turbine is a device that converts thermal energy into mechanical energy. The compressor inhales air from the outside atmosphere and compresses it step by step through the axial compressor to increase the pressure, and the air temperature is also increased accordingly; the compressed air is sent to the combustion chamber to mix with the injected fuel and burn to generate high-temperature and high-pressure gas; then it enters the turbine to expand and do work, driving the turbine to drive the compressor and the external load rotor to rotate at high speed, realizing the partial conversion of the chemical energy of the gas or liquid fuel into mechanical work and outputting electrical energy.

[0030]

[0031]

[0032] Where: is the heat value input to the gas turbine, α i , β i , γ i is a parameter determined by the gas turbine heat rate curve; is the power generation capacity of the gas turbine; is the equivalent gas load, H g is the calorific value of natural gas.

[0033] (6) Cogeneration unit model

[0034] The combined heat and power unit consists of a gas turbine and a waste heat recovery boiler. The gas turbine generates electricity through the consumption of natural gas, and also produces flue gas with thermal energy, outputting thermal power; and the waste heat boiler can also generate thermal power through heat exchange. The power generation capacity of the combined heat and power unit is as follows:

[0035]

[0036] In the formula, represents the natural gas consumption at time t, H g Indicates the calorific value of gas, α GT represents the power generation efficiency of the gas turbine, δ GT Represents the gas-to-electricity conversion efficiency of the gas turbine.

[0037] In addition, the heat generation capacity of the combined heat and power plant is as follows:

[0038]

[0039] In the formula, α SG Indicates the heat exchange efficiency of the waste heat boiler.

[0040] In addition, the waste heat recovery boiler collects the heat from the flue gas discharged by the gas turbine and supplies it to the heat network. Its output thermal power is

[0041]

[0042] Where: is the output thermal power of the waste heat recovery boiler during period t; is the thermal power output of the gas turbine during period t; SG is the heat exchange efficiency of the waste heat boiler.

[0043] (7) Energy storage device model

[0044] When the energy storage device is connected to the energy grid, energy exchange will be carried out as the system operates. In this invention, the electric energy storage device is mainly considered. The remaining energy SOC(t) of the energy storage device at time t can be expressed as follows:

[0045]

[0046] In the formula, The charging and discharging power of the energy storage device, λ B represents the exchange efficiency, Represents the charge and discharge status at time t, and takes the value of {0,1} integer variable.

[0047] Also included are build system constraints, including:

[0048] In the energy system, it is necessary to maintain a balance between energy supply and demand. At every moment, the power on both sides of the grid supply and demand must be kept in a balanced state. The constraints of the low-carbon economic dispatch system of the integrated energy system include power balance constraints, unit output constraints, unit ramp constraints, energy storage state constraints, and upper-level grid and gas grid interaction constraints.

[0049] (4) Electric power balance constraints

[0050]

[0051] In the system, the power generated by distributed power generation units such as photovoltaics, wind power, and cogeneration and the energy exchange power of energy storage devices must be balanced with the total electrical load demand.

[0052] (5) Thermal power balance constraints

[0053]

[0054] In the formula, E EHB is the heating power of the electric boiler, E GBis the heating power of the gas boiler, E GHP is the thermal power of the combined heat and power generation unit, E SG is the heat output power of the waste heat boiler, E Load is the system heat load requirement.

[0055] (6) Power Constraint

[0056] Each device in the integrated energy system has an upper and lower operating power limit, including wind turbines, photovoltaic generators, cogeneration units and electrical energy storage devices. For the output of each device, there are the following constraints:

[0057]

[0058] In the formula, the power constraints of wind turbines, photovoltaic generators, and cogeneration units are listed respectively, and the capacity constraint of the remaining energy of the energy storage device is given.

[0059] (4) Climbing constraints

[0060] The output power of some equipment in the integrated energy system cannot change suddenly, so there is a ramp limit on the output power of the cogeneration unit.

[0061] R CHP,min ≤P CHP,t -P CHP,t-1 ≤R CHP,max (16)

[0062] Similarly, gas turbine and gas boiler units also have ramp restrictions, as follows:

[0063] R GT,min ≤P GT,t -P GT,t-1 ≤R GT,max (17)

[0064] R EB,min ≤P EB,t -P EB,t-1 ≤R EB,max (18)

[0065] It also includes the construction of a low-carbon economic dispatch objective function for an integrated energy system:

[0066] The core goal of the economic dispatch problem is to minimize the total economic cost while maintaining the normal operation of the system. In this invention, in order to simplify the calculation, the cost function is mainly set to consider the following: the power generation cost of the wind power generation unit, the cost of the photovoltaic power generation unit, the power generation cost of natural gas power generation, the cost of interacting with the power grid, and the loss cost of battery charging and discharging, which are set as C WG (t), C PV (t), C Grid(t), C SOC (t), C Gas (t). The dispatching goal of the present invention is to optimize the sum of these five costs and minimize them, that is, to achieve the following optimal economic dispatching goal.

[0067]

[0068] Where F is the total operating cost of the system.

[0069] Building a reinforcement learning model for optimizing integrated energy system operations

[0070] Compared with other DRL algorithms based on policy gradients, the maximum entropy action encouragement mechanism introduced by the SAC algorithm improves the robustness of the algorithm and can explore better scheduling strategies in complex energy systems. Among them:

[0071] (1) SAC algorithm description: The training goal of the reinforcement learning model is to find the optimal strategy π * Maximize the expected value of cumulative return, that is:

[0072]

[0073] In the formula ρπ is the state-action trajectory distribution formed by strategy π. For the high-dimensional non-convex and nonlinear optimization problem formed by the scheduling of the electric-gas coupling system, the model is easy to fall into the local optimal solution, and the model effect is very sensitive to the hyperparameters and neural network structure. Therefore, in order to improve the exploration ability of the reinforcement learning model, the present invention adopts flexible actor-judge reinforcement learning, and the objective function of the model under the maximum entropy framework is

[0074]

[0075] Where: H is state s t The entropy term for taking actions under the condition s; α is the temperature coefficient, which indicates the proportion of exploration ability. It can be seen that the objective function of the method of the present invention contains two terms. The first term is the sum of cumulative rewards, and the second term is the entropy regularization term, that is, in a certain state s t Under this condition, the entropy of the selected action distribution is as large as possible. Therefore, while the agent pursues the maximum cumulative reward, it is also encouraged to explore more action spaces to avoid falling into the local optimal solution too early. The modified Bellman equation is:

[0076]

[0077] Where: γ is the reward discount factor; V(s t ) is the state value function.

[0078] It can be proved that the improved strategy distribution is transformed into a probability distribution in the form of an energy function, and is proportional to the exponential form of the Q function. The action-value function at this time is called a soft Q-value function (softQvalue), that is:

[0079] π(a t |s t )~exp(Q(s t ,a t )(twenty four)

[0080] In the process of improving the agent's strategy, the strategy should be made to tend to the exponential form of the Q-value function. Here, the strategy is output as a Gaussian distribution, and the gap between the two distributions is minimized by minimizing the KL divergence.

[0081]

[0082] Where: KLD is the KL divergence; Π is the strategy set; For the old strategy π old The value function under ; is the allocation function under the old strategy, and is the normalized distribution of Q values.

[0083] In order to improve the adaptive learning and generalization capabilities of the model, a deep neural network is used to represent Value functions and policy distributions. The parameters of the value network are learned by minimizing the residual To achieve, that is:

[0084]

[0085] Where: θ is the parameter of the Q-value network; φ is the parameter of the policy network; is the parameter of the target Q value network; the following update method is used to assign values ​​during training, namely:

[0086]

[0087] Where λ is the update parameter, and the present invention takes 0.008. In practice, in order to enhance the stability of training, two Q value networks are introduced, and the one with the smaller value is selected each time in the iteration. For the update of the policy network, that is, to minimize the KL divergence of formula (6),

[0088] Degrees, remove the constant term And multiply it by the coefficient α, and after transformation we get:

[0089]

[0090] In the formula, action a t It is represented by the following function:

[0091]

[0092] That is, the strategy's neural network outputs the mean μ and the logarithm of the variance lnσ, and then takes the exponent to get the variance σ (because it is difficult to ensure that the output of the neural network is always positive, so the logarithmic function is output first, and then the exponent is taken), ε t is the noise sampled from a standard normal distribution.

[0093] In addition, the temperature coefficient α should also be adaptively updated during the training process by minimizing J(α), that is,

[0094]

[0095] Where H0 represents the dimension of the action matrix.

[0096] During the interaction between the intelligent agent and the environment, equations (27-30) are continuously updated to achieve gradual convergence of model parameters.

[0097] Since the above model adopts a flexible Value function, so the method of the present invention is called flexible actor-judge method. For the problem of electricity-gas integrated energy optimization scheduling, the key elements of the deep reinforcement learning model include environmental model, agent state space, action space, reward mechanism, neural network model and training, etc. The next section will be based on the above framework to carry out detailed design of each link of the IES system optimization scheduling agent.

[0098] 3.1.2 State Space Design

[0099] The operation process of the integrated energy system is the operation process of each energy device in the system. The output power of typical energy equipment is divided into T time periods per unit time. The change of output of the same equipment in adjacent time periods will affect the operation status of the system, such as the maintenance cost of the equipment and the limit of the unit climbing. Therefore, it is necessary to optimize the output power of the equipment in each time period of the system as a whole. Therefore, in the multi-energy system environment of the present invention, the system environment provides the deep reinforcement learning agent with information including: wind energy, solar energy, time-of-use electricity price, electric load, thermal load, and electric energy storage. The state space at time t is expressed as follows:

[0100]

[0101] Among them, Γ t The goal of low-carbon economic dispatch of integrated energy systems is to determine the optimal unit output.

[0102] 3.1.3 Action Space Design

[0103] After the agent obtains state information from the environment, it will select an action in the action space according to its own strategy. The equipment model in the regional integrated energy system is relatively complex, and there are many types of energy storage and energy conversion equipment. As can be seen from Section 3.1.1, there is coupling between the output power of the equipment. In order to simplify the action space, the action of the energy conversion equipment here uses the power output of the electric boiler and cogeneration. The action space is as follows:

[0104]

[0105] in, Indicates whether the energy storage device performs discharge action at time t.

[0106] In order to make the constructed action satisfy the actual operation process of the IES model, it should satisfy the constraints in equation (15). t and action a t , the energy system transitions to the next state s t+1 , defined as:

[0107]

[0108] Where: It is a collection of environmental state transition probabilities, which simulates the dynamic changes and uncertainties involved in the low-carbon economic dispatch of the integrated energy system.

[0109] 3.1.4 Reward Strategy

[0110] The reward strategy is to quantify the target task. The reinforcement learning method learns by maximizing the cumulative reward value and guides the intelligent agent to optimize towards the goal. For this integrated energy system, the reward function mainly comes from operating costs, energy sales income, and carbon emissions. The operating cost is shown in formula (). Considering the small scale of the regional integrated energy system, the cost of starting and stopping the equipment can be ignored. In addition, the penalty for exceeding the output power limit of energy equipment in the system is expressed as follows:

[0111]

[0112] in, represents the penalty variable for energy conversion equipment i’s output exceeding the constraint condition during period t; is the output power of energy device i in period t, P i,min , P i,max are the upper and lower limits of the output power of energy device i respectively.

[0113] The cost of energy equipment output exceeding the limit, C T The sum of the penalty variables of each energy device in each unit time is as follows:

[0114]

[0115] The energy system also needs to consider carbon emissions during operation. The present invention will emit 0.386kg CO2 / kwh per unit of electricity and 1.433kg CO2 / kwh per unit of natural gas. Carbon emissions during a period are defined as:

[0116]

[0117] The reward fed back to the agent by the environmental model at each step is the component of the above objective functions. The reinforcement learning agent learns in the form of maximizing the cumulative return. The optimization scheduling of the present invention takes economy and carbon emission as the goals.

[0118] From the above analysis, we can get the total reward function of period t as:

[0119] R t =ω1·F+ω2·C T +ω3·C c (37)

[0120] Among them, ω1, ω2, and ω3 represent the weight coefficients of the three in the target reward. ω2 is higher in magnitude than the other two coefficients, so that the overall reward is balanced through coefficient adjustment to ensure the convergence and stability of model training.

[0121] Further Improvement of SAC Algorithm

[0122] Improvement ideas:

[0123] During the exploration process of the agent, if the action space disturbance is too large, the action network loss function of the model will oscillate and fail to converge during the initial training process. If the action space and action noise are too small, it will easily lead to insufficient model exploration ability and low sample training efficiency. In order to ensure the stability of model training, reduce the action space, and improve the exploration ability of the network model, a dynamic update strategy for action disturbance is proposed, and the standard deviation of the action noise distribution is processed as follows:

[0124] μ n =μ0+δ·n,μ n ≤μ max (38)

[0125] Among them, μ n represents the current value of the standard deviation, μ0 represents the initial value of the standard deviation, n represents the number of iterations, and δ is the standard deviation growth coefficient. The actions after output are as follows:

[0126]

[0127] Where f is the action value output by the action network; N represents Gaussian distribution sampling.

[0128] In the initial stage of network model training, the disturbance of the action is reduced to reduce the impact of excessive weight of the constraint penalty variable.

[0129] To ensure the stability of the network model training process, as the number of iterations increases, the noise of the action gradually increases, and the exploration ability of the network is gradually expanded to increase the sample training efficiency, improve the learning ability of the network model, and enhance exploration to help avoid local optimal solutions.

[0130] The overall process of the algorithm:

[0131] Deep reinforcement learning based on the improved SAC framework is used to optimize the scheduling of the integrated energy system. The model framework is as follows: Figure 2 The actuator firstly operates according to the operating status s of the electric-gas integrated energy system observed at this time. t , based on the action perturbation dynamic update strategy to generate action a t After that, the energy system transfers state according to the control strategy at that time and reaches the state s at the next moment. t+1 Since the system's new energy output and load are inherently random, when the state is transferred, it is necessary to sample according to the probability distribution of the source and load to generate a random operation scenario for the next moment. At the same time, the system environment calculates the distribution of the electric-gas energy flow and further gives the reward r at time t. t , feedback to the agent, the agent records (s t ,a t ,r t ,s t+1 ) to the experience pool. Then, the action strategy sampling of the actor and the system state are simultaneously input to the judge, and the action-value function Q is output to evaluate the strategy. This process is repeated, and the actor and the judge update their neural network parameters through the gradient descent method to achieve model adaptive learning. During the training process, the cumulative reward of the agent in the scheduling cycle will gradually increase and eventually stabilize.

[0132] Technical effects and advantages of the present invention:

[0133] This method has carried out detailed design of the action space, state space, reward mechanism, neural network structure, etc. of the intelligent agent, and on this basis, it considers the minimization of the system operation cost and adopts the deep reinforcement learning method for scheduling simulation. This method uses the typical park comprehensive energy structure to simulate the algorithm. The results show that the method proposed in this invention can perform optimal scheduling according to the load demand of the system, and has good scheduling performance and self-learning ability, which proves the effectiveness of the proposed method in reducing the carbon emissions of the system and improving the economy.

[0134] The proposed reinforcement learning method can optimize the continuous actions of the equipment, effectively avoid the dimensionality curse problem of optimization calculation, and improve the convergence speed and exploration ability;

[0135] The reinforcement learning of the present invention does not require retraining of the model under the condition of random fluctuations in source and load, and can provide a more reasonable optimization scheduling strategy;

[0136] The proposed reinforcement learning model was used to carry out simulation tests and result analysis in the system, verifying the effectiveness of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] Figure 1 It is the energy system block diagram;

[0138] Figure 2 To optimize the flow chart;

[0139] Figure 3 is a reward value curve graph;

[0140] Figure 4 is the loss value curve;

[0141] Figure 5 is the power exchange curve diagram;

[0142] Figure 6 is the cost curve diagram;

[0143] Figure 7 is the electric power dispatch curve diagram;

[0144] Figure 8 is the thermal power dispatch curve diagram;

[0145] Fig. 9 It is the remaining capacity curve of the energy storage device. DETAILED DESCRIPTION

[0146] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0147] Embodiment 1

[0148] Test scenario description

[0149] In order to verify the effectiveness of the proposed low-carbon economic dispatch of the integrated energy system based on deep reinforcement learning, Figure 1The structure of the comprehensive energy system shown in the figure is subjected to simulation experimental analysis. The daily operation of a park in central China is taken as the simulation research object, with 24 hours as a complete operation cycle and a unit interval time of 1 hour. The installed capacity of wind power in the system is 3.6MW; the total installed area of ​​the photovoltaic system is 3000㎡, and the installed capacity is 146.7kw. The variables that need to be optimized in the numerical experiment include the power generation power of the CHP unit, the heating power of the electric boiler, the heating power of the gas boiler and the grid access power in each time period. The installed capacity of the micro gas turbine in the cogeneration device is 3600kw, and the heat consumption parameters are 1.76, 1.44, and 0867, respectively, and the electric thermal efficiency is 0.3 and 0.4, respectively; the installed capacity of the gas boiler is 325.87kw, and the thermal efficiency is 0.92; the maximum capacity of the power storage system is 400kwh, and the storage and discharge efficiencies are 0.95 and 0.91, respectively. In the present invention, the calorific value of natural gas is 39.8MJ / Nm 3 .

[0150] The natural gas tiered price in the region is shown in Table 1.

[0151]

[0152] The time-of-use electricity prices in this area are shown in Table 2.

[0153]

[0154] The reinforcement learning model of the present invention uses a deep neural network to construct an actor and a judge, wherein the input of the neural network model of the actor Actor is the state set at the current moment, the activation function is the ReLU function, the mean μ and the logarithm of the variance lnσ of each action are output, and then the equation σ is obtained by taking the exponent. The action is obtained through the Tanh layer, and then the control amount of the actual system can be obtained through mapping. The Actor network of the present invention is 3 layers, and the number of neural units is 128, 128, and 32 respectively. The input of the judge Critic is a set of state s and action a, and the two are spliced ​​after passing through a layer of neural network and activation function, and then passing through multiple hidden layers and ReLU activation layers to finally obtain the Q value. The number of neurons in the Critic network of the present invention is 32, 128, 128, and 1 respectively. In the reinforcement learning parameters, the reward discount factor is 0.87, the exploration round is 50, the maximum number of iterations is 10000, the experience replay pool capacity is 5000, the batch size is 40, the learning rate is 0.001, and the soft update coefficient is 0.005.

[0155] Embodiment 2

[0156] Results Analysis

[0157] In order to verify the effectiveness of the energy system optimization operation based on deep reinforcement learning proposed in this invention, the historical operation data of the park is used as training data and test data to analyze the algorithm convergence, system operation cost, carbon emissions and other results.

[0158] 1.1 Algorithm Performance Analysis

[0159] like Figure 3 , 4 respectively show the comparison between the agent’s reward function value curve and loss curve during the exploration and learning process.

[0160] Compared with the traditional SAC algorithm, the method of the present invention can obtain higher reward values ​​and converge faster. It can be observed that the reward value received by the agent after executing the scheduling decision is very small because the agent is unfamiliar with the environment in the early stage. As the training process continues, the agent continues to interact with the environment and accumulate experience, and the overall trend of the reward value gradually increases. This shows that the agent has learned the optimal scheduling strategy that minimizes the system operation cost. At the same time, it is noted that due to the randomness of exploration, the reward value will oscillate during the training process.

[0161] 1.2 Scheduling Result Analysis

[0162] After the algorithm network is trained using historical data, the obtained network is saved for dynamic economic dispatch of the system. In order to verify the robustness and generalization of the proposed method, a day in the test set is randomly selected as the verification data.

[0163] from Figure 5 It can be seen that in the three time periods of 7-10 o'clock, 12-16 o'clock, and 19-21 o'clock, the integrated energy system not only obtains electricity from the power grid, but also relies on its own equipment to supply power. It provides electric power for the system through cogeneration devices, fans, photovoltaics and other production equipment. The combination of these two methods meets the system's electrical load balance.

[0164] from Figure 6 It can be seen that the overall operating cost of the comprehensive energy system is greatly increased in the two time periods of 10 o'clock and 18-21 o'clock. The reason is that in order to meet the load demand of the system in these two time periods, on the one hand, a large amount of electricity and gas energy needs to be purchased from outside the system, and at the same time, the equipment inside the system needs to keep high power output and continuously turned on, which increases the system operation cost. Figure 6 and Figure 7The power supply of the equipment in each time period is shown in the figure. The energy storage device discharges at 22:00-6:00 and 11:00. The main reason is that the system's power load demand is relatively small during this period. The use of the energy storage device to provide power improves economic efficiency. The system purchases a large amount of electricity from the power grid at 7:00-10:00, 12:00-16:00, and 19:00-21:00. On the one hand, the energy system requires a large power load during this period. On the other hand, the time-of-use electricity price during the flat and valley periods is relatively cheap, which can be used for energy supply and storage. In addition, renewable energy sources also supply power to the system from 8:00 to 15:00. During this period, sunlight, wind power, etc. are relatively sufficient and can provide a considerable part of the electricity. During the valley period of electricity prices, the proportion of combined heat and power generation is reduced, and the power supply is mainly purchased from the external power grid. During the peak period of electricity prices, the amount of electricity purchased from the external power grid decreases, and the electricity is mainly provided by the system generator sets to reduce the pressure on power supply and save the system's power purchase costs.

[0165] The heat load in the system is mainly provided by cogeneration devices, gas boilers, and electric boilers, and their main input media are natural gas and electricity, respectively. Figure 8 It can be seen that between 0 and 8 o'clock, the combined heat and power generation device and the electric boiler provide more heat, and the time-of-use electricity price is lower at this time, which is more economical. Between 9 and 19 o'clock, the electricity price is at its peak, and it is more reasonable to use gas as the main source of heat supply. The proportion of the three heat supply has changed and is in dynamic scheduling. When the gas price rises, the heat energy is mainly provided by the combined heat and power, and the gas boiler is used for peak load adjustment, saving the system's gas purchase cost. On the contrary, when the gas price decreases, the heat energy is mainly provided by the gas boiler, and the combined heat and power device and the electric boiler are used for peak load adjustment, saving the system's electricity purchase cost. Comprehensive analysis shows that, in view of the volatility of electricity and gas prices, the optimization results can improve the economy of system operation and achieve the purpose of optimization.

[0166] Fig. 9 The operation and dispatch of the electric energy storage device is shown in the figure. The energy storage device continues to discharge during the period of 0-5 o'clock. The main reason is that the system's electrical load demand is relatively low during this period. The use of the electric energy storage device to provide electricity can effectively meet the system's electrical load while improving economic efficiency. During the period of 6-10 o'clock, the energy storage device is mainly in a charging state, and the capacity has increased. During the period of 11-19 o'clock, the system's electrical load is input by other equipment, and the energy storage capacity of the electric energy storage device has basically not changed. Around 20 o'clock, the electric energy storage is in a charging state again, which increases the capacity. Because there is little energy supply from renewable energy at this time, the surplus electricity of the cogeneration device can be stored in the energy storage device.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A comprehensive energy system economic dispatch model method based on deep reinforcement learning, characterized by: include: Firstly, the park equipment environment model is constructed, and the low-carbon economic dispatch problem of the park's integrated energy system is mathematically described with economy and carbon emissions as the goals. Secondly, the scheduling problem is expressed as a Markov decision process framework, and the state space, action space and reward mechanism of the system are reasonably designed; Then, the proximal strategy optimization algorithm is used to make low-carbon economic dispatch decisions, and the action noise dynamic update strategy is proposed to gradually improve the exploration and learning capabilities of the network model; The environmental model includes the flow of electrical energy, thermal energy, cold energy and the coupling links between energy sources in the integrated energy system. The environmental model is built around various energy equipment in the system and the balance conditions of supply and demand of the system energy load.

2. The comprehensive energy system economic dispatch model method based on deep reinforcement learning according to claim 1 is characterized by: The environmental model is an integrated energy system environment that interacts with the intelligent agent. It calculates the output status of the energy equipment under the action strategy given by the deep reinforcement learning intelligent agent and feeds back the reward function. The environmental model includes the electric energy flow, thermal energy flow, cold energy flow and coupling links between energy sources in the integrated energy system, and constructs the environmental model around various energy equipment in the system and the supply and demand balance conditions of the system energy load.

3. The method for economic dispatching model of integrated energy system based on deep reinforcement learning according to claim 1 is characterized in that: The integrated energy system mainly includes three heterogeneous energy forms: electricity, heat and gas, and meets the diversified energy needs within the system through a variety of energy conversion equipment; the integrated energy system is divided according to the different functions of the equipment during the operation of the system, and includes three types of renewable energy equipment, energy conversion equipment and energy storage equipment; the system renewable energy equipment includes photovoltaic (PV) and wind energy (WT); the energy conversion equipment includes electric heat boiler (EHB), gas boiler (GB), gas turbine (GT), waste heat boiler, and combined heat and power (CHP); the energy storage equipment mainly considers battery energy storage (BES); the load side mainly includes electric load, gas load and thermal load; the economic scheduling goal is to minimize the total economic cost while maintaining the normal operation of the system; among them, renewable energy power generation and energy storage devices are connected to the grid bus in the form of nodes to realize the coordinated work of multi-source power generation to supply the load power demand.

4. The method for economic dispatching model of integrated energy system based on deep reinforcement learning according to claim 3 is characterized in that: Model building of energy conversion equipment includes: (1) Wind turbine device model: The power of a wind turbine unit is completely controlled by the wind speed. The power of wind turbine generation is expressed as follows: In the above formula, v is the current operating wind speed of the fan, v i is the set cut-in wind speed, v o is the set cut-out wind speed, v r is the set rated wind speed; from the above formula, it can be seen that when the fan is below the cut-in wind speed or above the cut-out wind speed, it cannot work normally because it cannot drive the fan to operate or is easy to damage the fan; when the wind speed is below the rated wind speed and above the cut-in wind speed, the fan is in a low-power operation state; when the wind speed is above the rated wind speed and below the cut-out wind speed, the fan operates at rated power and is also running near the optimal tip speed ratio; (2) Photovoltaic device model: The power output of the photovoltaic power generation unit mainly depends on the intensity of the light source and the system parameters of the device itself; the expression of the photovoltaic generator power is generated as follows: P PV =λ P,t cosθκ m A pv or pv (2) In the formula, λ P,t represents the light intensity at time t, k m ,A pv ,η pv They represent the efficiency of the controller, the area of ​​the photovoltaic panel, and the rated power generation efficiency of the photovoltaic panel respectively; (3) Electric boiler model: Electric Boiler (EB) is an electric-thermal coupling device in the integrated energy system. It uses electric heating components to convert electrical energy into thermal energy. The operation of electric boilers is less affected by the ambient temperature and can still operate normally in extremely cold weather. Therefore, they are widely used in heating. The process of converting electricity into thermal energy is as follows: In the formula, represents the thermal power of the electric boiler at time t, represents the power consumption at time t, η EHB and κ l They represent the electric-to-heat conversion efficiency and the heat utilization efficiency respectively; (4) Gas boiler model: Gas Boiler (GB) is a device that uses natural gas as energy input and heats low-temperature water through high-temperature flue gas to achieve gas-heat conversion; in, are the heating power of the gas boiler during period t and the energy consumed by natural gas respectively; For the operating efficiency of gas boilers; (5) Gas turbine model: A gas turbine is a device that converts thermal energy into mechanical energy. The compressor draws in air from the outside atmosphere and compresses it step by step through the axial compressor to increase the pressure. The air temperature also increases accordingly. The compressed air is sent to the combustion chamber to mix with the injected fuel and burn to generate high-temperature and high-pressure gas. The gas then enters the turbine to expand and do work, driving the turbine to drive the compressor and the external load rotor to rotate at high speed, thus partially converting the chemical energy of the gas or liquid fuel into mechanical work and outputting electrical energy. Where: is the heat value input to the gas turbine, α i , β i , γ i is a parameter determined by the gas turbine heat rate curve; is the power generation capacity of the gas turbine; is the equivalent gas load, H g is the calorific value of natural gas; (6) Cogeneration unit model The cogeneration unit includes a gas turbine and a waste heat recovery boiler; the gas turbine generates electricity through the consumption of natural gas, and also produces flue gas with thermal energy, outputting thermal power; and the waste heat boiler can also generate thermal power through heat exchange; the power generation capacity of the cogeneration unit is as follows: In the formula, represents the natural gas consumption at time t, H g Indicates the calorific value of gas, α GT represents the power generation efficiency of the gas turbine, δ GT It represents the gas-to-electricity conversion efficiency of the gas turbine; In addition, the heat generation capacity of the combined heat and power plant is as follows: In the formula, α SG Indicates the heat exchange efficiency of the waste heat boiler; In addition, the waste heat recovery boiler collects the heat from the flue gas discharged by the gas turbine and supplies it to the heat network; its output thermal power is Where: is the output thermal power of the waste heat recovery boiler during period t; is the thermal power output of the gas turbine during period t; SG is the heat exchange efficiency of the waste heat boiler; (7) Energy storage device model When the energy storage device is connected to the energy grid, energy exchange will be carried out as the system operates. The present invention mainly considers the electric energy storage device; the remaining energy SOC(t) of the energy storage device at time t can be expressed as follows: In the formula, The charging and discharging power of the energy storage device, λ B represents the exchange efficiency, Represents the charge and discharge status at time t, and takes the value of {0,1} integer variable.

5. The method for economic dispatching model of integrated energy system based on deep reinforcement learning according to claim 3 is characterized in that: Also included are build system constraints, including: In the energy system, it is necessary to maintain a balance between energy supply and demand; at every moment, the power on both sides of the grid supply and demand must be kept in a balanced state; the constraints of the low-carbon economic dispatch system of the integrated energy system include power balance constraints, unit output constraints, unit ramp constraints, energy storage state constraints, and upper-level grid and gas grid interaction constraints; (1) Electric power balance constraints In the system, the power generated by distributed generation units such as photovoltaic, wind, and cogeneration and the energy exchange power of energy storage devices must be balanced with the total electrical load demand; (2) Thermal power balance constraints In the formula, E EHB is the heating power of the electric boiler, E GB is the heating power of the gas boiler, E GHP is the thermal power of the combined heat and power generation unit, E SG is the heat output power of the waste heat boiler, E Load is the system heat load demand; (3) Power Constraint Each device in the integrated energy system has an upper and lower operating power limit, including wind turbines, photovoltaic generators, cogeneration units and electrical energy storage devices; the output of each device has the following constraints: In the formula, the power generation and heat generation constraints of wind turbines, photovoltaic generators, and cogeneration units are listed respectively, and the capacity constraint of the remaining energy of the energy storage device is given; (4) Climbing constraints The output power of some equipment in the integrated energy system cannot change suddenly, so there is a ramp limit on the output power of the cogeneration unit; R CHP,min ≤P CHP,t -P CHP,t-1 ≤R CHP,max (16) Similarly, gas turbine and gas boiler units also have ramp restrictions, as follows: R GT,min ≤P GT,t -P GT,t-1 ≤R GT,max (17) R EB,min ≤P EB,t -P EB,t-1 ≤R EB,max (18)。 6. The method for economic dispatch model of integrated energy system based on deep reinforcement learning according to claim 3 is characterized in that: It also includes the construction of a low-carbon economic dispatch objective function for an integrated energy system: The core goal of the economic dispatch problem is to minimize the total economic cost while maintaining the normal operation of the system. In this invention, in order to simplify the calculation, the cost function is mainly set to consider the following: the power generation cost of the wind power generation unit, the cost of the photovoltaic power generation unit, the power generation cost of natural gas power generation, the cost of interacting with the power grid, and the loss cost of battery charging and discharging, which are set as C WG (t), C PV (t), C Grid (t), C SOC (t), C Gas (t); The dispatching goal of the present invention is to optimize the sum of these five costs and minimize them, that is, to achieve the following optimal economic dispatching goal; Where F is the total operating cost of the system.

7. The method for economic dispatching model of integrated energy system based on deep reinforcement learning according to claim 1, characterized in that: It also includes building a reinforcement learning model for optimizing the operation of integrated energy systems Compared with other DRL algorithms based on policy gradients, the maximum entropy action encouragement mechanism introduced by the SAC algorithm improves the robustness of the algorithm and can explore better scheduling strategies in complex energy systems; among them: (1) SAC algorithm description: The training goal of the reinforcement learning model is to find the optimal strategy π * Maximize the expected value of cumulative return, that is: In the formula ρπ is the state-action trajectory distribution formed by strategy π; for the high-dimensional non-convex and nonlinear optimization problem formed by the scheduling of the electric-gas coupling system, the model is easy to fall into the local optimal solution, and the model effect is very sensitive to the hyperparameters and neural network structure; therefore, in order to improve the exploration ability of the reinforcement learning model, the present invention adopts flexible actor-judge reinforcement learning, and the objective function of the model under the maximum entropy framework is Where: H is state s t The entropy term for taking actions under s; α is the temperature coefficient, which indicates the proportion of exploration ability; it can be seen that the objective function of the method of the present invention contains two terms, the first term is the sum of cumulative rewards, and the second term is the entropy regularization term, that is, in a certain state s t Under this condition, the entropy of the selected action distribution is as large as possible; therefore, while the agent pursues the maximum cumulative reward, it is also encouraged to explore more action spaces to avoid falling into the local optimal solution too early; the modified Bellman equation is: Where: γ is the reward discount factor; V(s t ) is the state value function; It can be proved that the improved strategy distribution is transformed into a probability distribution in the form of an energy function, and is proportional to the exponential form of the Q function. The action-value function at this time is called a soft Q-value function (softQvalue), that is: π(a t |s t )~exp(Q(s t ,a t )(24) In the process of improving the agent's strategy, the strategy should be made to tend to the exponential form of the Q-value function; here the strategy is output as a Gaussian distribution, and the gap between the two distributions is minimized by minimizing the KL divergence; Where: KLD is the KL divergence; Π is the strategy set; For the old strategy π old The value function under ; is the allocation function under the old strategy, and is the normalized distribution of Q value; In order to improve the adaptive learning and generalization ability of the model, a deep neural network is used to represent the Q-value function and policy distribution; the parameter learning of the Q-value network is achieved by minimizing the residual JQ(θ), that is: Where: θ is the parameter of the Q-value network; φ is the parameter of the policy network; is the parameter of the target Q value network; the following update method is used to assign values ​​during training, namely: Where λ is the update parameter, which is taken as 0.008 in this paper. In practice, in order to enhance the stability of training, two Q value networks are introduced, and the one with the smaller value is selected each time in the iteration. For the update of the policy network, the KL divergence of formula (6) is minimized. Degrees, remove the constant term And multiply it by the coefficient α, and after transformation we get: In the formula, action a t It is represented by the following function: That is, the strategy's neural network outputs the mean μ and the logarithm of the variance lnσ, and then takes the exponent to get the variance σ (because it is difficult to ensure that the output of the neural network is always positive, so the logarithmic function is output first, and then the exponent is taken), ε t is the noise sampled from a standard normal distribution; In addition, the temperature coefficient α should also be adaptively updated during the training process by minimizing J(α), that is, Where H0 represents the dimension of the action matrix; In the process of interaction between the agent and the environment, equations (27-30) are continuously updated to achieve gradual convergence of model parameters; Since the above model adopts a flexible Q-value function under the actor-critic framework, the method of the present invention is called a flexible actor-critic method. For the problem of electricity-gas integrated energy optimization scheduling, the key elements of the deep reinforcement learning model include the environment model, agent state space, action space, reward mechanism, neural network model and training, etc. The next section will be based on the above framework to carry out detailed design of each link of the IES system optimization scheduling agent. 3.1.2 State Space Design The operation process of the integrated energy system is the operation process of each energy device in the system. The output power of typical energy equipment is divided into T time periods per unit time. The change of output of the same equipment in adjacent time periods will affect the operation status of the system, such as the maintenance cost of the equipment and the limit of the unit climbing. Therefore, it is necessary to optimize the output power of the equipment in each time period of the system as a whole. Therefore, in the multi-energy system environment of the present invention, the system environment provides the deep reinforcement learning agent with information including: wind energy, solar energy, time-of-use electricity price, electric load, thermal load, and electric energy storage. The state space at time t is expressed as follows: Among them, Γ t Indicates the time-of-use electricity price; the goal of low-carbon economic dispatch of the integrated energy system is to determine the optimal unit output; 3.1.3 Action Space Design After the agent obtains state information from the environment, it will select an action in the action space according to its own strategy; the equipment model in the regional integrated energy system is relatively complex, and there are many types of energy storage and energy conversion equipment; from 3.1.1, it can be seen that there is coupling between the output power of the equipment. In order to simplify the action space, the action of the energy conversion equipment here uses the power output of the electric boiler and the cogeneration; the action space is as follows: in, Indicates whether the electric energy storage device performs the discharge action at time t; In order to make the constructed action satisfy the actual operation process of the IES model, it should satisfy the constraints in formula (15); given the state s of time step t t and action a t , the energy system transitions to the next state s t+1 , defined as: Where: is a collection of environmental state transition probabilities, which simulates the dynamic changes and uncertainties involved in the low-carbon economic dispatch of the integrated energy system; 3.1.4 Reward Strategy The reward strategy is to quantify the target task. The reinforcement learning method learns by maximizing the cumulative return value and guides the agent to optimize towards the goal. For this integrated energy system, the reward function mainly comes from the operating cost, energy sales income, and carbon emissions. The operating cost is shown in formula (). Considering the small scale of the regional integrated energy system, the cost of starting and stopping the equipment can be ignored. In addition, the penalty for exceeding the output power limit of the energy equipment in the system is expressed as follows: in, represents the penalty variable for energy conversion equipment i’s output exceeding the constraint condition during period t; is the output power of energy device i in period t, P i,min , P i,max are the upper and lower limits of the output power of energy device i respectively; The cost of energy equipment output exceeding the limit, C T The sum of the penalty variables of each energy device in each unit time is as follows: The energy system also needs to consider carbon emissions during operation. The unit electricity of the present invention will emit 0.386kg CO2 / kwh, and the unit natural gas will produce 1.433kg CO2 / m 3 ; Carbon emissions in period t are defined as: The reward fed back to the agent by the environmental model at each step is the component of the above objective functions. The reinforcement learning agent learns in the form of maximizing the cumulative return. The optimization scheduling of the present invention takes economy and carbon emission as the goals. From the above analysis, we can get the total reward function of period t as: R t =ω1·F+ω2·C T +ω3·C c (37) Among them, ω1, ω2, and ω3 represent the weight coefficients of the three in the target reward respectively; ω2 is higher in magnitude than the other two coefficients, so that the overall reward is balanced through coefficient adjustment to ensure the convergence and stability of model training.

8. The method for economic dispatching model of integrated energy system based on deep reinforcement learning according to claim 7 is characterized in that: Also includes further improvement of SAC algorithm Improvement ideas: When the action space perturbation is too large during the exploration process of the intelligent agent, the action network loss function will oscillate and fail to converge during the initial training of the model. When the action space and action noise are too small, the model exploration ability is insufficient and the sample training efficiency is low. In order to ensure the stability of model training, reduce the action space, and improve the exploration ability of the network model, a dynamic update strategy for action perturbation is proposed, and the standard deviation of the action noise distribution is processed as follows: m n =μ0+δ·n,μ n ≤μ max (38) Among them, μ n represents the current value of the standard deviation, μ0 represents the initial value of the standard deviation; n represents the number of iterations; δ is the standard deviation growth coefficient; the actions after output are as follows: Where f is the action value output by the action network; N represents Gaussian distribution sampling; In the initial stage of network model training, the disturbance of the action is reduced to reduce the impact of excessive weight of the constraint penalty variable. To ensure the stability of the network model training process, as the number of iterations increases, the noise of the action gradually increases, and the exploration ability of the network is gradually expanded to increase the sample training efficiency, improve the learning ability of the network model, and enhance exploration to help avoid local optimal solutions; Overall algorithm process: Deep reinforcement learning based on the improved SAC framework is used to optimize the scheduling of the integrated energy system; the actor firstly calculates the operating status s of the electric-gas integrated energy system observed at this time. t , based on the action perturbation dynamic update strategy to generate action a t ; After that, the energy system transfers state according to the control strategy at this time and reaches the state s at the next moment t+1 ; Since the system's new energy output and load are inherently random, when the state is transferred, it is necessary to sample according to the probability distribution of the source and load to generate a random operation scenario for the next moment; at the same time, the system environment calculates the distribution of the electric-gas energy flow and further gives the reward r at time t t , feedback to the agent, the agent records (s t ,a t ,r t ,s t+1 ) into the experience pool; then, the action strategy sampling of the actor and the system state are simultaneously input into the judge, and the action-value function Q is output to evaluate the strategy; this process is repeated, and the actor and the judge update their neural network parameters through the gradient descent method to achieve model adaptive learning; during the training process, the cumulative reward of the agent within the scheduling cycle will gradually increase and eventually stabilize.

Images