Real-time regulation and control method and device for adjustable load resources

Abstract

The invention relates to the technical field of energy scheduling, and particularly provides an adjustable load resource real-time regulation and control method and device, and the method comprises the steps: enabling an observation state of an integrated energy system to serve as the input of a pre-trained deep reinforcement learning model, obtaining an adjustable load resource real-time regulation and control strategy output by the pre-trained deep reinforcement learning model; adjusting and controlling the adjustable load resources of the integrated energy system in real time by using the adjustable load resource real-time adjusting and controlling strategy; according to the technical scheme provided by the invention, through coordinated optimization of various adjustable load resources, the utilization rate of renewable energy sources can be improved, the deep learning model can quickly adjust a scheduling strategy, stable operation and power supply safety of the adjustable load resources are ensured, meanwhile, the carbon emission of the adjustable load resources is reduced, and the power supply efficiency is improved. And the goal of regional energy conservation and emission reduction can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of energy dispatching, and in particular to a method and device for real-time control of adjustable load resources. Background Art

[0002] With the massive increase of uncertain and unreliable factors in the power system, there is an urgent need to improve the simulation level of real-time adjustment of load resources. However, the low-carbon economic regulation of adjustable load resources involves multiple factors such as the operating status of electrical and thermal equipment, electricity and heat prices, energy costs, and environmental protection requirements. The randomness and diversity of energy bring huge challenges to the real-time optimization and dispatching management of loads.

[0003] Most of the existing control research methods are traditional optimization algorithms. However, algorithms such as linear programming, nonlinear programming, and mixed integer linear programming rely on precise mathematical models, which are very complex to solve and time-consuming to solve, and cannot meet the requirements of real-time online strategy generation.

[0004] For example, patent CN107681783A discloses a precise load shedding control system control terminal. Both the hardware and software adopt a modular design concept, mainly including two parts: stability control and power consumption information collection. The main control modules all adopt an ARM-based hardware platform. The stability control part is connected to the precise load shedding control system to achieve emergency and sub-emergency precise load shedding. The power consumption information collection part collects the data of the exchange board through the serial port, and calculates the exchange data and converts it into a network port message to send it to the main station to realize data collection and user power usage management functions. Through the refined regulation of the load, the regulation of non-productive loads and non-important loads replaces the total power outage of the region, thereby improving the reliability of power grid operation and power supply.

[0005] This invention only considers the fine-grained regulation of the load by the load control terminal based on the control instructions issued by the load control system, and does not consider the regulation method based on price signals, incentive guidance signals, and transaction information. Specifically, it does not consider the electricity price, project validity period and other information for price-type responses, and the participation conditions, incentive standards, advance notice time, response validity judgment principles, project validity period and other information for incentive-type responses. The regulation method is relatively simple.

[0006] For another example, patent CN112488531A discloses a real-time control method and device for heterogeneous flexible loads based on deep reinforcement learning. First, for different types of heterogeneous flexible loads, a single flexible load model is established respectively, and then an aggregated load model is established for multiple flexible loads with different parameters, so as to obtain the Markov decision process when heterogeneous flexible loads participate in demand response, and the decision function of the aggregate is trained based on historical data through the machine learning framework of deep reinforcement learning to obtain a real-time optimization control decision model for the heterogeneous flexible load aggregate, thereby obtaining the optimal strategy for real-time control of the aggregated load, and improving the flexible load response capability on the user side. The technical problem that the existing load control method has low responsiveness of the flexible load on the user side and is difficult to stimulate the demand response potential on the user side is solved.

[0007] The deep learning method of this invention stores the collected samples in the experience pool through the experience replay mechanism, and uniformly randomly extracts samples from it during training. However, random sampling lacks the distinction of sample importance, and it is difficult for the model to quickly focus on samples that are more critical to performance improvement during training, resulting in slow convergence. Summary of the invention

[0008] In order to overcome the above-mentioned defects, the present invention proposes a method and device for real-time control of adjustable load resources.

[0009] In a first aspect, a method for real-time control of adjustable load resources is provided, the method comprising: Taking the observed state of the integrated energy system as the input of a pre-trained deep reinforcement learning model, and obtaining an adjustable load resource real-time control strategy output by the pre-trained deep reinforcement learning model; Using the real-time control strategy for adjustable load resources to control the adjustable load resources of the integrated energy system in real time; Among them, the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, and gas boilers. The pre-trained deep reinforcement learning model is trained using the DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of a Markov decision process established for the real-time control problem of adjustable load resources.

[0010] Preferably, the mathematical model corresponding to the Markov decision process is: (S t ,A t ,S t+1 ,R t ), where S tis the observed state of the integrated energy system at time t, A t is the action variable of the adjustable load resource in the integrated energy system at time t, S t+1 is the observed state of the integrated energy system at time t+1, R t is the reward function at time t.

[0011] Furthermore, the observed state of the integrated energy system at time t is as follows: S t =S ta ×S tb S ta =[P t ,H t ] S tb =[D Et ,D Ht ] In the above formula, S ta is the controllable observable quantity, S tb is the uncontrollable observable quantity, P t is the electrical load state at time t, H t is the heat load state at time t, D Et is the electricity price at time t, D Ht is the hot price at time t.

[0012] Furthermore, the action variables of the adjustable load resources in the integrated energy system at time t are as follows: A t =A Et ×A Ht In the above formula, A Et is the control action of the electric load, A Ht Control action for heat load.

[0013] Furthermore, the state transfer function adopted by the Markov decision process is an MDP random transfer probability function.

[0014] Furthermore, the reward function at time t is as follows: R t =-(C omt +C gridt +C cart )×w In the above formula, C omt is the operation and maintenance cost at time t, C gridt is the electricity purchase and sales cost at time t, C cart is the step-wise carbon trading cost at time t, and w is the conversion coefficient.

[0015] Furthermore, the operation and maintenance cost at time t is as follows: C omt =∑ρ n P nt The electricity purchase and sale costs at time t are as follows: C gridt =ρ buyt P buyt +ρ salet P salet In the above formula, ρ n is the unit price of the operating cost of the nth unit, P nt is the power of the nth unit at time t, ρ buyt is the electricity purchase price at time t, P buyt is the power purchased from the public grid at time t, ρ salet is the electricity price at time t, P salet is the power sold to the public grid at time t.

[0016] Furthermore, the tiered carbon trading cost at time t is as follows: In the above formula, E MG is the system carbon emissions, E MG =E c -E q , E c is the actual carbon emissions, E q is the carbon emission quota, η is the carbon price, is the length of the carbon emission interval, and σ is the price growth rate.

[0017] Furthermore, the electrical load includes: wind power output, photovoltaic output, gas turbine output electrical power, power of heat-to-electricity conversion of organic Rankine cycle low-temperature waste heat power generation equipment, heat pump output electrical power, and charging and discharging power of power storage device; the thermal load includes: waste heat boiler output thermal power, gas boiler output thermal power, and charging and discharging power of heat storage device.

[0018] Furthermore, the electrical load satisfies: P buyt -P salet +P WTt + P PVt + P EGTt + P ORCt -P HPt -P ESCt + P ESDt =P Et The heat load meets: P WHBt +P GBt +P HPt COP HPt-P HSCt + P HSDt =P Ht In the above formula, P buyt is the power purchased from the public grid at time t, P salet is the power sold to the public grid at time t, P WTt is the wind power output at time t, P PVt is the photovoltaic output at time t, P EGTt is the gas turbine output power at time t, P ORCt is the power of the organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity at time t, P HPt is the output power of the heat pump at time t, P ESCt is the charging power of the power storage device at time t, P ESDt is the discharge power of the storage device at time t, P Et is the total electrical power of the system at time t, P WHBt is the heat output power of the waste heat boiler at time t, P GBt is the thermal power output of the gas boiler at time t, COP HPt is the heat pump heating performance coefficient, P HSCt is the charging power of the heat storage device at time t, P HSDt is the heat release power of the heat storage device at time t, P Ht is the total thermal power of the system at time t.

[0019] Preferably, in the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, when there are samples in the experience replay pool, samples in the experience replay pool are preferentially collected as training samples compared to pre-prepared training sample data.

[0020] Furthermore, the sampling probability of sample i in the experience replay pool is as follows: p(i)=p i / P In the above formula, p(i) is the sampling probability of sample i in the experience replay pool, p i is the priority of sample i in the experience replay pool, and P is the sum of the priorities of all samples in the experience replay pool.

[0021] Furthermore, in the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, the loss function is as follows: L = W × [y t -Q] / 2 In the above formula, L is the loss function value, W is the sum of the sampling weights of all samples in the experience replay pool, Q is the action value function estimated by the policy network, and y t is the target value at time t.

[0022] Furthermore, the sampling weight w of sample i in the experience replay pool i as follows: w i =1 / [ p(i) / p min ] α In the above formula, p min is the minimum sampling probability of samples in the experience replay pool, and α is the number of priority settings for samples in the experience replay pool.

[0023] Furthermore, the target value at time t is as follows: In the above formula, R t is the reward function at time t, β is the discount factor, and max{Q} is the action value function estimated using the target network.

[0024] In a second aspect, a real-time control device for adjustable load resources is provided, wherein the real-time control device for adjustable load resources comprises: An analysis module, used to use the observed state of the integrated energy system as an input of a pre-trained deep reinforcement learning model to obtain an adjustable load resource real-time control strategy output by the pre-trained deep reinforcement learning model; A control module, used to use the real-time control strategy of the adjustable load resources to control the adjustable load resources of the integrated energy system in real time; Among them, the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, and gas boilers. The pre-trained deep reinforcement learning model is trained using the DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of a Markov decision process established for the real-time control problem of adjustable load resources.

[0025] Preferably, the mathematical model corresponding to the Markov decision process is: (S t ,A t ,S t+1 ,R t ), where S t is the observed state of the integrated energy system at time t, A t is the action variable of the adjustable load resource in the integrated energy system at time t, S t+1 is the observed state of the integrated energy system at time t+1, R t is the reward function at time t.

[0026] Furthermore, the observed state of the integrated energy system at time t is as follows: S t =Sta ×S tb S ta =[P t ,H t ] S tb =[D Et ,D Ht ] In the above formula, S ta is the controllable observable quantity, S tb is the uncontrollable observable quantity, P t is the electrical load state at time t, H t is the heat load state at time t, D Et is the electricity price at time t, D Ht is the hot price at time t.

[0027] Furthermore, the action variables of the adjustable load resources in the integrated energy system at time t are as follows: A t =A Et ×A Ht In the above formula, A Et is the control action of the electric load, A Ht Control action for heat load.

[0028] Furthermore, the state transfer function adopted by the Markov decision process is an MDP random transfer probability function.

[0029] Furthermore, the reward function at time t is as follows: R t =-(C omt +C gridt +C cart )×w In the above formula, C omt is the operation and maintenance cost at time t, C gridt is the electricity purchase and sales cost at time t, C cart is the step-wise carbon trading cost at time t, and w is the conversion coefficient.

[0030] Furthermore, the operation and maintenance cost at time t is as follows: C omt =∑ρ n P nt The electricity purchase and sale costs at time t are as follows: C gridt =ρ buyt P buyt +ρ salet P salet In the above formula, ρ nis the unit price of the operating cost of the nth unit, P nt is the power of the nth unit at time t, ρ buyt is the electricity purchase price at time t, P buyt is the power purchased from the public grid at time t, ρ salet is the electricity price at time t, P salet is the power sold to the public grid at time t.

[0031] Furthermore, the tiered carbon trading cost at time t is as follows: In the above formula, E MG is the system carbon emissions, E MG =E c -E q , E c is the actual carbon emissions, E q is the carbon emission quota, η is the carbon price, is the length of the carbon emission interval, and σ is the price growth rate.

[0032] Furthermore, the electrical load includes: wind power output, photovoltaic output, gas turbine output electrical power, power of heat-to-electricity conversion of organic Rankine cycle low-temperature waste heat power generation equipment, heat pump output electrical power, and charging and discharging power of power storage device; the thermal load includes: waste heat boiler output thermal power, gas boiler output thermal power, and charging and discharging power of heat storage device.

[0033] Furthermore, the electrical load satisfies: P buyt -P salet +P WTt + P PVt + P EGTt + P ORCt -P HPt -P ESCt + P ESDt =P Et The heat load meets: P WHBt +P GBt +P HPt COP HPt -P HSCt + P HSDt =P Ht In the above formula, P buyt is the power purchased from the public grid at time t, P salet is the power sold to the public grid at time t, P WTt is the wind power output at time t, P PVt is the photovoltaic output at time t, P EGTtis the gas turbine output power at time t, P ORCt is the power of the organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity at time t, P HPt is the output power of the heat pump at time t, P ESCt is the charging power of the power storage device at time t, P ESDt is the discharge power of the storage device at time t, P Et is the total electrical power of the system at time t, P WHBt is the heat output power of the waste heat boiler at time t, P GBt is the thermal power output of the gas boiler at time t, COP HPt is the heat pump heating performance coefficient, P HSCt is the charging power of the heat storage device at time t, P HSDt is the heat release power of the heat storage device at time t, P Ht is the total thermal power of the system at time t.

[0034] Preferably, in the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, when there are samples in the experience replay pool, samples in the experience replay pool are preferentially collected as training samples compared to pre-prepared training sample data.

[0035] Furthermore, the sampling probability of sample i in the experience replay pool is as follows: p(i)=p i / P In the above formula, p(i) is the sampling probability of sample i in the experience replay pool, p i is the priority of sample i in the experience replay pool, and P is the sum of the priorities of all samples in the experience replay pool.

[0036] Furthermore, in the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, the loss function is as follows: L = W × [y t -Q] / 2 In the above formula, L is the loss function value, W is the sum of the sampling weights of all samples in the experience replay pool, Q is the action value function estimated by the policy network, and y t is the target value at time t.

[0037] Furthermore, the sampling weight w of sample i in the experience replay pool i as follows: w i =1 / [ p(i) / p min ] α In the above formula, p min is the minimum sampling probability of samples in the experience replay pool, and α is the number of priority settings for samples in the experience replay pool.

[0038] Furthermore, the target value at time t is as follows: In the above formula, R t is the reward function at time t, β is the discount factor, and max{Q} is the action value function estimated using the target network.

[0039] In a third aspect, a computer device is provided, comprising: one or more processors; The processor is configured to execute one or more programs; When the one or more programs are executed by the one or more processors, the adjustable load resource real-time control method is implemented.

[0040] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed, the method for real-time control of adjustable load resources is implemented.

[0041] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects: The present invention provides a method and device for real-time control of adjustable load resources, including: using the observed state of the integrated energy system as the input of a pre-trained deep reinforcement learning model to obtain a real-time control strategy for adjustable load resources output by the pre-trained deep reinforcement learning model; using the real-time control strategy for adjustable load resources to control the adjustable load resources of the integrated energy system in real time; wherein the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, gas boilers, the pre-trained deep reinforcement learning model is trained using a DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of a Markov decision process established by the real-time control problem of adjustable load resources. The technical solution provided by the present invention can improve the utilization rate of renewable energy by coordinating and optimizing multiple adjustable load resources, and the deep learning model can quickly adjust the scheduling strategy to ensure the stable operation and power supply safety of the adjustable load resources, while reducing the carbon emissions of the adjustable load resources, which is helpful to achieve the goal of regional energy conservation and emission reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the main steps of the method for real-time control of adjustable load resources according to an embodiment of the present invention; Figure 2 4 is a Markov decision process framework diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] As disclosed in the background technology, with the massive increase of uncertain and unreliable factors in the power system, there is an urgent need to improve the simulation level of real-time adjustment of load resources. However, the low-carbon economic regulation of adjustable load resources involves multiple factors such as the operating status of electrical and thermal equipment, electricity and heat prices, energy costs, environmental protection requirements, etc. The randomness and diversity of energy bring huge challenges to the real-time optimization and dispatching management of loads.

[0046] Most of the existing control research methods are traditional optimization algorithms. However, algorithms such as linear programming, nonlinear programming, and mixed integer linear programming rely on precise mathematical models, which are very complex to solve and time-consuming to solve, and cannot meet the requirements of real-time online strategy generation.

[0047] For example, patent CN107681783A discloses a precise load shedding control system control terminal. Both the hardware and software adopt a modular design concept, mainly including two parts: stability control and power consumption information collection. The main control modules all adopt an ARM-based hardware platform. The stability control part is connected to the precise load shedding control system to achieve emergency and sub-emergency precise load shedding. The power consumption information collection part collects the data of the exchange board through the serial port, and calculates the exchange data and converts it into a network port message to send it to the main station to realize data collection and user power usage management functions. Through the refined regulation of the load, the regulation of non-productive loads and non-important loads replaces the total power outage of the region, thereby improving the reliability of power grid operation and power supply.

[0048] This invention only considers the fine-grained regulation of the load by the load control terminal based on the control instructions issued by the load control system, and does not consider the regulation method based on price signals, incentive guidance signals, and transaction information. Specifically, it does not consider the electricity price, project validity period and other information for price-type responses, and the participation conditions, incentive standards, advance notice time, response validity judgment principles, project validity period and other information for incentive-type responses. The regulation method is relatively simple.

[0049] For another example, patent CN112488531A discloses a real-time control method and device for heterogeneous flexible loads based on deep reinforcement learning. First, for different types of heterogeneous flexible loads, a single flexible load model is established respectively, and then an aggregated load model is established for multiple flexible loads with different parameters, so as to obtain the Markov decision process when heterogeneous flexible loads participate in demand response, and the decision function of the aggregate is trained based on historical data through the machine learning framework of deep reinforcement learning to obtain a real-time optimization control decision model for the heterogeneous flexible load aggregate, thereby obtaining the optimal strategy for real-time control of the aggregated load, and improving the flexible load response capability on the user side. The technical problem that the existing load control method has low responsiveness of the flexible load on the user side and is difficult to stimulate the demand response potential on the user side is solved.

[0050] The deep learning method of this invention stores the collected samples in the experience pool through the experience replay mechanism, and uniformly randomly extracts samples from it during training. However, random sampling lacks the distinction of sample importance, and it is difficult for the model to quickly focus on samples that are more critical to performance improvement during training, resulting in slow convergence.

[0051] In order to improve the above problems, the present invention provides a method and device for real-time control of adjustable load resources, including: taking the observed state of the integrated energy system as the input of a pre-trained deep reinforcement learning model, obtaining the real-time control strategy of the adjustable load resources output by the pre-trained deep reinforcement learning model; using the real-time control strategy of the adjustable load resources to control the adjustable load resources of the integrated energy system in real time; wherein the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, gas boilers, the pre-trained deep reinforcement learning model is trained using the DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of Markov decision processes established by the real-time control problem of adjustable load resources. The technical solution provided by the present invention can improve the utilization rate of renewable energy by coordinating and optimizing multiple adjustable load resources, and the deep learning model can quickly adjust the scheduling strategy to ensure the stable operation and power supply safety of the adjustable load resources, while reducing the carbon emissions of the adjustable load resources, which is helpful to achieve the goal of regional energy conservation and emission reduction.

[0052] The above scheme is described in detail below.

[0053] Example 1 See attached Figure 1 , Figure 1FIG. 1 is a flow chart showing the main steps of a method for real-time control of adjustable load resources according to an embodiment of the present invention. Figure 1 As shown, the method for real-time control of adjustable load resources in the embodiment of the present invention mainly includes the following steps: Step S101: using the observed state of the integrated energy system as the input of a pre-trained deep reinforcement learning model to obtain an adjustable load resource real-time control strategy output by the pre-trained deep reinforcement learning model; Step S102: using the real-time control strategy for adjustable load resources to control the adjustable load resources of the integrated energy system in real time; Among them, the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, and gas boilers. The pre-trained deep reinforcement learning model is trained using the DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of a Markov decision process established for the real-time control problem of adjustable load resources.

[0054] In this embodiment, the mathematical model corresponding to the Markov decision process is: (S t ,A t ,S t+1 ,R t ), where S t is the observed state of the integrated energy system at time t, A t is the action variable of the adjustable load resource in the integrated energy system at time t, S t+1 is the observed state of the integrated energy system at time t+1, R t is the reward function at time t.

[0055] In a specific embodiment, Figure 2 As shown, at a certain time t, the environment provides the agent with the observed running state S t , select action A in action space A t Interacting with the environment, the agent is based on the strategy π and the system operation state S t Generate action A t And get reward R t , and migrate to the new state S t+1 , the agent according to R t Gradually learn and improve strategy π.

[0056] In one embodiment, the observed state of the integrated energy system at time t is as follows: S t =S ta ×S tb Sta =[P t ,H t ] S tb =[D Et ,D Ht ] In the above formula, S ta is the controllable observable quantity, S tb is the uncontrollable observable quantity, P t is the electrical load state at time t, H t is the heat load state at time t, D Et is the electricity price at time t, D Ht is the hot price at time t.

[0057] In one embodiment, the action variables of the adjustable load resources in the integrated energy system at time t are as follows: A t =A Et ×A Ht In the above formula, A Et is the control action of the electric load, A Ht Control action for heat load.

[0058] In one embodiment, the state transition function used by the Markov decision process is an MDP random transition probability function.

[0059] In one embodiment, the reward function at time t is as follows: R t =-(C omt +C gridt +C cart )×w In the above formula, C omt is the operation and maintenance cost at time t, C gridt is the electricity purchase and sales cost at time t, C cart is the step-wise carbon trading cost at time t, and w is the conversion coefficient.

[0060] In one embodiment, the operation and maintenance cost at time t is as follows: C omt =∑ρ n P nt The electricity purchase and sale costs at time t are as follows: C gridt =ρ buyt P buyt +ρ salet P salet In the above formula, ρ n is the unit price of the operating cost of the nth unit, P ntis the power of the nth unit at time t, ρ buyt is the electricity purchase price at time t, P buyt is the power purchased from the public grid at time t, ρ salet is the electricity price at time t, P salet is the power sold to the public grid at time t.

[0061] In one embodiment, in order to reduce system carbon emissions, the present invention adopts a stepped carbon trading mechanism. The stepped carbon trading mechanism divides the carbon emission indicators of different industries or enterprises into different levels. For enterprises with high emissions, they need to purchase more emission rights, while for enterprises with low emissions, the excess emission indicators can be sold to enterprises in need. Through this mechanism, high-carbon emission enterprises need to pay higher costs, prompting them to reduce emissions, which is conducive to promoting enterprises to shift to a low-carbon and environmentally friendly development model. At the same time, low-emission enterprises can reduce operating costs, improve competitiveness, and gain an advantage in the market. Thereby achieving the goal of controlling the total amount of carbon emissions at a macro level and slowing down the rate of global greenhouse gas emissions. Therefore, the stepped carbon trading cost at time t is as follows: In the above formula, E MG is the system carbon emissions, E MG =E c -E q , E c is the actual carbon emissions, E q is the carbon emission quota, η is the carbon price, is the length of the carbon emission interval, and σ is the price growth rate.

[0062] In one embodiment, the electrical load includes: wind power output, photovoltaic output, gas turbine output electrical power, power of heat-to-electricity conversion of organic Rankine cycle low-temperature waste heat power generation equipment, heat pump output electrical power, and charging and discharging power of power storage device; the thermal load includes: waste heat boiler output thermal power, gas boiler output thermal power, and charging and discharging power of heat storage device.

[0063] In one embodiment, the electrical load satisfies: P buyt -P salet +P WTt + P PVt + P EGTt + P ORCt -P HPt -P ESCt + P ESDt =P Et The heat load meets: P WHBt +P GBt +PHPt COP HPt -P HSCt + P HSDt =P Ht In the above formula, P buyt is the power purchased from the public grid at time t, P salet is the power sold to the public grid at time t, P WTt is the wind power output at time t, P PVt is the photovoltaic output at time t, P EGTt is the gas turbine output power at time t, P ORCt is the power of the organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity at time t, P HPt is the output power of the heat pump at time t, P ESCt is the charging power of the power storage device at time t, P ESDt is the discharge power of the storage device at time t, P Et is the total electrical power of the system at time t, P WHBt is the heat output power of the waste heat boiler at time t, P GBt is the thermal power output of the gas boiler at time t, COP HPt is the heat pump heating performance coefficient, P HSCt is the charging power of the heat storage device at time t, P HSDt is the heat release power of the heat storage device at time t, P Ht is the total thermal power of the system at time t.

[0064] In addition, the electric thermal load must also meet the power constraints of the interconnection lines interacting with the power grid, the operation constraints of the electric thermal energy storage equipment, the constraints of each unit, etc.

[0065] Reinforcement learning requires that the intelligent agent, when facing a complex and dynamically changing environment, focuses on the current state and action, and does not dwell too much on historical experience or future uncertainties, and then seeks long-term maximization of cumulative rewards through iteration, that is, the process of finding the optimal strategy. The strategy is essentially the conditional probability distribution of each action under the current state. The DQN algorithm is an offline learning method, and the experience replay mechanism is a key mechanism for offline learning. By storing historical experience data, the intelligent agent can interact with the environment more effectively and use experience to update strategies and network parameters multiple times during learning. The experience replay mechanism of the traditional DQN algorithm is random sampling. The present invention improves the traditional DQN and preferentially samples the data in the experience replay pool. In the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, when there are samples in the experience replay pool, compared with the pre-prepared training sample data, the samples in the experience replay pool are preferentially collected as training samples.

[0066] In one embodiment, the sampling probability of sample i in the experience replay pool is as follows: p(i)=p i / P In the above formula, p(i) is the sampling probability of sample i in the experience replay pool, p i is the priority of sample i in the experience replay pool, and P is the sum of the priorities of all samples in the experience replay pool.

[0067] In one embodiment, in the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, the loss function is as follows: L = W × [y t -Q] / 2 In the above formula, L is the loss function value, W is the sum of the sampling weights of all samples in the experience replay pool, Q is the action value function estimated by the policy network, and y t is the target value at time t.

[0068] In one embodiment, the sampling weight w of sample i in the experience replay pool is i as follows: w i =1 / [ p(i) / p min ] α In the above formula, p min is the minimum sampling probability of samples in the experience replay pool, and α is the number of priority settings for samples in the experience replay pool.

[0069] In one embodiment, the target value at time t is as follows: In the above formula, R t is the reward function at time t, β is the discount factor, and max{Q} is the action value function estimated using the target network.

[0070] In a specific implementation, Python 3.8 is used to develop a customized reinforcement learning environment through the gym package, and the deep learning network is built and executed using the Pycharm 1.3 framework.

[0071] Example 2 Based on the same inventive concept, the present invention also provides an adjustable load resource real-time control device, the adjustable load resource real-time control device comprising: An analysis module, used to use the observed state of the integrated energy system as an input of a pre-trained deep reinforcement learning model to obtain an adjustable load resource real-time control strategy output by the pre-trained deep reinforcement learning model; A control module, used to use the real-time control strategy of the adjustable load resources to control the adjustable load resources of the integrated energy system in real time; Among them, the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, and gas boilers. The pre-trained deep reinforcement learning model is trained using the DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of a Markov decision process established for the real-time control problem of adjustable load resources.

[0072] Preferably, the mathematical model corresponding to the Markov decision process is: (S t ,A t ,S t+1 ,R t ), where S t is the observed state of the integrated energy system at time t, A t is the action variable of the adjustable load resource in the integrated energy system at time t, S t+1 is the observed state of the integrated energy system at time t+1, R t is the reward function at time t.

[0073] Furthermore, the observed state of the integrated energy system at time t is as follows: S t =S ta ×S tb S ta =[P t ,H t ] S tb =[D Et ,D Ht ] In the above formula, S ta is the controllable observable quantity, S tb is the uncontrollable observable quantity, P t is the electrical load state at time t, H t is the heat load state at time t, D Et is the electricity price at time t, D Ht is the hot price at time t.

[0074] Furthermore, the action variables of the adjustable load resources in the integrated energy system at time t are as follows: A t =A Et ×A Ht In the above formula, A Et is the control action of the electric load, A HtControl action for heat load.

[0075] Furthermore, the state transfer function adopted by the Markov decision process is an MDP random transfer probability function.

[0076] Furthermore, the reward function at time t is as follows: R t =-(C omt +C gridt +C cart )×w In the above formula, C omt is the operation and maintenance cost at time t, C gridt is the electricity purchase and sales cost at time t, C cart is the step-wise carbon trading cost at time t, and w is the conversion coefficient.

[0077] Furthermore, the operation and maintenance cost at time t is as follows: C omt =∑ρ n P nt The electricity purchase and sale costs at time t are as follows: C gridt =ρ buyt P buyt +ρ salet P salet In the above formula, ρ n is the unit price of the operating cost of the nth unit, P nt is the power of the nth unit at time t, ρ buyt is the electricity purchase price at time t, P buyt is the power purchased from the public grid at time t, ρ salet is the electricity price at time t, P salet is the power sold to the public grid at time t.

[0078] Furthermore, the tiered carbon trading cost at time t is as follows: In the above formula, E MG is the system carbon emissions, E MG =E c -E q , E c is the actual carbon emissions, E q is the carbon emission quota, η is the carbon price, is the length of the carbon emission interval, and σ is the price growth rate.

[0079] Furthermore, the electrical load includes: wind power output, photovoltaic output, gas turbine output electrical power, power of heat-to-electricity conversion of organic Rankine cycle low-temperature waste heat power generation equipment, heat pump output electrical power, and charging and discharging power of power storage device; the thermal load includes: waste heat boiler output thermal power, gas boiler output thermal power, and charging and discharging power of heat storage device.

[0080] Furthermore, the electrical load satisfies: P buyt -P salet +P WTt + P PVt + P EGTt + P ORCt -P HPt -P ESCt + P ESDt =P Et The heat load meets: P WHBt +P GBt +P HPt COP HPt -P HSCt + P HSDt =P Ht In the above formula, P buyt is the power purchased from the public grid at time t, P salet is the power sold to the public grid at time t, P WTt is the wind power output at time t, P PVt is the photovoltaic output at time t, P EGTt is the gas turbine output power at time t, P ORCt is the power of the organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity at time t, P HPt is the output power of the heat pump at time t, P ESCt is the charging power of the power storage device at time t, P ESDt is the discharge power of the storage device at time t, P Et is the total electrical power of the system at time t, P WHBt is the heat output power of the waste heat boiler at time t, P GBt is the thermal power output of the gas boiler at time t, COP HPt is the heat pump heating performance coefficient, P HSCt is the charging power of the heat storage device at time t, P HSDt is the heat release power of the heat storage device at time t, P Ht is the total thermal power of the system at time t.

[0081] Preferably, in the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, when there are samples in the experience replay pool, samples in the experience replay pool are preferentially collected as training samples compared to pre-prepared training sample data.

[0082] Furthermore, the sampling probability of sample i in the experience replay pool is as follows: p(i)=p i / P In the above formula, p(i) is the sampling probability of sample i in the experience replay pool, p i is the priority of sample i in the experience replay pool, and P is the sum of the priorities of all samples in the experience replay pool.

[0083] Furthermore, in the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, the loss function is as follows: L = W × [y t -Q] / 2 In the above formula, L is the loss function value, W is the sum of the sampling weights of all samples in the experience replay pool, Q is the action value function estimated by the policy network, and y t is the target value at time t.

[0084] Furthermore, the sampling weight w of sample i in the experience replay pool i as follows: w i =1 / [ p(i) / p min ] α In the above formula, p min is the minimum sampling probability of samples in the experience replay pool, and α is the number of priority settings for samples in the experience replay pool.

[0085] Furthermore, the target value at time t is as follows: In the above formula, R t is the reward function at time t, β is the discount factor, and max{Q} is the action value function estimated using the target network.

[0086] Example 3 Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a method for real-time control of adjustable load resources in the above embodiment.

[0087] Example 4 Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of a method for real-time control of adjustable load resources in the above embodiment.

[0088] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for real-time control of adjustable load resources, characterized in that: The method comprises: Taking the observed state of the integrated energy system as the input of a pre-trained deep reinforcement learning model, and obtaining an adjustable load resource real-time control strategy output by the pre-trained deep reinforcement learning model; Using the real-time control strategy for adjustable load resources to control the adjustable load resources of the integrated energy system in real time; Among them, the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, and gas boilers. The pre-trained deep reinforcement learning model is trained using the DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of a Markov decision process established for the real-time control problem of adjustable load resources.

2. The method according to claim 1, characterized in that The mathematical model corresponding to the Markov decision process is: (S t ,A t ,S t+1 ,R t ), where S t is the observed state of the integrated energy system at time t, A t is the action variable of the adjustable load resource in the integrated energy system at time t, S t+1 is the observed state of the integrated energy system at time t+1, R t is the reward function at time t.

3. The method according to claim 2, characterized in that The observed state of the comprehensive energy system at time t is as follows: S t =S ta ×S tb S ta =[P t ,H t ] S tb =[D Et ,D Ht ] In the above formula, S ta is the controllable observable quantity, S tb is the uncontrollable observable quantity, P t is the electrical load state at time t, H t is the heat load state at time t, D Et is the electricity price at time t, D Ht is the hot price at time t.

4. The method according to claim 2, characterized in that The action variables of the adjustable load resources in the integrated energy system at time t are as follows: A t =A Et ×A Ht In the above formula, A Et is the control action of the electric load, A Ht It is the control action of heat load.

5. The method according to claim 2, characterized in that The state transfer function adopted by the Markov decision process is the MDP random transfer probability function.

6. The method according to claim 2, characterized in that The reward function at time t is as follows: R t =-(C omt +C gridt +C cart )×w In the above formula, C omt is the operation and maintenance cost at time t, C gridt is the electricity purchase and sales cost at time t, C cart is the step-wise carbon trading cost at time t, and w is the conversion coefficient.

7. The method according to claim 6, characterized in that The operation and maintenance costs at time t are as follows: C omt =∑ρ n P nt The electricity purchase and sale costs at time t are as follows: C gridt =p buyt P buyt +r salet P salet In the above formula, ρ n is the unit price of the operating cost of the nth unit, P nt is the power of the nth unit at time t, ρ buyt is the electricity purchase price at time t, P buyt is the power purchased from the public grid at time t, ρ salet is the electricity price at time t, P salet is the power sold to the public grid at time t.

8. The method according to claim 6, characterized in that The tiered carbon trading cost at time t is as follows: In the above formula, E MG is the system carbon emissions, E MG =E c -E q , E c is the actual carbon emissions, E q is the carbon emission quota, η is the carbon price, is the length of the carbon emission interval, and σ is the price growth rate.

9. The method according to claim 3 or 4, characterized in that The electrical load includes: wind power output, photovoltaic output, gas turbine output electrical power, power of organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity, heat pump output electrical power, and power storage device charging and discharging power; the thermal load includes: waste heat boiler output thermal power, gas boiler output thermal power, and power storage device charging and discharging thermal power.

10. The method according to claim 9, characterized in that The electrical load satisfies: P buyt -P salet +P WTt + P PVt + P EGTt + P ORCt - P HPt - P ESCt + P ESDt = P Et The heat load meets: P WHBt +P GBt +P HPt COP HPt -P HSCt + P HSDt =P Ht In the above formula, P buyt is the power purchased from the public grid at time t, P salet is the power sold to the public grid at time t, P WTt is the wind power output at time t, P PVt is the photovoltaic output at time t, P EGTt is the gas turbine output power at time t, P ORCt is the power of the organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity at time t, P HPt is the output power of the heat pump at time t, P ESCt is the charging power of the power storage device at time t, P ESDt is the discharge power of the power storage device at time t, P Et is the total electrical power of the system at time t, P WHBt is the heat output power of the waste heat boiler at time t, P GBt is the thermal power output of the gas boiler at time t, COP HPt is the heat pump heating performance coefficient, P HSCt is the charging power of the heat storage device at time t, P HSDt is the heat release power of the heat storage device at time t, P Ht is the total thermal power of the system at time t.

11. The method according to claim 1, characterized in that In the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, when there are samples in the experience replay pool, samples in the experience replay pool are preferentially collected as training samples compared to pre-prepared training sample data.

12. The method according to claim 11, characterized in that The sampling probability of sample i in the experience replay pool is as follows: p(i)=p i / P In the above formula, p(i) is the sampling probability of sample i in the experience replay pool, p i is the priority of sample i in the experience replay pool, and P is the sum of the priorities of all samples in the experience replay pool.

13. The method according to claim 12, characterized in that When training the pre-trained deep reinforcement learning model using the DQN algorithm, the loss function is as follows: L=W×[y t -Q] / 2 In the above formula, L is the loss function value, W is the sum of the sampling weights of all samples in the experience replay pool, Q is the action value function estimated by the policy network, and y t is the target value at time t.

14. The method according to claim 13, characterized in that The sampling weight w of sample i in the experience replay pool i as follows: w i =1 / [ p(i) / p min ] α In the above formula, p min is the minimum sampling probability of samples in the experience replay pool, and α is the number of priority settings for samples in the experience replay pool.

15. The method according to claim 14, characterized in that The target value at time t is as follows: In the above formula, R t is the reward function at time t, β is the discount factor, and max{Q} is the action value function estimated using the target network.

16. A real-time control device for adjustable load resources, characterized in that: The device comprises: An analysis module, used to use the observed state of the integrated energy system as an input of a pre-trained deep reinforcement learning model to obtain an adjustable load resource real-time control strategy output by the pre-trained deep reinforcement learning model; A control module, used to use the real-time control strategy of the adjustable load resources to control the adjustable load resources of the integrated energy system in real time; Among them, the adjustable load resources include at least one of the following: wind turbines, photovoltaic turbines, organic Rankine cycle low-temperature waste heat power generation equipment, gas turbines, heat pumps, power storage devices, heat storage devices, waste heat boilers, and gas boilers. The pre-trained deep reinforcement learning model is trained using the DQN algorithm, and the training samples of the pre-trained deep reinforcement learning model are composed of a Markov decision process established for the real-time control problem of adjustable load resources.

17. The device according to claim 16, characterized in that The mathematical model corresponding to the Markov decision process is: (S t ,A t ,S t+1 ,R t ), where S t is the observed state of the integrated energy system at time t, A t is the action variable of the adjustable load resource in the integrated energy system at time t, S t+1 is the observed state of the integrated energy system at time t+1, R t is the reward function at time t.

18. The device according to claim 17, characterized in that The observed state of the comprehensive energy system at time t is as follows: S t =S ta ×S tb S ta =[P t ,H t ] S tb =[D Et ,D Ht ] In the above formula, S ta is the controllable observable quantity, S tb is the uncontrollable observable quantity, P t is the electrical load state at time t, H t is the heat load state at time t, D Et is the electricity price at time t, D Ht is the hot price at time t.

19. The device according to claim 17, characterized in that The action variables of the adjustable load resources in the integrated energy system at time t are as follows: A t =A Et ×A Ht In the above formula, A Et is the control action of the electric load, A Ht It is the control action of heat load.

20. The device according to claim 17, characterized in that The state transfer function adopted by the Markov decision process is the MDP random transfer probability function.

21. The device according to claim 17, characterized in that The reward function at time t is as follows: R t =-(C omt +C gridt +C cart )×w In the above formula, C omt is the operation and maintenance cost at time t, C gridt is the electricity purchase and sales cost at time t, C cart is the step-wise carbon trading cost at time t, and w is the conversion coefficient.

22. The device according to claim 21, characterized in that The operation and maintenance costs at time t are as follows: C omt =∑ρ n P nt The electricity purchase and sale costs at time t are as follows: C gridt =p buyt P buyt +r salet P salet In the above formula, ρ n is the unit price of the operating cost of the nth unit, P nt is the power of the nth unit at time t, ρ buyt is the electricity purchase price at time t, P buyt is the power purchased from the public grid at time t, ρ salet is the electricity price at time t, P salet is the power sold to the public grid at time t.

23. The device according to claim 21, characterized in that The tiered carbon trading cost at time t is as follows: In the above formula, E MG is the system carbon emissions, E MG =E c -E q , E c is the actual carbon emissions, E q is the carbon emission quota, η is the carbon price, is the length of the carbon emission interval, and σ is the price growth rate.

24. The device according to claim 18 or 19, characterized in that The electrical load includes: wind power output, photovoltaic output, gas turbine output electrical power, power of organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity, heat pump output electrical power, and power storage device charging and discharging power; the thermal load includes: waste heat boiler output thermal power, gas boiler output thermal power, and power storage device charging and discharging thermal power.

25. The device according to claim 24, characterized in that The electrical load satisfies: P buyt -P salet +P WTt + P PVt + P EGTt + P ORCt - P HPt - P ESCt + P ESDt = P Et The heat load meets: P WHBt +P GBt +P HPt COP HPt -P HSCt + P HSDt =P Ht In the above formula, P buyt is the power purchased from the public grid at time t, P salet is the power sold to the public grid at time t, P WTt is the wind power output at time t, P PVt is the photovoltaic output at time t, P EGTt is the gas turbine output power at time t, P ORCt is the power of the organic Rankine cycle low-temperature waste heat power generation equipment to convert heat into electricity at time t, P HPt is the output power of the heat pump at time t, P ESCt is the charging power of the power storage device at time t, P ESDt is the discharge power of the power storage device at time t, P Et is the total electrical power of the system at time t, P WHBt is the heat output power of the waste heat boiler at time t, P GBt is the thermal power output of the gas boiler at time t, COP HPt is the heat pump heating performance coefficient, P HSCt is the charging power of the heat storage device at time t, P HSDt is the heat release power of the heat storage device at time t, P Ht is the total thermal power of the system at time t.

26. The device according to claim 16, characterized in that In the process of training the pre-trained deep reinforcement learning model using the DQN algorithm, when there are samples in the experience replay pool, samples in the experience replay pool are preferentially collected as training samples compared to pre-prepared training sample data.

27. The device according to claim 26, characterized in that The sampling probability of sample i in the experience replay pool is as follows: p(i)=p i / P In the above formula, p(i) is the sampling probability of sample i in the experience replay pool, p i is the priority of sample i in the experience replay pool, and P is the sum of the priorities of all samples in the experience replay pool.

28. The device according to claim 27, characterized in that When training the pre-trained deep reinforcement learning model using the DQN algorithm, the loss function is as follows: L=W×[y t -Q] / 2 In the above formula, L is the loss function value, W is the sum of the sampling weights of all samples in the experience replay pool, Q is the action value function estimated by the policy network, and y t is the target value at time t.

29. The device according to claim 28, characterized in that The sampling weight w of sample i in the experience replay pool i as follows: w i =1 / [ p(i) / p min ] α In the above formula, p min is the minimum sampling probability of samples in the experience replay pool, and α is the number of priority settings for samples in the experience replay pool.

30. The device according to claim 29, characterized in that The target value at time t is as follows: In the above formula, R t is the reward function at time t, β is the discount factor, and max{Q} is the action value function estimated using the target network.

31. A computer device, characterized in that: include: one or more processors; The processor is configured to execute one or more programs; When the one or more programs are executed by the one or more processors, the adjustable load resource real-time control method as described in any one of claims 1 to 15 is implemented.

32. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the real-time control method for adjustable load resources as described in any one of claims 1 to 15 is implemented.

Images