Comprehensive energy system optimization scheduling method based on knowledge-guided reinforcement learning

Abstract

The invention relates to an integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning, and is suitable for the field of operation regulation and control of an integrated energy system. The method comprises the following steps: setting a state variable, an action variable and a reward function of reinforcement learning; acquiring a system operation state vector, and determining a control action vector corresponding to the state vector based on the strategy network; performing border crossing judgment on the control action vector; if the control action vector is judged to be border crossing, the control action influencing the operation cost is corrected, and a corrected control action vector meeting border crossing judgment conditions is obtained; based on the corrected control action vector, obtaining the reward result of the current round and the running state vector of the next round; storing a tetrad consisting of the running state of the current round, the control action, the reward result and the running state vector of the next round into an experience playback pool; and updating the policy network, the value network and the old policy network based on the data in the experience playback pool, and repeating the steps until the algorithm converges.

Description

Technical Field

[0001] This invention relates to a method for optimizing the scheduling of integrated energy systems based on knowledge-guided reinforcement learning. It is applicable to the field of integrated energy system operation and control. Background Technology

[0002] An integrated energy system is a new type of energy system that integrates multiple energy forms such as electricity, heat, and cooling. It enables the coordinated optimization and efficient complementarity of various energy flows, thereby efficiently absorbing new energy sources such as wind and solar power while meeting the diverse energy needs of users within the region and improving overall energy utilization efficiency. Conducting research on the optimized operation of integrated energy systems is of great significance for improving the overall system performance and the absorption rate of new energy sources, and can effectively promote the practical application and promotion of integrated energy systems.

[0003] Reinforcement learning is a mainstream approach in the field of integrated energy system optimization scheduling. This method establishes a reinforcement learning controller that outputs control actions, interacts with the actual environment to receive rewards, and autonomously adjusts its own controller parameters to gradually master the optimal system operation rules. For example, a deep reinforcement learning-based integrated energy system optimization scheduling method (Chinese invention patent, publication number CN117455183A) uses the TD3 algorithm to implement reinforcement learning-based optimization scheduling of integrated energy systems; a deep reinforcement learning-based integrated energy system planning optimization method and system (Chinese invention patent, publication number CN119444502A) uses the DDPG algorithm to construct the reinforcement learning framework for integrated energy systems; and a deep reinforcement learning-based integrated energy system economic scheduling model method (Chinese invention patent, publication number CN119273066A) uses the PPO algorithm to establish a reinforcement learning-based scheduling method for integrated energy systems.

[0004] However, because reinforcement learning algorithms require extensive exploration of the control action space during the initial training phase, this stage may output a large number of control actions with random characteristics. This can lead to problems such as insufficient end-point load supply and equipment exceeding operating limits in practical applications of existing reinforcement learning-based integrated energy system optimization scheduling methods, seriously affecting the safety and stability of the system. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a comprehensive energy system optimization scheduling method based on knowledge-guided reinforcement learning, which addresses the problems mentioned above.

[0006] The technical solution adopted in this invention is: a comprehensive energy system optimization scheduling method based on knowledge-guided reinforcement learning, comprising: S100. Based on the integrated energy system structure, set the state variables for reinforcement learning. Action variables and reward function The reward function Based on the total system operating cost; S200, Obtain the current operating state vector of the system. Determining the state vector based on the policy network Corresponding control action vector ; S300, Control Action Vector The boundary judgment is carried out. The boundary judgment conditions include whether the system power supply and demand are balanced, whether the system heat supply and demand are balanced, and whether the battery charge state is out of bounds. S400, If the control motion vector If the control action is judged to be out of bounds, then the control action that affects operating costs is corrected to obtain a corrected control action vector that satisfies the out-of-bounds judgment condition. Otherwise, let ; S500, based on the modified control motion vector The results of this round of rewards were obtained. and the running state vector for the next round. ; S600, Current Operating Status Controlling actions Award Results and the running state vector for the next round. The resulting quadruple is saved to the experience replay pool; S700: Every preset number of rounds, based on the data in the experience replay pool, update the policy network, value network, and old policy network, return to step S200, until the algorithm converges and the trained policy network is obtained.

[0007] The state variable Including electricity load demand Heat load demand Photovoltaic power generation output Wind power output Battery state of charge Electricity sales price from the power grid and natural gas prices ;

[0008] The action variable Including fuel input for combined heat and power units Fuel input of gas boiler Input electrical power of electric heat pump The charging and discharging power of the storage battery Electricity purchased from the power grid .

[0009] The reward function It is the opposite of the total operating cost of the system.

[0010] If the control action vector If the control action is judged to be out of bounds, then the control action that affects operating costs is corrected to obtain a corrected control action vector that satisfies the out-of-bounds judgment condition. ,include: Based on the relationship between the current electricity load demand and the output of photovoltaic power generation and wind power generation, as well as the relationship between the current heat load demand and the heat supply of gas boilers, we will determine whether to start the combined heat and power unit. If the combined heat and power unit does not need to be turned on, the electrical load is provided by wind power and photovoltaic power, the heat load is provided by gas boilers, and the battery stores the excess electrical energy. If it needs to be turned on, the combined heat and power (CHP) unit will provide the heat load, and the battery and grid will provide the remaining electrical load; or, the CHP unit will provide the remaining electrical load, and the gas boiler and electric heat pump will provide the heat load.

[0011] Compare the costs required for a combined heat and power (CHP) unit to uniformly provide heat load versus a CHP unit to uniformly provide surplus electrical load; if the cost of providing heat load is lower, then the CHP unit should uniformly provide heat load; if the cost of providing surplus electrical load is lower, then the CHP unit should uniformly provide surplus electrical load.

[0012] The experience replay pool has a capacity limit. If the experience replay pool is full, the oldest set of data is removed and the latest quadruple is added.

[0013] The updating of the policy network, value network, and old policy network includes: The parameter update method for the policy network is shown in the following equation: ; ; In the formula, For the sample size, For policy network parameters, and These represent the states in the policy network and the old policy network, respectively. Taking action probability distribution The advantage function is calculated using generalized advantage estimation. This is a truncation function. To truncate the threshold, is the learning rate of the policy network.

[0014] A comprehensive energy system optimization scheduling device based on knowledge-guided reinforcement learning, comprising: The first module is used to set the state variables for reinforcement learning based on the integrated energy system structure. Action variables and reward function The reward function Based on the total system operating cost; The second module is used to obtain the current operating state vector of the system. Determining the state vector based on the policy network Corresponding control action vector ; The third module is used for controlling the action vector. The boundary judgment is carried out. The boundary judgment conditions include whether the system power supply and demand are balanced, whether the system heat supply and demand are balanced, and whether the battery charge state is out of bounds. The fourth module is used to control the action vector. If the control action is judged to be out of bounds, then the control action that affects operating costs is corrected to obtain a corrected control action vector that satisfies the out-of-bounds judgment condition. Otherwise, let ; The fifth module is used to base the modified control action vector. The results of this round of rewards were obtained. and the running state vector for the next round. ; The sixth module is used to record the running status of this round. Controlling actions Award Results and the running state vector for the next round. The resulting quadruple is saved to the experience replay pool; The seventh module is used to update the policy network, value network, and old policy network every preset number of rounds based on the data in the experience replay pool, and then return to the second module until the algorithm converges and the trained policy network is obtained.

[0015] A storage medium storing a computer program executable by a processor, wherein the computer program, when executed, implements the steps of the knowledge-guided reinforcement learning-based integrated energy system optimization scheduling method.

[0016] An integrated energy system optimization scheduling device has a memory and a processor, storing a computer program that can be executed by the processor. When the computer program is executed, it implements the steps of the integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning.

[0017] The beneficial effects of this invention are: this invention improves the control action vector By performing boundary violation judgments and making safety assessments and autonomous corrections on the control actions output by the reinforcement learning controller based on the boundary violation judgment conditions, it is possible to effectively ensure that the control actions output during the reinforcement learning training process always meet the requirements for safe operation, thereby effectively improving the safety and reliability of reinforcement learning methods in practical integrated energy systems. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the integrated energy system in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of the knowledge-guided reinforcement learning-based optimized scheduling method in an embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0021] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0022] Example 1: This example is a comprehensive energy system optimization scheduling method based on knowledge-guided reinforcement learning, specifically including the following steps:

[0023] S100. Based on the integrated energy system structure, set the state variables for reinforcement learning. Action variables and reward function Furthermore, a reinforcement learning framework is constructed using a proximal policy optimization algorithm, which initializes the policy network, value network, old policy network, and experience replay pool.

[0024] In this embodiment, state variables Including electricity load demand Heat load demand Photovoltaic power generation output Wind power output Battery state of charge Electricity sales price from the power grid and natural gas prices As shown in the following formula:

[0025]

[0026] The search space for each state variable is shown in Table 1 below: Table 1 Search space for each state variable in this embodiment

[0027] Action variables Including fuel input for combined heat and power units Fuel input of gas boiler The input electrical power of a heat pump The charging and discharging power of the storage battery Electricity purchased from the power grid As shown in the following formula: .

[0028] The search space for each action variable is shown in Table 2 below: Table 2 Search space for each action variable in this embodiment

[0029] The Z-score standardization method is used to standardize the state variables and action variables.

[0030] reward function This is the negative of the total system operating cost, i.e., minimizing the total system operating cost, as shown in the following formula: ; Subsequently, a reinforcement learning framework is constructed using the proximal policy optimization algorithm, and the policy network, value network, old policy network, and experience replay pool are initialized. In this embodiment, the capacity of the experience replay pool is set to 50,000, and standard random sampling is used for sample extraction.

[0031] The policy network, value network, and old policy network are all constructed using deep neural networks. The policy network and old policy network take a vector of state variables as input and output a vector of the mean and standard deviation of action variables. The value network takes a vector of state variables as input and outputs estimated values ​​for different states, representing an estimate of the cumulative reward that can be obtained in the future from the current state. The structural parameters of the policy network, value network, and old policy network are shown in Table 3 below. Table 3 PPO Algorithm Network Structure Parameter Settings

[0032] S200, Obtain the current operating state vector of the system. Determining the state vector based on the policy network Corresponding control action vector .

[0033] In this embodiment, the control action vector Internal control actions The following formula is used to calculate: ; In the formula, and These are the mean and standard deviation of the Gaussian probability distribution of the control actions output by the policy network, respectively. (The sentence is incomplete and requires further context.) The input policy is obtained through network computation. The sampling noise is set to 0.15 in this embodiment to improve the exploratory nature of the optimal control action.

[0034] S300, employs expert knowledge to control motion vectors The boundary crossing judgment is performed, and the boundary crossing judgment conditions include whether the system's power supply and demand are balanced, whether the system's heat supply and demand are balanced, and whether the battery's state of charge has exceeded the limit, as shown in the following formulas: ; ; ; in, The calorific value of natural gas is taken as 48,000 kJ / kg in this embodiment. and These represent the power generation efficiency and heat production efficiency of the combined heat and power (CHP) unit, respectively. In this embodiment, these are taken as 0.35 and 0.55. For the heat pump's heat production efficiency, For the heat production efficiency of gas-fired boilers, For the energy storage efficiency of the battery, The charging and discharging efficiency of the battery. For battery capacity, Tolerance for energy conservation constraints, and These are the lower and upper limits of the state of charge, respectively. To optimize the interval.

[0035] In this embodiment, the above parameters are set as follows: Table 4 System Parameter Settings

[0036] S400, If the control motion vector If the operation is deemed to be out of bounds (the above three conditions are not met simultaneously), then the control actions affecting operating costs are corrected (including whether to start the cogeneration unit, the unit's power supply mode, and the auxiliary equipment's power supply mode), resulting in a corrected control action vector that meets the out-of-bounds judgment conditions. If all three conditions are met, then the control action is deemed to meet the requirements for safe operation. .

[0037] S410. Based on the relationship between the current electricity load demand and the output of photovoltaic power generation and wind power generation, as well as the relationship between the current heat load demand and the heat supply of gas boilers, determine whether to start the cogeneration unit.

[0038] like and If the above two conditions are not met simultaneously, then a combined heat and power (CHP) unit is not required; if the above two conditions are not met simultaneously, then the unit must be turned on.

[0039] S420. If the cogeneration unit does not need to be turned on, the electrical load is provided by wind power and photovoltaic power, the heat load is provided by gas boiler, and the battery stores the excess electrical energy. S430 If it needs to be turned on, the heat load shall be provided by the cogeneration unit and the remaining electrical load shall be provided by the battery and the power grid; or, the remaining electrical load shall be provided by the cogeneration unit and the heat load shall be provided by the gas boiler and the electric heat pump.

[0040] Compare the costs required for a combined heat and power (CHP) unit to uniformly provide heat load versus a CHP unit to uniformly provide surplus electrical load; if the cost of providing heat load is lower, then the CHP unit should uniformly provide heat load; if the cost of providing surplus electrical load is lower, then the CHP unit should uniformly provide surplus electrical load.

[0041] ① Comparison and The size of the two. If the former is larger than the latter, then the heat-driven power generation mode is adopted, that is, the heat load is uniformly provided by the combined heat and power unit.

[0042] If the electricity sales price of the power grid At peak prices, the remaining electricity demand will be supplied primarily by battery storage, and secondarily by the grid; that is, the control action will be adjusted accordingly. ;

[0043] Otherwise, the remaining electrical load demand will be supplied by the power grid, meaning the control action will be corrected to... .

[0044] ② If the latter is greater than the former, then the electricity-driven heating mode is adopted, that is, the electricity load is uniformly provided by the cogeneration unit.

[0045] Furthermore, if the electricity sales price of the power grid... If the price is at its peak, the remaining heat load demand will be supplied uniformly using gas-fired boilers, meaning the control action will be adjusted accordingly. .

[0046] Otherwise, the remaining heat load demand will be supplied by electric heat pumps, with the electricity demand provided by the power grid; that is, the control action will be corrected to... .

[0047] S500, based on the modified control motion vector The results of this round of rewards were obtained. and the running state vector for the next round. .

[0048] S600, Current Operating Status Controlling actions Award Results and the running state vector for the next round. The resulting quadruple is saved to the experience replay pool. If the experience replay pool is full, the oldest set of data is replaced.

[0049] S700: Every preset number of rounds, based on the data in the experience replay pool, update the policy network, value network, and old policy network, return to step S200, until the algorithm converges and the trained policy network is obtained.

[0050] In this embodiment, the update frequency is set to once every 5000 time steps, the number of samples updated in a single update is set to 64, and the number of updates is set to 5. The parameter update method of the policy network is shown in the following formula: ; ; In the formula, For the sample size, For policy network parameters, and These represent the states in the policy network and the old policy network, respectively. Taking action probability distribution The advantage function is calculated using generalized advantage estimation. This is a truncation function. In this embodiment, the truncation threshold is set to 0.1. The learning rate of the policy network is set to 0.0001 in this embodiment.

[0051] After the policy network parameters are updated, the parameters of the current policy network are copied to the old policy network to update the parameters of the old policy network, as shown in the following formula: ; The parameter update method for the value network is shown in the following equation: ; ; In the formula, For value network parameters, These are the value estimates calculated using a value network under different states. The learning rate for the value network is set to 0.0003 in this embodiment. The discount factor is set to 0.98 in this embodiment.

[0052] In this embodiment, when performing optimization scheduling, the optimal control action is calculated by the trained policy network obtained in step S700 based on the actual operating state. Expert knowledge is then used to make boundary judgments and correct actions (using steps S300 and S400) to finally obtain the optimal control action scheme that can ensure the safe operation of the system.

[0053] The following is an example to illustrate this: Four years of data on heat load, electricity load, photovoltaic power output, and wind power output in a specific region were selected to verify the performance of the method of this invention. The data from the first three years were used as training data, and the data from the last year was used as validation data. The time-of-use electricity price and natural gas price in this region are shown in Table 5. A traditional reinforcement learning-based optimization scheduling method was compared in performance. This method uses the standard PPO algorithm to construct a reinforcement learning controller, and the output control action is directly input into the system for regulation without any correction processing. Optimization interval. Set to 1 hour, and each optimization period to 24 hours.

[0054] Table 5 Time-of-use electricity prices and natural gas prices

[0055] Table 6 shows the supply rates and average supply deviations of electrical and thermal loads during the training phase for both methods. The former represents the ratio of scenarios where energy supply exceeds the end load, and the latter represents the average deviation between energy supply and end load. As the table shows, because the traditional method does not perform boundary judgment and correction on the control actions output by the reinforcement learning controller, its electrical and thermal load supply rates during the training phase are all less than 60%, and the average supply deviation is greater than 250kW. This will severely affect the stability and reliability of the system in the early stages of actual deployment. In contrast, the method of this invention uses domain knowledge to perform boundary judgment and autonomous correction on the output control actions, effectively ensuring the reliability of the output actions during the early random exploration phase of the reinforcement learning controller, thereby significantly improving the load supply rate and reducing the supply deviation.

[0056] Table 6 Load supply rate and average supply deviation during the training phase

[0057] The frequency of battery SOC exceeding limits (i.e., the number of times the actual SOC falls below the lower limit or exceeds the upper limit) during the training phase for both methods is shown in Table 7 below. Similar to the aforementioned situations, the method of this invention can effectively avoid overcharging or undercharging of the battery, thereby improving the operational safety of energy storage devices. Compared to traditional methods, the frequency of battery SOC exceeding limits is reduced by 99.9%.

[0058] Table 7 Battery SOC Out-of-Limit Frequency During Training Phase

[0059] Table 8 below compares the annual operating costs of the two methods during the validation phase. As can be seen from Table 8, the method of this invention effectively improves optimization performance by introducing expert knowledge, thus exhibiting better optimization performance in the actual validation phase. Compared to the traditional method, the annual operating cost of the method of this invention is reduced by 5.33%.

[0060] Table 8 Comparison of annual operating costs during the validation phase

[0061] Example 2: This example is a comprehensive energy system optimization scheduling device based on knowledge-guided reinforcement learning, comprising: The first module is used to set the state variables for reinforcement learning based on the integrated energy system structure. Action variables and reward function The reward function Based on the total system operating cost; The second module is used to obtain the current operating state vector of the system. Determining the state vector based on the policy network Corresponding control action vector ; The third module is used for controlling the action vector. The boundary judgment is carried out. The boundary judgment conditions include whether the system power supply and demand are balanced, whether the system heat supply and demand are balanced, and whether the battery charge state is out of bounds. The fourth module is used to control the action vector. If the control action is judged to be out of bounds, then the control action that affects operating costs is corrected to obtain a corrected control action vector that satisfies the out-of-bounds judgment condition. Otherwise, let ; The fifth module is used to base the modified control action vector. The results of this round of rewards were obtained. and the running state vector for the next round. ; The sixth module is used to record the running status of this round. Controlling actions Award Results and the running state vector for the next round. The resulting quadruple is saved to the experience replay pool; The seventh module is used to update the policy network, value network, and old policy network every preset number of rounds based on the data in the experience replay pool, and then return to the second module until the algorithm converges and the trained policy network is obtained.

[0062] Example 3: This example is a storage medium that stores a computer program that can be executed by a processor. When the computer program is executed, it implements the steps of the integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning described in Example 1.

[0063] Example 4: This example is a comprehensive energy system optimization scheduling device, which has a memory and a processor. It stores a computer program that can be executed by the processor. When the computer program is executed, it implements the steps of the comprehensive energy system optimization scheduling method based on knowledge-guided reinforcement learning described in Example 1.

[0064] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0065] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0067] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0068] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0069] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0070] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0071] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A comprehensive energy system optimization scheduling method based on knowledge-guided reinforcement learning, characterized in that, include: S100. Based on the integrated energy system structure, set the state variables for reinforcement learning. Action variables and reward function The reward function Based on the total system operating cost; S200, Obtain the current operating state vector of the system. Determining the state vector based on the policy network Corresponding control action vector ; S300, Control Action Vector The boundary judgment is carried out. The boundary judgment conditions include whether the system power supply and demand are balanced, whether the system heat supply and demand are balanced, and whether the battery charge state is out of bounds. S400, If the control motion vector If the control action is judged to be out of bounds, then the control action that affects operating costs is corrected to obtain a corrected control action vector that satisfies the out-of-bounds judgment condition. Otherwise, let ; S500, based on the modified control motion vector The results of this round of rewards were obtained. and the running state vector for the next round. ; S600, Current Operating Status Controlling actions Award Results and the running state vector for the next round. The resulting quadruple is saved to the experience replay pool; S700: Every preset number of rounds, based on the data in the experience replay pool, update the policy network, value network, and old policy network, return to step S200, until the algorithm converges and the trained policy network is obtained.

2. The integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning according to claim 1, characterized in that, The state variable Including electricity load demand Heat load demand Photovoltaic power generation output Wind power output Battery state of charge Electricity sales price from the power grid and natural gas prices ; The action variable Including fuel input for combined heat and power units Fuel input of gas boiler Input electrical power of electric heat pump The charging and discharging power of the storage battery Electricity purchased from the power grid .

3. The integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning according to claim 1, characterized in that, The reward function It is the opposite of the total operating cost of the system.

4. The integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning according to claim 2, characterized in that, If the control action vector If the control action is judged to be out of bounds, then the control action that affects operating costs is corrected to obtain a corrected control action vector that satisfies the out-of-bounds judgment condition. ,include: Based on the relationship between the current electricity load demand and the output of photovoltaic power generation and wind power generation, as well as the relationship between the current heat load demand and the heat supply of gas boilers, we will determine whether to start the combined heat and power unit. If the combined heat and power unit does not need to be turned on, the electrical load is provided by wind power and photovoltaic power, the heat load is provided by gas boilers, and the battery stores the excess electrical energy. If it needs to be turned on, the combined heat and power (CHP) unit will provide the heat load, and the battery and grid will provide the remaining electrical load; or, the CHP unit will provide the remaining electrical load, and the gas boiler and electric heat pump will provide the heat load.

5. The integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning according to claim 4, characterized in that, Compare the costs required for a combined heat and power (CHP) unit to uniformly provide heat load versus a CHP unit to uniformly provide surplus electrical load; if the cost of providing heat load is lower, then the CHP unit should uniformly provide heat load; if the cost of providing surplus electrical load is lower, then the CHP unit should uniformly provide surplus electrical load.

6. The integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning according to claim 1, characterized in that, The experience replay pool has a capacity limit. If the experience replay pool is full, the oldest set of data is removed and the latest quadruple is added.

7. The integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning according to claim 1, characterized in that, The updating of the policy network, value network, and old policy network includes: The parameter update method for the policy network is shown in the following equation: ； ； In the formula, For the sample size, For policy network parameters, and These represent the states in the policy network and the old policy network, respectively. Taking action probability distribution The advantage function is calculated using generalized advantage estimation. This is a truncation function. To truncate the threshold, is the learning rate of the policy network.

8. A comprehensive energy system optimization scheduling device based on knowledge-guided reinforcement learning, characterized in that, include: The first module is used to set the state variables for reinforcement learning based on the integrated energy system structure. Action variables and reward function The reward function Based on the total system operating cost; The second module is used to obtain the current operating state vector of the system. Determining the state vector based on the policy network Corresponding control action vector ; The third module is used for controlling the action vector. The boundary judgment is carried out. The boundary judgment conditions include whether the system power supply and demand are balanced, whether the system heat supply and demand are balanced, and whether the battery charge state is out of bounds. The fourth module is used to control the action vector. If the control action is judged to be out of bounds, then the control action that affects operating costs is corrected to obtain a corrected control action vector that satisfies the out-of-bounds judgment condition. Otherwise, let ; The fifth module is used to base the modified control action vector. The results of this round of rewards were obtained. and the running state vector for the next round. ; The sixth module is used to record the running status of this round. Controlling actions Award Results and the running state vector for the next round. The resulting quadruple is saved to the experience replay pool; The seventh module is used to update the policy network, value network, and old policy network every preset number of rounds based on the data in the experience replay pool, and then return to the second module until the algorithm converges and the trained policy network is obtained.

9. A storage medium storing a computer program executable by a processor, characterized in that, When the computer program is executed, it implements the steps of the integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning as described in any one of claims 1 to 7.

10. A comprehensive energy system optimization and scheduling device, comprising a memory and a processor, storing a computer program executable by the processor, characterized in that, When the computer program is executed, it implements the steps of the integrated energy system optimization scheduling method based on knowledge-guided reinforcement learning as described in any one of claims 1 to 7.

Images