Optimized scheduling method and system for electric heating cooperative system

Abstract

The invention discloses an electric heating cooperative system optimization scheduling method and system, and the method comprises the steps: building a physical feature model of each equipment part of an electric heating cooperative system which comprises a heat pump, energy storage equipment, renewable energy equipment, a compressor, a power grid, an electric load and a thermal load; establishing an objective function and a constraint condition of optimal scheduling of the electric heating cooperative system; the source load uncertainty is considered, a Markov decision-making process is established, the optimization scheduling problem is converted into a sequential decision-making problem, and the Markov decision-making process is used for representing the operation characteristics of the electric heating cooperative system; and solving the sequential decision problem based on a TD3 algorithm to obtain an optimized scheduling strategy. According to the method, the uncertainty of the source load can be considered, a new solution is provided for research on wind and light absorption and peak load shifting in the electric heating cooperative system, and the comprehensiveness and accuracy of optimization scheduling are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optimization scheduling of electric-thermal coordinated systems, and specifically, relates to an optimization scheduling method and system for electric-thermal coordinated systems. Background Art

[0002] At present, the optimization scheduling methods for electric-heat coordinated systems are divided into three categories: rule-based algorithms, optimization-based algorithms, and learning-based algorithms.

[0003] Rule-based algorithms are divided into deterministic rule algorithms and fuzzy rule algorithms. Deterministic rule algorithms are based on clear conditions and results. Their rules are precise and produce unique and definite behaviors for specific inputs. Fuzzy rule algorithms deal with uncertainty and ambiguity. Their rules allow conditions and results with unclear boundaries. They use fuzzy logic to simulate the ambiguity of human decision-making, so that reasonable reasoning and decisions can be made even when the input data is inaccurate.

[0004] Optimization-based algorithms use various mathematical and technical means to find the optimal solution or approximate optimal solution to a problem. Optimization-based algorithms mainly include linear programming, nonlinear programming, integer programming, combinatorial optimization, and heuristic algorithms.

[0005] Learning-based algorithms are a type of algorithm that improves performance by learning patterns and regularities from data. They rely on machine learning techniques, especially supervised learning, unsupervised learning, and reinforcement learning. They train models to predict the output or behavior of new data points and adaptively adjust their internal parameters to minimize prediction errors or maximize certain reward signals, thereby demonstrating high flexibility and accuracy in handling complex tasks.

[0006] However, the above existing technologies for wind and solar power consumption and peak load shifting are mostly focused on optimized scheduling under fixed scenarios, but the consideration of source and load uncertainty is still insufficient. Therefore, it is necessary to provide a new technical solution to improve one or more problems existing in the above solutions. Summary of the invention

[0007] In order to solve the deficiencies in the prior art, the present invention provides an optimization scheduling method for an electric-thermal coordinated system, which takes into account the multiple uncertainties of source and load output and provides a new solution for the stable operation and sustainable development of the energy system.

[0008] The present invention adopts the following technical solution.

[0009] A first aspect of the present invention provides an optimization scheduling method for an electric-heat coordinated system, comprising:

[0010] Establishing a physical characteristic model of each device component of the electric-heat coordinated system, wherein the electric-heat coordinated system includes: a heat pump, an energy storage device, a renewable energy device, a compressor, a power grid, an electric load, and a thermal load;

[0011] Establish the objective function and constraints for the optimal scheduling of the electric-heat coordinated system;

[0012] Considering the uncertainty of source and load, a Markov decision process is established to transform the optimization scheduling problem into a sequential decision problem. The Markov decision process is used to characterize the operating characteristics of the electric-heat coordinated system.

[0013] Based on the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm, the sequential decision-making problem is solved to obtain the optimized scheduling strategy.

[0014] Optionally, the constraint conditions include system safety operation constraints and power balance constraints.

[0015] Optionally, the energy storage device includes a water storage tank and a steam accumulator, and the system safety constraint is expressed by the following formula:

[0016] HSD_L min ≤HSD_L k ≤HSD_L max

[0017] HSD_H min ≤HSD_H k ≤HSD_H max

[0018]

[0019] in,

[0020] HSD_L k and HSD_H k are the energy storage states of the water storage tank and the steam accumulator, respectively.

[0021] HSD_L min and HSD_L max are the lower and upper limits of the energy storage state of the water storage tank, respectively.

[0022] HSD_H min and HSD_H max are the lower and upper limits of the energy storage state of the steam accumulator, respectively.

[0023] is the power consumption of the first heat pump,

[0024] is the power consumption of the second heat pump,

[0025] is the power consumption of the compressor,

[0026] and are the lower and upper limits of the power consumption of the first heat pump, respectively.

[0027] and are the lower and upper limits of the power consumption of the second heat pump,

[0028] and They are the lower and upper limits of the power consumption of the compressor respectively.

[0029] Optionally, the renewable energy equipment includes photovoltaic power generation equipment and wind turbine generator sets, and the power balance constraint is expressed by the following formula:

[0030]

[0031] in,

[0032] is the electrical load at time k,

[0033] is the heat load at time k,

[0034] is the power generation of the photovoltaic power generation equipment at time k,

[0035] is the power generation of the wind turbine at time k,

[0036] Indicates the purchase or sale of electricity from the power grid. If it is a positive number, it means purchasing electricity from the power grid, and a negative number means selling electricity to the power grid.

[0037] is the power consumption of the first heat pump at time k,

[0038] is the power consumption of the second heat pump at time k,

[0039] is the power consumption of the compressor at time k,

[0040] Indicates the heat generated by the compressor.

[0041] Optionally, the objective function is established as follows:

[0042]

[0043] in,

[0044] C1 is the cost of electricity purchase and sale from the power grid,

[0045] C2 is the state cost of energy storage equipment,

[0046] The power purchased and sold by the power grid,

[0047] p is the current electricity price,

[0048] HSD_L T is the energy storage state of the water storage tank at the end of the scheduling period,

[0049] HSD_H T is the energy storage state of the steam accumulator at the end of the dispatch cycle,

[0050] HSD_L0 is the initial energy storage state of the water tank,

[0051] HSD_H0 is the initial energy storage state of the steam accumulator,

[0052] T represents a scheduling cycle,

[0053] k1, k2, and k3 are the adjustable coefficients of the power grid's electricity purchase and sales cost, the water tank's energy storage state cost, and the steam accumulator's energy storage state cost, respectively, and are used to adjust the power grid's electricity purchase and sales cost and the energy storage equipment's state cost to the same order of magnitude.

[0054] Optionally, considering the uncertainty of source and load, a Markov decision process is established to transform the optimization scheduling problem into a sequential decision problem, including:

[0055] According to the operating characteristics of the electric-heat coordinated system, the state space is established as follows:

[0056]

[0057] Where t is the time step, and are the electrical load and thermal load at the current moment, and are the power generation of photovoltaic power generation equipment and wind turbine generator set, respectively, and HSD_L and HSD_H are the energy storage states of water storage tank and steam accumulator, respectively;

[0058] The action space is established as follows:

[0059]

[0060] in, and Respectively represent the power consumption of the first heat pump, the second heat pump and the compressor;

[0061] The reward function is established as follows:

[0062]

[0063] Among them, C1 is the cost of electricity purchase and sale from the power grid, C2 is the state cost of energy storage equipment,

[0064] To supply excess power to the electric-heat coordinated system,

[0065] s To provide insufficient power for the electric-heat coordinated system.

[0066] The excess heating power of the electric heat coordinated system,

[0067] To provide insufficient heating power for the electric-heat coordinated system.

[0068] The power purchased and sold by the power grid,

[0069] T is the scheduling period,

[0070] P E Preset benchmark values ​​for the electric heat coordination system,

[0071] l1, l2, l3, l4, and l5 are the adjustable coefficients for excess power supply penalty, insufficient power supply penalty, excess heat supply penalty, insufficient heat supply penalty, and excessive electricity purchase penalty, respectively.

[0072] Optionally, the sequential decision problem is solved based on the TD3 algorithm to obtain an optimized scheduling strategy, including:

[0073] Constructing a deep neural network as a policy network P, and constructing two deep neural networks as a first critic network Q1 and a second critic network Q2, wherein the first critic network Q1 and the second critic network Q2 have different initialization parameters;

[0074] Establish corresponding target networks P', Q'1, and Q'2 for the policy network P, the first critic network Q1, and the second critic network Q2 respectively;

[0075] Use the first critic network Q1 and the second critic network Q2 to evaluate the given state s t Take action a t The expected return value Q1(s t ,a t ) and Q2(s t ,a t ), in the expected return value Q1(s t ,at ) and Q2(s t ,a t ) selects the minimum value as the valuation, where, when calculating the expected return value, when taking action a t Add random noise, s t ∈S,a t ∈A;

[0076] The policy network P is iteratively trained based on the TD3 algorithm. The iteration is stopped when the number of iterations reaches the maximum value, the objective function reaches the minimum value, or the reward function converges. The policy network P is used to output the state s in a given state. t The optimal action strategy under , obtains the optimal scheduling strategy.

[0077] Optionally, solving the sequential decision problem based on the TD3 algorithm to obtain an optimized scheduling strategy also includes:

[0078] The OU noise generation mechanism is used to generate exploratory noise;

[0079] Add the opposite value of the exploratory noise to the action output by the policy network P, retain the action output by the policy network P when no noise is added, and construct three different behavioral strategies;

[0080] Three different behavioral strategies interact with the environment to form three different experience trajectories;

[0081] Select the experience trajectory corresponding to the highest reward function value and add it to the experience pool for the next training of the policy network P.

[0082] Optionally, the method further comprises:

[0083] When establishing the physical characteristic models of each equipment component of the electric-heat coordinated system, the electric load and thermal load data are increased or decreased according to the preset fluctuation range to simulate the uncertainty of the operation scenario.

[0084] The second aspect of the present invention provides an optimization scheduling system, which includes a first establishment module, a second establishment module, a third establishment module and an acquisition module. Among them, the first establishment module is used to establish a physical characteristic model of each equipment component of the electric heat collaborative system, and the electric heat collaborative system includes: a heat pump, an energy storage device, a renewable energy device, a compressor, a power grid, an electric load and a thermal load. The second establishment module is used to establish the objective function and constraint conditions for the optimization scheduling of the electric heat collaborative system. The third establishment module is used to consider the uncertainty of the source and load, and establish a Markov decision process to transform the optimization scheduling problem into a sequential decision problem. The acquisition module is used to solve the sequential decision problem based on the TD3 algorithm to obtain an optimization scheduling strategy.

[0085] A third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when loaded into the processor, implements the above-mentioned method for optimizing the scheduling of the electric-heat coordinated system.

[0086] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for optimizing the scheduling of the electric-heat coordinated system.

[0087] Compared with the prior art, the beneficial effects of the present invention include at least:

[0088] (1) The present invention fully considers the uncertainty of energy and load, including the uncertainty of photovoltaic power generation, wind power generation and electric heat demand. The present invention conducts scheduling simulation research on the electric heat coordinated system. When dealing with uncertain scenarios, the present invention adds fluctuation simulation on the basis of typical daily scenarios to simulate the uncertainty of operation scenarios and trains the neural network accordingly. This is intended to improve the system's operational robustness under uncertain conditions.

[0089] (2) The present invention optimizes the existing algorithm, and the specific improvement measures are as follows: In the process of training with the TD3 algorithm, the OU (Ornstein-Uhlenbeck) noise generation mechanism is used to generate exploratory noise. Then, the opposite value of this noise is added to the action output by the policy network, while retaining the original action decision, thereby constructing three different behavioral strategies. When interacting with the environment, these three strategies form independent experience trajectories. Then, the trajectory with the highest reward value is selected from these trajectories and added to the experience replay pool to facilitate subsequent learning and training iterations.

[0090] (3) In view of the operating characteristics of the system, the present invention proposes a method that combines a rule-based algorithm with a learning-based algorithm, aiming to accelerate the convergence speed of the algorithm and improve the scheduling efficiency by introducing rules. Specifically, the present invention takes into account the time-of-use electricity price and electric and thermal load demand of the power grid and sets a benchmark value. When the system enters a high electricity price period, if the system's electricity purchase amount exceeds the benchmark value, a penalty term is introduced in the reward function; if it does not exceed the benchmark value, no penalty is imposed. This strategy ensures that the system avoids excessive electricity purchases during peak electricity price periods, while promoting the rapid convergence of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0092] Figure 1 A schematic diagram of the structure of an electric-heat cooperative system provided in an embodiment of the present invention;

[0093] Figure 2 A schematic diagram of a scheduling optimization process of an electric-heat coordinated system provided in an embodiment of the present invention;

[0094] Figure 3 A schematic diagram of meteorological data and load data of a typical day scenario provided by an embodiment of the present invention;

[0095] Figure 4 A TD3 algorithm training reward curve diagram provided in an embodiment of the present invention;

[0096] Figure 5 A diagram showing changes in energy storage state of an energy storage device according to an embodiment of the present invention;

[0097] Figure 6 A schematic diagram of a typical daily scenario scheduling result provided by an embodiment of the present invention;

[0098] Figure 7 A schematic diagram of another typical day scenario scheduling result provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0100] The technical problem to be solved by the present invention is that the existing technology usually only involves electric-thermal decoupling and system-level scheduling optimization of the electric-thermal coordinated system in a certain scenario, and does not involve efficient, intelligent and real-time scheduling of the system under source and load uncertainty.

[0101] To solve this problem, combined with Figure 1 and Figure 2 As shown, embodiment 1 of the present invention provides an optimization scheduling method for an electric-heat coordinated system, comprising the following steps:

[0102] S1. Establish a physical property model of each equipment component of the electric heat coordinated system, wherein the electric heat coordinated system includes: a heat pump, an energy storage device, a renewable energy device, a compressor, a power grid, an electric load, and a thermal load.

[0103] Specifically, the electric-thermal synergistic system based on steam equivalent energy storage characteristics is taken as the research object, and the physical characteristic models of various equipment components are established.

[0104] Specifically, the heat pump includes a low-temperature heat pump and a high-temperature heat pump, the energy storage equipment includes a water storage tank and a steam accumulator, the water storage tank can be a medium-temperature water storage tank, and the renewable energy equipment includes photovoltaic power generation equipment and a wind turbine.

[0105] Furthermore, in S1, a physical property model of each device component of the electric-heat synergistic system is established, including:

[0106] S1.1. The heat pump described is a compression heat pump, which generates heat by consuming electrical energy. Its physical characteristic model is:

[0107]

[0108] in, and are the power consumption of low-temperature heat pump and high-temperature heat pump, COP L and COP H are the heating efficiencies of the two heat pumps, and The heat generated by the two heat pumps respectively.

[0109] S1.2. The energy storage equipment mentioned above involves water storage tanks and steam accumulators, which are often used to alleviate the uncertainty factors of renewable energy and user loads. The states are all represented by heat storage, and the physical characteristic model is:

[0110]

[0111] Among them, HSD_L t and HSD_H t are the energy storage states of the two energy storage devices at the current time t, and The heat entering and leaving the water storage tank, and are the heat entering and leaving the steam accumulator, HSD_L t+1 and HSD_H t+1 After the status is updated respectively, the status of the heat storage device at the next moment is obtained.

[0112] S1.3, the renewable energy power generation equipment includes photovoltaic power generation equipment and wind turbine generator sets, wherein the physical characteristic model of the photovoltaic power generation equipment is:

[0113]

[0114] Among them, η PV and η inv are photovoltaic efficiency and inverter efficiency respectively, I0 is the solar radiation intensity in watts per square meter (W / m2), A PV is the effective illumination area.

[0115] The physical characteristic model of the wind turbine generator set is:

[0116]

[0117] Among them, v t , v in , v out and v r They are real-time wind speed, start-up wind speed, shutdown wind speed and rated wind speed, Cap wt is the installed capacity of wind turbines, is the power generation of the wind turbine.

[0118] S1.4. The high-temperature steam from the steam accumulator is compressed by the compressor to obtain superheated steam, which is used to supply the user's heat load demand. Its physical characteristic model can be expressed as:

[0119]

[0120] in, is the electrical power consumed by the compressor, η CP is the compression efficiency of the compressor, is the thermal power generated by the compressor.

[0121] S2. Establish the objective function and constraints for optimal scheduling of the electric-heat coordinated system.

[0122] Specifically, the objective function aims to reduce the system operating cost and improve the system sustainable scheduling capability, and the optimization scheduling problem is the energy scheduling optimization problem.

[0123] Specifically, in S2, the constraints include system safety operation constraints and power balance constraints.

[0124] S2 includes:

[0125] S2.1. Establish the following system safety operation constraints:

[0126] HSD_L min ≤HSD_L k ≤HSD_L max

[0127] HSD_H min ≤HSD_H k ≤HSD_H max

[0128]

[0129] Among them, HSD_L k and HSD_H k are the energy storage states of the water storage tank and steam accumulator, respectively, HSD_L min and HSD_L max are the lower and upper limits of the energy storage state of the water storage tank, HSD_H min and HSD_H max are the lower and upper limits of the energy storage state of the steam accumulator, respectively. is the power consumption of the first heat pump,

[0130] is the power consumption of the second heat pump, is the power consumption of the compressor, and are the lower and upper limits of the power consumption of the first heat pump, respectively. and are the lower and upper limits of the power consumption of the second heat pump, and They are the lower and upper limits of the power consumption of the compressor respectively.

[0131] It should be noted that the first heat pump can be Figure 1 The heat pump 1 in the figure, the second heat pump can be Figure 1 The heat pump 2 in the figure will not be described in detail later.

[0132] S2.2. Establish the following power balance constraints:

[0133]

[0134]

[0135] in, is the electrical load at time k, is the heat load at time k, is the power generation of the photovoltaic power generation equipment at time k, is the power generation of the wind turbine at time k, Indicates the purchase or sale of electricity from the power grid. If it is a positive number, it means purchasing electricity from the power grid, and a negative number means selling electricity to the power grid. is the power consumption of the first heat pump at time k, is the power consumption of the second heat pump at time k, is the power consumption of the compressor at time k, Indicates the heat generated by the compressor.

[0136] S2.3. Establish the following objective function:

[0137]

[0138] Among them, C1 is the cost of electricity purchase and sale from the power grid, C2 is the state cost of energy storage equipment, is the power purchased and sold by the power grid, p is the electricity price at the current moment, HSD_L T is the energy storage state of the water storage tank at the end of the scheduling period, HSD_H T is the energy storage state of the steam accumulator at the end of the scheduling cycle, HSD_L0 is the initial energy storage state of the water storage tank, HSD_H0 is the initial energy storage state of the steam accumulator, T represents a scheduling cycle, k1, k2, k3 are the adjustable coefficients of the power grid purchase and sales cost, the water storage tank energy storage state cost and the steam accumulator energy storage state cost, respectively, which are used to adjust the power grid purchase and sales cost and the energy storage equipment state cost to the same order of magnitude.

[0139] During the operation of the system, the water tank and steam accumulator have three states: charging, discharging and non-working. The operating power of the heat pump can be adjusted appropriately according to the current heat load demand and the energy storage state of the water tank and steam accumulator. In view of the scheduling optimization problem of this electric heat coordinated system, it is necessary to accurately control the power output of the heat pump and the compressor to minimize the scheduling cost and improve the overall sustainability of the system. Specifically, the system scheduling cost mainly comes from the cost of purchasing electricity from the power grid. To ensure the stable and continuous operation of the system, at the end of the daily scheduling cycle, the energy storage state of the water tank and the steam accumulator should be restored to the initial level as much as possible to meet the needs of thermal and electric decoupling the next day.

[0140] S3. Considering the uncertainty of source and load, a Markov decision process is established to transform the optimization scheduling problem into a sequential decision problem. The Markov decision process is used to characterize the operating characteristics of the electric-heat coordinated system.

[0141] S3 specifically includes:

[0142] S3.1. According to the operating characteristics of the system, the following state space is established:

[0143]

[0144] The state space is all the information in the process of interaction between the agent and the environment, where t is the time step, and are the electrical load and thermal load at the current moment, and are the power generation of photovoltaic power generation equipment and wind turbine generator set respectively, HSD_L and HSD_H are the energy storage states of water storage tank and steam accumulator respectively.

[0145] S3.2. Establish the following action space:

[0146]

[0147] The action space is the set of all actions that the agent takes after selecting an action based on the state at a certain moment. and Respectively represent the power consumption of the first heat pump, the second heat pump and the compressor.

[0148] S3.3. Establish the following reward function:

[0149]

[0150] The reward function is a reward function based on the environment state and action, which is the reward or punishment obtained by the agent after taking an action.

[0151] Among them, C1 is the cost of electricity purchase and sale from the power grid, C2 is the state cost of energy storage equipment, To supply excess power to the electric-heat coordinated system, To provide insufficient power for the electric-heat coordinated system. The excess heating power of the electric heat coordinated system, To provide insufficient heating power for the electric-heat coordinated system. The power purchase and sale power P of the power grid E The benchmark value is preset for the electric-heat coordinated system. l1, l2, l3, l4, and l5 are adjustable coefficients for excess power supply penalty, insufficient power supply penalty, excess heat supply penalty, insufficient heat supply penalty, and electricity purchase penalty, which can actually be obtained based on historical experience.

[0152] Specifically, the first half of R is defined to be consistent with the objective function, including the cost of purchasing and selling electricity from the power grid and the state cost of the energy storage device. In the early stages of training, despite the consideration of the charging and discharging operations of the energy storage device, the actions output by the actor network may still not ensure power balance. Therefore, it is necessary to add the following additional penalty terms to the reward function to improve energy utilization: In the reward function, by introducing penalty terms for excess or insufficient electric power and thermal power, the system is encouraged to optimize energy management, avoid waste, and ensure that electricity and thermal power demand is met. At the same time, in high electricity price areas, penalties for purchasing electricity from the power grid are increased. These additional penalty terms will help the actor network learn effective scheduling strategies faster during training, thereby improving the operating efficiency and energy management performance of the entire electric-thermal collaborative system.

[0153] S4. Based on TD3, the sequential decision-making problem is solved to obtain the optimal scheduling strategy.

[0154] S4 includes:

[0155] S4.1. Construct a deep neural network as the policy network P, and construct two deep neural networks as the first critic network Q1 and the second critic network Q2, wherein the initialization parameters of the first critic network Q1 and the second critic network Q2 are different.

[0156] S4.2. Establish corresponding target networks P', Q'1, and Q'2 for the policy network P, the first critic network Q1, and the second critic network Q2 respectively.

[0157] S4.3, respectively use the first critic network Q1 and the second critic network Q2 to evaluate the given state s t Take action a t The expected return value Q1(s t ,a t ) and Q2(s t ,a t ), in the expected return value Q1(s t ,a t ) and Q2(s t ,a t ) selects the minimum value as the valuation, where, when calculating the expected return value, when taking action a t Add random noise, s t ∈S,a t ∈A;

[0158] S4.4. Iterate the policy network P based on the TD3 algorithm, stop the iteration when the number of iterations reaches the maximum value, the objective function reaches the minimum value, or the reward function converges, and use the policy network P to output the given state s. t The optimal action strategy under , obtains the optimal scheduling strategy.

[0159] More specifically, Table 1 shows key parameters for training based on the TD3 algorithm according to an embodiment of the present invention.

[0160] Table 1

[0161]

[0162] Optionally, S4 further includes:

[0163] The OU noise generation mechanism is used to generate exploratory noise;

[0164] Add the opposite value of the exploratory noise to the action output by the policy network P, retain the action output by the policy network P when no noise is added, and construct three different behavioral strategies;

[0165] Three different behavioral strategies interact with the environment to form three different experience trajectories;

[0166] Select the experience trajectory corresponding to the highest reward function value and add it to the experience pool for the next training of the policy network P.

[0167] In this embodiment, by adding random noise to the target action, the target action can be slightly disturbed, thereby smoothing the learning curve of the strategy, thereby avoiding overfitting of the strategy in certain specific areas in the network parameter space. In order to further optimize the update process of the strategy and improve its generalization ability,

[0168] Specifically, when the TD3 algorithm generates the target Q value through the target network, it does not simply use the deterministic action output by the target network, but adds a small random noise vector to this action. In this way, the target Q value no longer depends solely on the original target action, but is based on a set of actions perturbed by noise. This approach can be seen as an exploration mechanism that allows the algorithm to consider more possibilities in the action space during training, rather than relying solely on a single optimal action estimate.

[0169] By adding random noise to the target action, the TD3 algorithm encourages the agent to learn a more robust and general strategy. This is because the introduction of noise prevents the agent from being overly dependent on a specific action choice, but learns to perform well within a certain range of actions. The smoothness of this strategy helps reduce the performance degradation that may occur when the agent encounters unseen states or uncertainties in the real world.

[0170] In addition, this approach helps prevent the policy from overfitting in certain areas of the network parameter space. In reinforcement learning, overfitting usually means that the agent performs well in the training environment, but performs poorly when encountering new states or situations in the test environment. By adding random noise to the target action, the TD3 algorithm is able to introduce more variability in the training process, so that the learned policy is more generalized and can maintain stable performance in a wider range of environments. Therefore, this feature of the TD3 algorithm makes it more reliable and effective in solving real-world problems.

[0171] When calculating the target value, the TD3 algorithm will choose the smaller of the two estimates to update, which can effectively reduce the risk of overestimation. If there is only one Q network, the rewards of certain actions may be over-optimistically estimated when calculating the target value, resulting in overestimation. By using two networks and selecting the smaller estimate, TD3 is actually performing a form of "clipping" on the Q value, making the estimate more conservative and closer to the true value. In addition, this approach can also increase the robustness of the algorithm, because even if one network produces an overestimation in some cases, the other network may give a more reasonable estimate, thereby jointly guiding the learning process in the right direction. In this way, the TD3 algorithm can learn strategies more stably during training and show excellent performance in various continuous control tasks. At the same time, this approach also makes TD3 more advantageous than algorithms with a single Q network when dealing with complex environments and high-dimensional action spaces.

[0172] Furthermore, the method also includes: when establishing the physical characteristic model of each equipment component of the electric-heat coordinated system, increasing or decreasing the electric load and thermal load data according to a preset fluctuation range to simulate the uncertainty of the operating scenario.

[0173] Optionally, the fluctuation range is 30%.

[0174] In this embodiment, a set of diverse training scenarios is randomly generated by introducing random source load fluctuations, the fluctuation amplitude of which is limited to no more than 30% of the typical daily scenario load. This fluctuation simulates the load changes that may be encountered in actual operations and improves the algorithm's ability to cope with uncertainty. For each time step of each training round, the environment is explored and the training experience is stored based on the DDPG (Deep Deterministic Policy Gradient) algorithm. The training experience consists of a four-tuple (state-action-reward-next state), that is, (s t ,a t ,r t ,s t+1 ), these experience data sets are used as input to the learning algorithm to update the parameters of the policy network and the critic network.

[0175] During the network update process, instead of updating the policy network and Q network synchronously at every step, the Q network is given more time to iterate and learn to obtain a more accurate value estimate. Specifically, the TD3 algorithm updates the policy network only once after performing multiple Q network updates.

[0176] The use of this strategy, first, helps ensure that the Q network has enough time and data to capture the complex dynamics in the environment and learn more accurate action value estimates. Since the main task of the Q network is to estimate the future reward of taking a specific action in a given state, they need enough data and iterations to refine these estimates. If the Q network and the policy network are updated synchronously, it may cause the Q network to guide the update of the policy network before it converges to an accurate value estimate, which may introduce instability and cause fluctuations in the policy learning process.

[0177] Secondly, delaying the update of the policy network also helps reduce the interaction between the policy network and the Q network. In reinforcement learning, the policy network and the Q network are usually coupled, and the update of the policy network will affect the learning of the Q network, and vice versa. If the two are updated too frequently, the interaction between the two may become too complicated, thus affecting the stability of learning. By delaying the update of the policy network, the TD3 algorithm is able to reduce this interaction, allowing the Q network to learn independently of the policy network to a certain extent.

[0178] Finally, this delayed update strategy also helps improve the overall efficiency of the algorithm. Since the policy network is updated less frequently, the algorithm can focus more on optimizing the value estimate through the Q network, which helps speed up the learning process and provides more reliable and effective guidance when the policy network is finally updated. Therefore, the TD3 algorithm not only improves the stability of learning through this clever delay strategy, but also enhances the performance and adaptability of the algorithm in practical applications.

[0179] After completing the training process of the neural network, the strategy network model is saved to facilitate its deployment in practical applications. The main purpose of the strategy network is to achieve efficient and intelligent scheduling of the wind-solar coupled cogeneration unit energy supply system.

[0180] Embodiment 2 of the present invention provides a system that runs the method described in Embodiment 1, and the system includes a first establishment module, a second establishment module, a third establishment module and an acquisition module. Among them, the first establishment module is used to establish a physical characteristic model of each equipment component of the electric heat collaborative system, and the electric heat collaborative system includes: a heat pump, an energy storage device, a renewable energy device, a compressor, a power grid, an electric load and a thermal load. The second establishment module is used to establish the objective function and constraint conditions for the optimal scheduling of the electric heat collaborative system. The third establishment module is used to consider the uncertainty of the source and load, and establish a Markov decision process to transform the optimal scheduling problem into a sequential decision problem. The acquisition module is used to solve the sequential decision problem based on the TD3 algorithm to obtain an optimal scheduling strategy.

[0181] Embodiment 3 of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method described in Embodiment 1 when loaded into the processor.

[0182] Embodiment 4 of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to embodiment 1 is implemented.

[0183] Combination Figures 3 to 7 As shown, Example 5 of the present invention provides a verification example of an optimization scheduling method for an electric-heat coordinated system.

[0184] Before each training round, the system randomly generates a set of system operating conditions based on typical daily scenarios and the feasible domain of safe equipment operation. These conditions include parameters such as solar irradiance, wind speed, electrical load, and thermal load to simulate multiple uncertainties in actual operation. Figure 5 The changes in electric load, thermal load, photovoltaic output and wind power generation under typical daily scenarios are shown.

[0185] In order to simulate the uncertainty in actual operation, this study introduced a volatility of no more than 30% based on a typical daily scenario to simulate the uncertainty that the system may encounter. This treatment method can more realistically evaluate the performance and stability of the system when facing load changes. Figure 3 The fluctuation range of load data in the training scenario is shown, which is also the data range based on which the algorithm is trained. Specifically, Figure 3 The load data fluctuation range in reflects the potential changes in system load under different uncertainty levels, thus providing the algorithm with diverse training scenarios to ensure that it maintains good adaptability and predictive capabilities when facing various uncertain situations in actual operations.

[0186] Figure 4 The training status of the TD3 algorithm is shown, where "mean" represents the average value of the cumulative reward in every 5 training cycles. The training curve shows that in the early stage of training, due to the introduction of noise, the cumulative reward of the TD3 algorithm is low, reaching the lowest point of about -2200 at about 5 training rounds. As the training continues, the cumulative reward of the TD3 algorithm gradually increases and begins to converge at about 150 training rounds. Finally, the cumulative reward of the algorithm stabilizes at about -200, indicating that the algorithm has successfully learned an effective strategy and has stable performance.

[0187] Figure 5The figure shows the changes in the energy storage state of the water tank and the steam accumulator in a typical day scenario. The HSD (Heat Storage Degree) in the upper and lower parts of the figure represent the energy storage state of the water tank and the steam accumulator, respectively. For clarity, the energy storage state of the water tank and the steam accumulator are both in kilowatt-hours. The upper part of the figure shows the changes in the energy storage state of the water tank. It can be seen that in the first 7 moments, the decision-making agent tends to fill the water tank with water to prepare for subsequent hot water use needs, and gradually returns to the initial state at the end of the scheduling cycle. The lower part of the figure shows the changes in the energy storage state of the steam accumulator. It is obvious that at the initial moment, the decision-making agent replenishes the energy of the steam accumulator for subsequent use, and gradually returns to the initial state at the end of the scheduling cycle. In general, both the water tank and the steam accumulator tend to replenish energy in the early stage of scheduling to meet the subsequent heating needs.

[0188] Figure 6 The power balance in a typical day scenario is shown. It can be clearly seen from the figure that in the first 7 moments of the dispatch cycle, the system took advantage of the low electricity price period to purchase a large amount of electricity, and used wind and solar power generation to store water in the water tank. As the demand for electricity load increases during the day, wind and solar power generation can hardly meet the electricity demand. Therefore, the system purchases a small amount of electricity from the power grid to meet user needs, while controlling the water tank to stop taking in water to reduce unnecessary power purchase losses. In addition, the system also charges energy to the steam accumulator to cope with the growing heat load demand. At the end of the dispatch cycle, the system gradually reduces the charging of the steam accumulator to restore its state to the initial moment of dispatch.

[0189] Figure 7 The optimization results of the system heat supply and demand balance in the day-ahead scenario are shown. It can be clearly seen from the figure that at each moment of the scheduling cycle, the system's heat supply can fully meet the actual needs of users. This shows that through an efficient scheduling strategy, the system can maintain a good match between heat supply and demand, ensuring the thermal comfort of users and the stable operation of the system. In addition, Figure 7 It also reveals the dynamic adjustment process of heat supply. During peak demand periods, the system effectively increases the heat supply by increasing the operating power of the high-temperature heat pump to meet the sudden increase in heat demand. During low demand periods, the system reduces the heat supply accordingly, by reducing the operating efficiency of the equipment or adopting a heat storage strategy to optimize energy efficiency and reduce operating costs. This flexible thermal management not only ensures the quality of heating for users, but also promotes the improvement of the overall energy efficiency of the system and the reduction of energy consumption.

[0190] Compared with the prior art, the beneficial effects of the present invention include at least:

[0191] (1) The present invention fully considers the uncertainty of energy and load, including the uncertainty of photovoltaic power generation, wind power generation and electric heat demand. The present invention conducts scheduling simulation research on the electric heat coordinated system. When dealing with uncertain scenarios, the present invention adds fluctuation simulation on the basis of typical daily scenarios to simulate the uncertainty of operation scenarios and trains the neural network accordingly. This is intended to improve the system's operational robustness under uncertain conditions.

[0192] (2) The present invention optimizes the existing algorithm, and the specific improvement measures are as follows: In the process of training with the TD3 algorithm, the OU noise generation mechanism is used to generate exploratory noise. Then, the opposite value of this noise is added to the action output by the policy network, while retaining the original action decision, thereby constructing three different behavioral strategies. When interacting with the environment, these three strategies form independent experience trajectories. Then, the trajectory with the highest reward value is selected from these trajectories and added to the experience replay pool to facilitate subsequent learning and training iterations.

[0193] (3) In view of the operating characteristics of the system, the present invention proposes a method that combines a rule-based algorithm with a learning-based algorithm, aiming to accelerate the convergence speed of the algorithm and improve the scheduling efficiency by introducing rules. Specifically, the present invention takes into account the time-of-use electricity price and electric and thermal load demand of the power grid and sets a benchmark value. When the system enters a high electricity price period, if the system's electricity purchase amount exceeds the benchmark value, a penalty term is introduced in the reward function; if it does not exceed the benchmark value, no penalty is imposed. This strategy ensures that the system avoids excessive electricity purchases during peak electricity price periods, while promoting the rapid convergence of the algorithm.

[0194] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0195] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0196] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0197] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0198] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing the scheduling of an electric-heat coordinated system, characterized in that: include: Establishing a physical characteristic model of each device component of the electric-heat coordinated system, wherein the electric-heat coordinated system includes: a heat pump, an energy storage device, a renewable energy device, a compressor, a power grid, an electric load, and a thermal load; Establish the objective function and constraints for the optimal scheduling of the electric-heat coordinated system; Considering the uncertainty of source and load, a Markov decision process is established to transform the optimization scheduling problem into a sequential decision problem. The Markov decision process is used to characterize the operating characteristics of the electric-heat coordinated system. The sequential decision-making problem is solved based on the TD3 algorithm to obtain the optimal scheduling strategy.

2. The method for optimizing and scheduling an electric-heat coordinated system according to claim 1, characterized in that: The constraint conditions include system safety operation constraints and power balance constraints.

3. The method for optimizing and scheduling the electric-heat coordinated system according to claim 2, characterized in that: The energy storage equipment includes water storage tanks and steam accumulators. The system safety constraints are expressed as follows: HSD_L min ≤HSD_L k ≤HSD_L max HSD_H min ≤HSD_H k ≤HSD_H max in, HSD_L k and HSD_H k are the energy storage states of the water storage tank and the steam accumulator, respectively. HSD_L min and HSD_L max are the lower and upper limits of the energy storage state of the water storage tank, respectively. HSD_H min and HSD_H max are the lower and upper limits of the energy storage state of the steam accumulator, respectively. is the power consumption of the first heat pump, is the power consumption of the second heat pump, is the power consumption of the compressor, and are the lower and upper limits of the power consumption of the first heat pump, respectively. and are the lower and upper limits of the power consumption of the second heat pump, and They are the lower and upper limits of the power consumption of the compressor respectively.

4. The method for optimizing and scheduling the electric-heat coordinated system according to claim 3, characterized in that: Renewable energy equipment includes photovoltaic power generation equipment and wind turbines. The power balance constraint is expressed as follows: in, is the electrical load at time k, is the heat load at time k, is the power generation of the photovoltaic power generation equipment at time k, is the power generation of the wind turbine at time k, Indicates the purchase or sale of electricity from the power grid. If it is a positive number, it means purchasing electricity from the power grid, and a negative number means selling electricity to the power grid. is the power consumption of the first heat pump at time k, is the power consumption of the second heat pump at time k, is the power consumption of the compressor at time k, Indicates the heat generated by the compressor.

5. The method for optimizing and scheduling the electric-heat coordinated system according to claim 4, characterized in that: The objective function is established as follows: J=min(C1+C2) in, C1 is the cost of electricity purchase and sale from the power grid, C2 is the state cost of energy storage equipment, The power purchased and sold by the power grid, p is the current electricity price, HSD_L T is the energy storage state of the water storage tank at the end of the scheduling period, HSD_H T is the energy storage state of the steam accumulator at the end of the dispatch cycle, HSD_L0 is the initial energy storage state of the water tank, HSD_H0 is the initial energy storage state of the steam accumulator, T represents a scheduling cycle, k1, k2, and k3 are the adjustable coefficients of the power grid's electricity purchase and sales cost, the water tank's energy storage state cost, and the steam accumulator's energy storage state cost, respectively, and are used to adjust the power grid's electricity purchase and sales cost and the energy storage equipment's state cost to the same order of magnitude.

6. The method for optimizing and scheduling the electric-heat coordinated system according to claim 5, characterized in that: Considering the uncertainty of source and load, a Markov decision process is established to transform the optimization scheduling problem into a sequential decision problem, including: According to the operating characteristics of the electric-heat coordinated system, the state space is established as follows: Where t is the time step, and are the electrical load and thermal load at the current moment, and are the power generation of photovoltaic power generation equipment and wind turbine generator set, respectively, and HSD_L and HSD_H are the energy storage states of water storage tank and steam accumulator, respectively; The action space is established as follows: in, and Respectively represent the power consumption of the first heat pump, the second heat pump and the compressor; The reward function is established as follows: Among them, C1 is the cost of electricity purchase and sale from the power grid, C2 is the state cost of energy storage equipment, To supply excess power to the electric-heat coordinated system, To provide insufficient power for the electric-heat coordinated system. The excess heating power of the electric heat coordinated system, To provide insufficient heating power for the electric-heat coordinated system, The power purchased and sold by the power grid, T is the scheduling period, P E Preset benchmark values ​​for the electric heat coordination system, l1, l2, l3, l4, and l5 are the adjustable coefficients for excess power supply penalty, insufficient power supply penalty, excess heat supply penalty, insufficient heat supply penalty, and excessive electricity purchase penalty, respectively.

7. The method for optimizing and scheduling the electric-heat coordinated system according to claim 6, characterized in that: The sequential decision-making problem is solved based on the TD3 algorithm to obtain the optimal scheduling strategy, including: Constructing a deep neural network as a policy network P, and constructing two deep neural networks as a first critic network Q1 and a second critic network Q2, wherein the first critic network Q1 and the second critic network Q2 have different initialization parameters; Establish corresponding target networks P', Q1', and Q'2 for the policy network P, the first critic network Q1, and the second critic network Q2 respectively; Use the first critic network Q1 and the second critic network Q2 to evaluate the given state s t Take action a t The expected return value Q1(s t ,a t ) and Q2(s t ,a t ), in the expected return value Q1(s t ,a t ) and Q2(s t ,a t ) selects the minimum value as the valuation, where, when calculating the expected return value, when taking action a t Add random noise, s t ∈S,a t ∈A; The policy network P is iteratively trained based on the TD3 algorithm. The iteration is stopped when the number of iterations reaches the maximum value, the objective function reaches the minimum value, or the reward function converges. The policy network P is used to output the state s in a given state. t The optimal action strategy under , obtains the optimal scheduling strategy.

8. The method for optimizing and scheduling the electric-heat coordinated system according to claim 7, characterized in that: Solving the sequential decision-making problem based on the TD3 algorithm to obtain the optimal scheduling strategy also includes: The OU noise generation mechanism is used to generate exploratory noise; Add the opposite value of the exploratory noise to the action output by the policy network P, retain the action output by the policy network P when no noise is added, and construct three different behavioral strategies; Three different behavioral strategies interact with the environment to form three different experience trajectories; Select the experience trajectory corresponding to the highest reward function value and add it to the experience pool for the next training of the policy network P.

9. The method for optimizing and scheduling an electric-heat coordinated system according to any one of claims 1 to 8, characterized in that: The method further comprises: When establishing the physical characteristic models of each equipment component of the electric-heat coordinated system, the electric load and thermal load data are increased or decreased according to the preset fluctuation range to simulate the uncertainty of the operation scenario.

10. An optimization scheduling system using the optimization scheduling method for the electric-heat coordinated system according to any one of claims 1 to 9, characterized in that: The system comprises: A first building module is used to build a physical characteristic model of each device component of the electric-heat synergistic system, wherein the electric-heat synergistic system includes: a heat pump, an energy storage device, a renewable energy device, a compressor, a power grid, an electric load, and a thermal load; The second establishment module is used to establish the objective function and constraint conditions for the optimal scheduling of the electric-heat coordinated system; The third module is used to consider the uncertainty of source and load and establish a Markov decision process to transform the optimization scheduling problem into a sequential decision problem; The acquisition module is used to solve the sequential decision-making problem based on the TD3 algorithm to obtain the optimal scheduling strategy.

11. An electronic device, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instruction to execute the steps of the method for optimizing scheduling of an electric-heat coordinated system according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for optimizing the scheduling of an electric-heat coordinated system according to any one of claims 1 to 9 are implemented.

Images