Optimized dispatching method for electricity-hydrogen-heat comprehensive energy microgrid based on improved TD3 algorithm

Abstract

The invention relates to the technical field of integrated energy micro-grid scheduling optimization, and discloses an improved TD3 algorithm-based electricity-hydrogen-heat integrated energy micro-grid optimization scheduling method, which aims at minimizing the sum of the environmental cost, the equipment life loss cost and the large power grid interaction cost, and uses the improved TD3 algorithm to optimize the power-hydrogen-heat integrated energy micro-grid scheduling optimization. The optimization scheduling strategy of the electricity-hydrogen-heat comprehensive energy microgrid system is researched, a hydrogen energy storage mathematical model is constructed, the electrothermal coupling characteristic between the hydrogen energy storage system and the microgrid is explored, the optimization scheduling process of the microgrid is converted into a Markov decision process, and an experience playback mechanism is introduced to improve the algorithm performance. According to the method, the environmental friendliness and the scheduling strategy solving speed of the micro-grid energy storage system are improved, and the solving speed of the optimal scheduling strategy is accelerated by using the improved priority experience playback reinforcement learning algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated energy microgrid dispatch optimization, and specifically to an electric-hydrogen-heat integrated energy microgrid optimization dispatch method based on an improved TD3 (Twin Delayed Deep Deterministic policy gradient) algorithm. Background Art

[0002] Renewable energy represented by photovoltaics and wind power has developed rapidly in recent years due to its clean and sustainable advantages, and its penetration rate in the power grid has continued to increase.

[0003] At present, renewable energy is mostly connected to the power grid through power electronic devices such as inverters to form distributed power sources in the power system. Compared with the centralized power supply of fossil energy, distributed power sources have two operating modes: grid-connected and off-grid. They can improve the flexibility of power supply and alleviate the environmental pressure caused by fossil energy to a certain extent. However, their output has obvious regional distribution differences and time-space fluctuations. When they are connected on a large scale, they will cause shocks and reduce the stability of the power system.

[0004] In order to effectively resolve the contradiction between traditional power grids and distributed power sources, scholars such as RHLasserter proposed the concept of microgrids, which refers to small power systems that use a high proportion of distributed power sources as the main energy supply mode for loads and are equipped with energy storage and energy conversion devices, data acquisition and monitoring control systems. The operation mode of microgrids is relatively flexible. They can be connected to the grid or operate independently from the main grid. They can serve as the load of the main grid or transmit electricity to the main grid. Microgrids have six major advantages, including improving the two-way flow of information, improving power quality, power system reliability and flexibility, comprehensive energy utilization efficiency, enhancing energy coordination and interaction between regions, and realizing active load regulation. They are regarded as an important part of the future new power system.

[0005] As a flexible, efficient and clean secondary energy source, introducing hydrogen energy into microgrids and coupling it with renewable energy can effectively improve the flexibility, renewable energy absorption capacity and cleanliness of microgrids.

[0006] However, the following technical problems still exist in optimizing the dispatch of electric, hydrogen and heat integrated energy microgrids: 1. It is necessary to mathematically model each power component in the microgrid to ensure the accuracy of the scheduling results. However, the operating characteristics and constraints of each component in the entire scheduling process are time-varying. To establish an accurate mathematical model, it is necessary to comprehensively consider the dynamic characteristics, energy consumption, energy conversion and other complex factors of each component. It is difficult to establish a real-time parameter model during operation.

[0007] 2. The experience replay process based on deep reinforcement learning algorithm scheduling optimization is based on the "first in, first out" principle, without considering the importance or priority of experience. This may cause some important experiences to be frequently covered or forgotten, leading to low sampling efficiency. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide an optimized dispatching method for an electric, hydrogen and heat integrated energy microgrid based on an improved TD3 algorithm. By constructing a hydrogen energy storage mathematical model, the electric and thermal coupling characteristics between the hydrogen energy storage system and the microgrid are explored, the environmental friendliness of the microgrid energy storage system and the dispatching strategy solution speed are improved, and the improved priority experience replay reinforcement learning algorithm is used to speed up the solution speed of the optimal dispatching strategy. The technical solution is as follows: The optimized dispatching method of electric-hydrogen-heat integrated energy microgrid based on the improved TD3 algorithm includes the following steps: Step 1: Establish corresponding mathematical models for the components of the electric-hydrogen-heat integrated energy system: electrolyzer, combustion battery, hydrogen storage tank and compressor; Step 2: Based on the Markov decision process, establish the Markov decision process for dispatch optimization of electric, thermal and hydrogen integrated energy microgrid: ;in, It is the possible state space of each power component in the microgrid environment; The action space includes various operations and dispatch strategies adopted by the microgrid system; is the state transition probability, describing the state Take an action After that, the probability distribution of the microgrid system transferring from one state to another; is an immediate reward, representing the Take an action After that, the agent gets the reward; is a discount factor between 0 and 1, indicating the importance of future rewards; Step 3: Train the agent based on the priority experience replay TD3 algorithm to obtain a day-ahead dispatch strategy that minimizes the life depreciation cost of the electrolyzer and lead-acid battery, the carbon cost of the microgrid system, and the cost of natural gas purchased by the microgrid system from the natural gas supplier.

[0009] The beneficial effects of the present invention are: 1. The present invention improves the environmental friendliness of the microgrid energy storage system and the speed of solving the dispatching strategy: by introducing the hydrogen energy system and deeply studying its electric-thermal coupling characteristics, it not only improves the energy storage efficiency of the microgrid, but also significantly enhances the environmental friendliness of the system. At the same time, the improved priority experience replay reinforcement learning algorithm (TD3-PER Twin Delayed DeepDeterministic Policy Gradient with Prioritized Experience Replay, i.e. the improved TD3 algorithm) is used to greatly speed up the speed of solving the optimal dispatching strategy.

[0010] 2. The present invention achieves lower system daily operating costs and environmental costs, and verifies the transferability of the strategy: Compared with the traditional TD3 algorithm, the TD3-PER algorithm can more efficiently utilize the scheduling strategy experience and find a better action decision space, thereby reducing the system's daily operating costs and environmental costs while ensuring system stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of the method for optimizing the dispatching of an electric, hydrogen and heat integrated energy microgrid based on the improved TD3 algorithm of the present invention.

[0012] Figure 2 This is the structural diagram of the electric, hydrogen and heat integrated energy microgrid system of the simulation example of the present invention.

[0013] Figure 3 This is a graph showing how the reward value changes with the number of training rounds during the agent training process.

[0014] Figure 4 (a) shows the optimization scheduling results for a typical day in summer.

[0015] Figure 4(b) shows the optimization scheduling results for a typical day in winter.

[0016] Figure 5 The similarity calculation results between power grids.

[0017] Figure 6 This is the convergence result of the reward value for the day-ahead optimization scheduling. DETAILED DESCRIPTION

[0018] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The present invention provides an optimization scheduling method for an electric, hydrogen and heat integrated energy microgrid based on an improved TD3 algorithm, with the goal of minimizing the sum of environmental costs, equipment life depreciation costs, and interaction costs with the large power grid. The improved TD3 algorithm is used to study the optimization scheduling strategy of an electric, hydrogen and heat integrated energy microgrid system, construct a hydrogen energy storage mathematical model, explore the electric and thermal coupling characteristics between the hydrogen energy storage system and the microgrid, transform the optimization scheduling process of the microgrid into a Markov decision process, introduce an experience replay mechanism to improve the algorithm performance, and verify that transfer learning can effectively improve the training rate of the intelligent agent reward value.

[0020] like Figure 1 As shown, the electric-hydrogen-heat integrated energy microgrid optimization scheduling method based on the improved TD3 algorithm of the present invention comprises the following steps: Step 1: Establish corresponding mathematical models for the components of the electric hydrogen heat integrated energy system, namely, electrolyzer, combustion battery, hydrogen storage tank and compressor. The specific steps are: The structure of the electric, hydrogen and heat integrated energy microgrid system studied in this invention is as follows: Figure 2 As shown. Considering the cost, technical maturity and power demand of the microgrid, an alkaline electrolyzer is selected as the hydrogen production equipment of the electric-hydrogen-heat integrated energy microgrid. The alkaline electrolyzer usually includes an anode and a cathode, both of which are made of inert materials such as platinum. The anode and the cathode are connected to each other through an electrolyte solution or alkaline substances in the electrolyzer, and the electrolysis reaction mainly occurs at the anode and the cathode.

[0021] At the anode, water undergoes an oxidation reaction to produce oxygen and hydrogen ions, expressed as: ;

[0022] At the cathode, water undergoes a reduction reaction to generate hydrogen gas and hydroxide ions, expressed as: ;

[0023] By passing direct current through an alkaline electrolytic cell, water can be electrolyzed to produce hydrogen. The relationship between the hydrogen production rate and the external current is expressed as: ; In the formula, express The hydrogen production rate of the electrolyzer at the moment, is the Faraday constant, is the number of electrons produced in each reaction, for The external current of the electrolytic cell at the moment, is the number of electrolytic cells, is the Faraday efficiency, calculated as follows: ; In the formula, is the electrolysis module area, is the working temperature of the electrolyzer; is the Faraday efficiency coefficient.

[0024] The calculation formula of the electrolytic cell single cell voltage is expressed as: ; In the formula, is the standard free energy of liquid water, is the empirical temperature coefficient, , is the ohmic resistance parameter of the electrolytic cell, and are the empirical fitting parameters of the electrolyzer overvoltage model, and and is the basic activation overvoltage constant, and is the temperature linear correction coefficient, and is the secondary temperature correction coefficient.

[0025] The output power of the electrolyzer is expressed as: ; In the formula, is the efficiency of the electrolyzer.

[0026] The present invention combines it with lithium batteries to form a microgrid composite energy storage system. The voltage of the proton exchange membrane fuel cell stack is expressed as: ; In the formula, is the terminal voltage of the fuel cell, is the equivalent impedance inside the fuel cell, is the fuel cell current, , and is the empirical coefficient, is the number of fuel cells connected in series.

[0027] The fuel cell hydrogen consumption rate is expressed as: ; In the formula, is the hydrogen utilization efficiency of the fuel cell.

[0028] The present invention uses compressed hydrogen storage technology to store hydrogen. The internal pressure of the hydrogen storage tank is expressed as: ; In the formula, for The internal pressure of the hydrogen storage tank at any moment, is the gas constant, is the hydrogen storage tank temperature, is the volume of the hydrogen storage tank, Indicates the hydrogen consumption rate of heat load.

[0029] Since the pressure in the hydrogen storage tank is proportional to the amount of stored hydrogen, the relative capacity of hydrogen at that moment can be obtained by the following formula: ; In the formula, for The relative capacity of hydrogen at the time, It is the maximum pressure that a hydrogen storage tank can withstand.

[0030] Step 2: Based on the Markov decision process, establish the Markov decision process for dispatch optimization of electric, thermal and hydrogen integrated energy microgrid: ;in It is the possible state space of each power component in the microgrid environment; The action space includes various operations and dispatch strategies adopted by the microgrid system; is the state transition probability, describing the state Take an action After that, the probability distribution of the microgrid system transferring from one state to another; is an immediate reward, representing the Take an action After that, the agent gets the reward; is a discount factor between 0 and 1, indicating the importance of future rewards. The specific steps include: (1) Possible state space : Microgrids State variables of the integrated energy microgrid system within a certain time , expressed as: ; In the formula, , and Respectively indicate scheduling Photovoltaic and wind power output power and electrical load at the time, and Respectively, in scheduling The price of natural gas and carbon trading price at the time; is the state of charge of the lead-acid battery; express The capacity of the fuel cell at any moment; express Microgrid heat load demand at every moment; for Real-time electricity prices at any given moment.

[0031] (2) Action Space : Each The microgrid controls the charging and discharging of the composite energy storage system to achieve the balance of supply and demand of the microgrid, and at the same time controls the electrolyzer to produce hydrogen to provide raw materials for the fuel cell and meet the demand of heat load. Then the action variable of the system is , expressed as: ; In the formula, for Lead-acid battery power at the moment; express Fuel cell power at the time; express Electrolyzer power at the moment.

[0032] (3) Instant Rewards : The goal of reinforcement learning is to maximize the cumulative reward, and the objective function of the integrated energy microgrid of the present invention is to minimize the daily operating cost. Therefore, the reward function of reinforcement learning is set to the opposite of the objective function to achieve directional consistency, which is expressed as: ; In the formula, It is the objective function for optimizing the dispatch of microgrid on the day ahead.

[0033] (4) State transition probability : ; In the formula, for The state space at time, for Action space at any moment.

[0034] Step 3: Train the intelligent agent based on the priority experience replay TD3 algorithm to obtain a day-ahead scheduling strategy with the goal of minimizing the life depreciation cost of the electrolyzer and composite energy storage system, the carbon cost of the microgrid system, and the cost of natural gas purchased by the microgrid system from the natural gas supplier.

[0035] The specific steps are as follows: When the hydrogen in the system cannot meet the demand of heat load, the microgrid needs to purchase natural gas from the natural gas supplier to fill the gap. Natural gas will produce carbon dioxide during use, so the environmental cost of the microgrid needs to be considered; at the same time, the electrolyzer and energy storage equipment will also suffer life loss during operation. In summary, the objective function of the microgrid's optimal scheduling is constructed as follows: ; In the formula, and Respectively The life loss cost of electrolyzer and composite energy storage system at the same time, represents the carbon cost of the microgrid system, is the cost of natural gas purchased by the microgrid system from the natural gas supplier, for The interaction power between the microgrid and the main grid at any time, for Real-time electricity prices at any given moment.

[0036] At the same time, the optimization scheduling problem of the electric, hydrogen and heat integrated energy microgrid should meet the following constraints: (1) Electrical load balance: ; In the formula, express Fuel cell power at the moment.

[0037] (2) Heat load balance: ; In the formula, and Respectively The heat energy provided by hydrogen and natural gas at all times, express The heat load demand of the microgrid at each moment.

[0038] (3) Power constraints of each power component: ; In the formula, and Respectively represent the maximum and minimum state of charge of the lead-acid battery; and Respectively represent the maximum and minimum output of lead-acid batteries; express The capacity of the fuel cell at the moment, and Respectively represent the upper and lower limits of fuel cell capacity, and Respectively represent the upper and lower limits of fuel cell power, and Respectively represent the upper and lower limits of the electrolytic cell power, express The volume of hydrogen passing through the compressor at any given moment, Indicates the upper limit of the volume of hydrogen that can pass through the compressor. express The volume of hydrogen output by the compressor at the moment, Indicates the upper limit of the hydrogen volume output by the compressor.

[0039] According to a possible implementation, during the process of training an agent based on the Priority Experience Replay TD3 algorithm, the algorithm alternately performs strategy evaluation, strategy improvement, and Priority Experience Replay; wherein, In the strategy evaluation phase, it is necessary to calculate the state-action value, that is, , the Q function can be expressed by the Bellman equation: ; In the formula, Represents the state-action value in the value evaluation Q network; express Instant rewards at the moment and Discount factor at time State-action value The expectation of the sum.

[0040] After parameterizing the Q function using a neural network, the Q function is approximated by minimizing the Bellman residual: ; In the formula, is the minimized Bellman residual, and Represent the parameters of the value assessment Q network and the TargetQ network respectively.

[0041] In the strategy improvement stage, the Q function is parameterized using a neural network to minimize the objective function The gradient used to update the network parameters is: ; In the formula, Represents the policy network parameters Derivation; For expectations, Indicates The policy network value evaluates the network state-action value in the training round Actions in Seeking guidance, Indicates Policy network in training rounds Parameters in Derivative.

[0042] According to a possible implementation, during the training of the agent using the improved TD3 algorithm, two Target Q networks with different initial parameters are used to evaluate the target value, and the smaller value is selected as the target value. Therefore, the Bellman residual to be minimized is corrected to: ; In the formula, for The rewards of the moment, for The smaller value of the moment value assessment; for The action after adding disturbance at the moment; k represents the number of value evaluation Q networks; for The value assessment of the moment.

[0043] Use target policy smoothing regularization to enhance the stability of the policy and smooth the Q function; that is, when calculating the Bellman residual, in the next state Actions taken Will be selected as: ; Where μ represents the Target strategy network; ε is the added noise, which is generally selected as Gaussian noise, and its amplitude is clipped to limit it to a smaller range.

[0044] Prioritized experience replay uses the temporal difference error (TD-error) corresponding to each sample to measure its importance. The intelligent agent prioritizes samples with higher value for training to reduce exploration time. Specifically, samples with larger TD errors (i.e., the deviation between the actual Q value and the estimated Q value of the sample is large) are considered to be more important for network learning, and are therefore given a higher probability and are used for training first.

[0045] Introducing the concept of sample priority: defining each sample in the experience pool Probability of being sampled by the agent and the absolute value of TD-error Positive correlation, expressed as: ; In the formula, represents the sample sampling probability matrix, represents the probability of each sample being sampled, , is a hyperparameter used to solve the extreme value problem that occurs during the TD-error calculation process (when degenerates to uniform sampling), is a small positive constant that ensures that every sample has at least a small priority, even if their TD-error is zero;

[0046] Weighted importance sampling rate correction: When using priority sampling, high-priority samples will be selected more frequently. However, this may lead to an uneven distribution of samples. By introducing the weighted importance sampling rate, this unevenness can be balanced to ensure that both high-priority and low-priority samples receive appropriate attention. The importance sampling coefficient is defined as follows: ; In the formula, is the total number of samples in the experience pool; is a hyperparameter used to control the balance between sampling priority and weights in experience replay and smooth high-variance importance sampling weights.

[0047] Figure 3 The curve of the reward value changing with the number of training rounds during the agent training process is given. It can be observed from the figure that after 900 rounds of training, the DDPG (Deep Deterministic Policy Gradient) algorithm tends to converge near the reward value of 180. However, compared with the TD3 algorithm, the stability of the optimal scheduling strategy obtained by the DDPG algorithm is weaker. At the same time, it can be seen from the figure that in the initial stage, due to insufficient training of the agent and the small number of samples in the experience pool, the accuracy of the priority calculation is low, which limits the ability of TD3-PER to fully utilize the priority sampling mechanism, and the probability of the agent extracting low reward value experience is high; as the agent continues to interact with the environment, the accumulation of experience and the optimization of priority calculation enable TD3-PER to use high reward value experience to improve the strategy at a high frequency, and make better scheduling decisions and obtain higher reward values ​​in the subsequent training process.

[0048] The results of the selected optimization scheduling of typical days in summer and winter are shown in Figure 4 (a) and Figure 4 (b). It can be seen from the figure that in summer, the temperature is high, the photovoltaic supply is large, and the heat load demand in the power grid is small. During the day, on the basis of meeting the electricity load demand, most of the excess electricity is supplied to the electrolyzer to produce hydrogen for use when the heat load demand is large in winter. At the same time, the battery system is charged for use at night; the electricity load and heat load demand at night are relatively high compared to the daytime. At this time, the battery is discharged, and a small amount of hydrogen in the hydrogen storage tank is used for the fuel cell to release electricity to assist the lithium battery to achieve load balance and reduce the operating cost of the microgrid. The temperature on a typical day in winter is low, the photovoltaic output is low, and the heat load demand in the microgrid is high. Excluding the power supplied to the fuel cell, the net power of the electrolyzer for hydrogen production is greatly reduced, which can no longer meet the demand for heat load, resulting in a gap in heat demand. At this time, the microgrid needs to purchase natural gas from the natural gas supplier to meet the heat load consumption.

[0049] Step 4: Analyze the impact of transfer learning on the training rate of improving the agent's reward value.

[0050] The specific steps are as follows: In order to verify the effect of transfer learning on the generalization ability of the optimization scheduling strategy, the above microgrid is used as the source domain, and five microgrids with similar geographical locations are selected as the target domain. The geographical locations are shown in Table 1. The wind and solar load data are used as the standard, and the MMD function is used to calculate the distribution difference between domains. ,by As the similarity, the calculation result is as follows Figure 5 As shown in the figure, it can be seen that the wind and solar load data of Microgrid No. 3 within one year is most similar to the source domain microgrid, so it is regarded as the target domain for optimization scheduling strategy migration.

[0051] Table 1 Coordinates of the five target domain microgrids Power station number longitude latitude source domain -120.6867 37.5929 Target Area 1 Microgrid -120.6658 37.5795 Target area 2 microgrid -120.6987 37.5800 Target Area 3 Microgrid -120.6889 37.5961 Target Area 4 Microgrid -120.6735 36.5821 Target Area Microgrid No. 5 -120.6797 37.5885

[0052] The network parameters when the reward value of the source domain microgrid optimization dispatch task converges are “frozen” as the initial values ​​of the network parameters of the target domain microgrid optimization dispatch task for strategy migration. The convergence results of the optimization dispatch reward value are as follows: Figure 6 As shown, the agent is retrained to find the optimal solution.

[0053] The present invention provides strong support for the wide application and flexible scheduling of microgrids by verifying the portability of the optimal scheduling strategy between microgrids.

Claims

1. An optimized dispatching method for electric, hydrogen and heat integrated energy microgrid based on improved TD3 algorithm, characterized in that: The following steps are involved: Step 1: Establish corresponding mathematical models for the components of the electric-hydrogen-heat integrated energy system: electrolyzer, combustion battery, hydrogen storage tank and compressor; Step 2: Based on the Markov decision process, establish the Markov decision process for dispatch optimization of electric, thermal and hydrogen integrated energy microgrid: ;in, It is the possible state space of each power component in the microgrid environment; The action space includes various operations and dispatch strategies adopted by the microgrid system; is the state transition probability, describing the state Take an action After that, the probability distribution of the microgrid system transferring from one state to another; is an immediate reward, representing the Take an action After that, the agent gets the reward; is a discount factor between 0 and 1, indicating the importance of future rewards; Step 3: Train the agent based on the priority experience replay TD3 algorithm to obtain a day-ahead dispatch strategy that minimizes the life depreciation cost of the electrolyzer and lead-acid battery, the carbon cost and electricity transaction cost of the microgrid system, and the cost of natural gas purchased by the microgrid system from the natural gas supplier.

2. The method for optimizing the dispatching of electric, hydrogen and heat integrated energy microgrid based on the improved TD3 algorithm according to claim 1 is characterized in that: In step 1, the mathematical model is as follows: 1) Electrolyzer mathematical model: The relationship between the hydrogen production rate of the alkaline electrolyzer and the external current is expressed as: ； In the formula, express The hydrogen production rate of the electrolyzer at the moment, is the Faraday constant, represents the number of electrons produced in each reaction, express The external current of the electrolytic cell at the moment, Indicates the number of electrolytic cells, represents the Faraday efficiency; The calculation formula of the cell voltage is: ； In the formula, represents the standard free energy of liquid water, represents the empirical temperature coefficient, , They represent the ohmic resistance parameters of the electrolytic cell, and are the empirical fitting parameters of the electrolyzer overvoltage model, and and is the basic activation overvoltage constant, and is the temperature linear correction coefficient, and is the temperature secondary correction coefficient; represents the electrolysis module area, Indicates the working temperature of the electrolytic cell; The output power of the electrolyzer is expressed as: ； In the formula, is the efficiency of the electrolyzer; 2) Combustion battery mathematical model: The electrolyzer and lithium battery form a microgrid composite energy storage system, and the voltage of the proton exchange membrane fuel cell stack is expressed as: ； In the formula, is the terminal voltage of the fuel cell, is the equivalent impedance inside the fuel cell, is the fuel cell current, , and is the empirical coefficient, is the number of fuel cells connected in series; is the electromotive force of the fuel cell; The fuel cell hydrogen consumption rate is expressed as: ； In the formula, The hydrogen utilization efficiency of the fuel cell; 3) Mathematical model of hydrogen storage tank and compressor: The internal pressure of the hydrogen storage tank is expressed as: ； In the formula, express The internal pressure of the hydrogen storage tank at the moment, is the gas constant, Indicates the temperature of the hydrogen storage tank. Indicates the volume of the hydrogen storage tank, Indicates the hydrogen consumption rate of the heat load; The relative capacity of hydrogen at this moment is obtained by the following formula: ； In the formula, for The relative capacity of hydrogen at the time, The maximum pressure that the hydrogen storage tank can withstand; The compressor model is expressed as: ； In the formula, express The volume of hydrogen output by the compressor at the moment, express The volume of hydrogen passing through the compressor at any given moment, and Represent the compressor output and input hydrogen pressure respectively, Indicates the compressor efficiency.

3. The method for optimizing the dispatching of electric, hydrogen and heat integrated energy microgrid based on the improved TD3 algorithm according to claim 2 is characterized in that: In step 2, the Markov decision process The specific expressions are as follows: 1) State Space : Microgrids The state of the microgrid system of electric, hydrogen and heat integrated energy at the moment , expressed as: ； In the formula, , and Respectively indicate scheduling Photovoltaic and wind power output power and electrical load at all times, and Respectively, in scheduling The price of natural gas and carbon trading prices at the moment; Indicates the state of charge of the lead-acid battery; express The capacity of the fuel cell at any moment; express Microgrid heat load demand at every moment; express Real-time electricity prices at the moment; 2) Action Space : Operation of the composite energy storage system , expressed as: ； In the formula, express Lead-acid battery power at the moment; express Fuel cell power at the time; express Electrolyzer power at the moment; 3) Instant Rewards : The reward function of reinforcement learning is set to the opposite of the microgrid day-ahead optimization scheduling objective function to achieve directional consistency, which is expressed as: ； In the formula, represents the objective function of the day-ahead optimization dispatch of the microgrid; 4) State transition probability : In Status Take an action After that, the probability distribution of the microgrid system transferring from one state to another is expressed as: ； In the formula, express The state space at time, express Action space at any moment.

4. The method for optimizing the dispatching of electric, hydrogen and heat integrated energy microgrid based on the improved TD3 algorithm according to claim 3 is characterized in that: In step 3, the objective of minimizing the sum of the life depreciation cost of the microgrid composite energy storage system, the carbon cost and electricity transaction cost of the microgrid system, and the cost of natural gas purchased by the microgrid system from the natural gas supplier is to configure the microgrid day-ahead optimization scheduling objective function: ； In the formula, and Respectively The life loss cost of electrolyzers and lead-acid batteries at all times, represents the carbon cost of the microgrid system, represents the natural gas cost purchased by the microgrid system from the natural gas supplier, express The interaction power between the microgrid and the main grid at any time, express Real-time electricity prices at any given moment.

5. The method for optimizing the dispatching of electric, hydrogen and heat integrated energy microgrid based on the improved TD3 algorithm according to claim 3 is characterized in that: In step 3, the optimization scheduling problem of the electric, hydrogen and heat integrated energy microgrid satisfies the following constraints: 1) Electrical load balance: ； In the formula, express Fuel cell power at the time; 2) Heat load balance: ； In the formula, and Respectively The heat energy provided by hydrogen and natural gas at all times, express Microgrid heat load demand at every moment; 3) Power constraints of each power component: ； In the formula, and Respectively represent the maximum and minimum state of charge of the lead-acid battery; and Respectively represent the maximum and minimum output of lead-acid batteries; express The capacity of the fuel cell at the moment, and Respectively represent the upper and lower limits of fuel cell capacity, and Respectively represent the upper and lower limits of fuel cell power, and Respectively represent the upper and lower limits of the electrolytic cell power, express The volume of hydrogen passing through the compressor at any given moment, Indicates the upper limit of the volume of hydrogen that can pass through the compressor. express The volume of hydrogen output by the compressor at the moment, Indicates the upper limit of the hydrogen volume output by the compressor.

6. The method for optimizing the dispatching of electric, hydrogen and heat integrated energy microgrid based on the improved TD3 algorithm according to claim 3 is characterized in that: In step 3, the agent training based on the priority experience playback TD3 algorithm is specifically as follows: In the process of training the agent based on the Priority Experience Replay TD3 algorithm, the algorithm alternately performs strategy evaluation, strategy improvement, and Priority Experience Replay; Among them, in the strategy evaluation stage, the state-action value is calculated, and its Q function is expressed by the Bellman equation as follows: ； In the formula, Represents the state-action value in the value evaluation Q network; express Instant rewards at the moment and Discount factor at time State-action value The expectation of the sum of After parameterizing the Q function using a neural network, the Q function is approximated by minimizing the Bellman residual: ； In the formula, is the minimized Bellman residual, and Represent the parameters of the value assessment Q network and the Target Q network respectively; In the strategy improvement stage, the Q function is parameterized using a neural network to minimize the objective function The gradient used to update the network parameters is: ； In the formula, Represents the policy network parameters Derivation; For expectations, Expressing the The state-action value in the Q network in the training rounds Actions in Seeking guidance, Expressing the Policy network in training rounds Parameters in Derivation; In the process of training the agent with the improved TD3 algorithm, the target value is evaluated through two Target Q networks with different initial parameters, and the smaller value is selected as the target value, so the minimized Bellman residual is corrected to: ； In the formula, express The rewards of the moment, express The smaller value of the moment value assessment; express The action after adding disturbance at every moment; represents the number of value evaluation Q networks; express The value assessment of the moment; Use target policy smoothing regularization to enhance the stability of the policy and smooth the Q function; that is, when calculating the Bellman residual, the state at the next moment Actions taken Will be selected as: ； In the formula, Represents the Target policy network; For the added noise, is the action after adding disturbance.

7. The method for optimizing the dispatching of electric, hydrogen and heat integrated energy microgrid based on the improved TD3 algorithm according to claim 6 is characterized in that: In the Priority Experience Replay TD3 algorithm, the concept of sample priority is introduced: define each sample in the experience pool The probability of being sampled by the agent and the absolute value of the time difference error The correct sample sampling probability matrix for positive correlation and immediate reward replenishment is expressed as: ； In the formula, represents the sample sampling probability matrix, represents the probability of each sample being sampled, , is a hyperparameter; is a small positive constant; is the time difference error; Indicates that there is Samples; Weighted importance sampling rate correction: The weighted importance sampling rate is introduced to balance the uneven distribution of samples and ensure that both high-priority and low-priority samples receive appropriate attention. The importance sampling coefficient is defined as follows: ； In the formula, is the total number of samples in the experience pool, is a hyperparameter that controls the balance between sampling priority and weights in experience replay and smoothes high-variance importance sampling weights.

Images