Low-carbon economic operation optimization method for electricity-hydrogen-heat comprehensive energy system

Abstract

The invention discloses a low-carbon economic operation optimization method for an electricity-hydrogen-heat comprehensive energy system, and belongs to the crossing field of intelligent energy systems and artificial intelligence, and the method comprises the following steps: 1, building a low-carbon economic dual-target operation optimization problem model of the electricity-hydrogen-heat comprehensive energy system; 2, constructing a dual-objective operation optimization problem solving framework with a third-generation non-dominated sorting genetic algorithm as a core; 3, modeling a population evolution decision problem in the genetic algorithm into a Markov decision process, and constructing a deep reinforcement learning agent and population evolution environment interaction mechanism; 4, training the intelligent agent by adopting a double-depth Q network algorithm, and outputting a genetic action according to an environment state; 5, obtaining a new filial generation in combination with a selection strategy associated with the reference points and genetic actions output by the intelligent agent; the step 4 and the step 5 are iteratively evolved until the maximum iteration generation number is reached, and the operation cost can be reduced under the same carbon emission condition.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of intelligent energy systems and artificial intelligence, specifically a method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system. Background Technology

[0002] As a new type of energy system, the integrated electric-hydrogen-heat energy system can achieve synergistic optimization of multiple energy forms by integrating hydrogen electrolysis, hydrogen fuel cell power generation, electrothermal conversion and multiple types of energy storage systems, providing a new technical path for the low-carbon economic operation of energy systems.

[0003] However, optimizing the operation of an integrated electricity-hydrogen-heat energy system faces numerous challenges. First, the system comprises multiple energy forms and energy conversion devices, with complex coupling relationships between subsystems, resulting in a highly nonlinear overall system. Second, there is often a conflict between the goals of reducing operating costs and reducing carbon emissions, requiring a reasonable balance to be struck. Furthermore, system operation must consider constraints such as the start-up and shutdown states of various devices, ramp-up constraints, and the charging and discharging characteristics of the energy storage system.

[0004] For multi-objective optimization problems, existing solution methods mainly include traditional mathematical programming methods and evolutionary algorithms. Traditional mathematical programming methods include weighted sum methods, While constraint methods offer high computational accuracy, they often only yield a single optimal solution, failing to comprehensively reflect the trade-offs between multiple objectives. Evolutionary algorithms, such as non-dominated sorting genetic algorithms and multi-objective particle swarm optimization, can generate a series of non-dominated solutions in a single run. However, their genetic operations typically employ fixed crossover and mutation strategies, failing to adaptively adjust operational parameters based on population characteristics during optimization. This can easily lead to local optima or slow convergence when exploring complex Pareto fronts.

[0005] Existing reinforcement learning methods have been applied to solving single-objective optimization problems, but their application in multi-objective optimization is still insufficient. In particular, how to organically combine reinforcement learning with evolutionary algorithms, and utilize the adaptive decision-making ability of reinforcement learning to guide the genetic operation selection of evolutionary algorithms, thereby improving the efficiency and solution quality of multi-objective optimization, is a direction worthy of in-depth research.

[0006] In summary, in the dual-objective optimization problem of low-carbon economic operation of the integrated energy system of electricity, hydrogen, and heat, how to design an algorithm that can adaptively adjust the genetic strategy, efficiently explore the Pareto front, and generate a uniformly distributed non-dominated solution set has become an urgent technical problem to be solved. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method for optimizing the low-carbon and economical operation of integrated electricity-hydrogen-heat energy systems. This invention fully considers the complex coupling relationships between electricity, heat, and hydrogen energy in integrated electricity-hydrogen-heat energy systems, as well as the trade-off between the conflicting objectives of operating costs and carbon emissions. Based on this, it solves the technical problem of existing multi-objective optimization algorithms having fixed genetic operation strategies and being unable to adaptively adjust according to population characteristics. It also overcomes the shortcomings of traditional methods in exploring the Pareto front, such as low efficiency and difficulty in generating uniformly distributed non-dominated solution sets, thus providing a comprehensive and efficient optimization scheme for system operation decisions.

[0008] To achieve the above objectives, the present invention employs the following technical solution:

[0009] This invention provides a method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system, comprising:

[0010] Step 1: Establish a dual-objective optimization model for the low-carbon economic operation of the integrated electricity-hydrogen-heat energy system;

[0011] Step 2: Construct a bi-objective operational optimization problem-solving framework based on the third-generation non-dominated sorting genetic algorithm;

[0012] Step 3: Model the population evolution decision problem in the genetic algorithm as a Markov decision process, and construct an interaction mechanism between the deep reinforcement learning agent and the population evolution environment;

[0013] Step 4: The agent is trained using a dual-deep Q-network algorithm and outputs genetic actions based on the environmental state;

[0014] Step 5: Combine the selection strategy associated with the reference point and the genetic actions output by the agent to obtain new offspring;

[0015] Step 6: Iterate through Steps 4 and 5 above until the maximum number of iterations is reached.

[0016] Furthermore, the expression for minimizing the operating cost and carbon emissions of the aforementioned integrated electricity-hydrogen-heat energy system is as follows:

[0017] (1),

[0018] (2),

[0019] (3),

[0020] (4),

[0021] (5),

[0022] (6)，

[0023] (7)，

[0024] (8)，

[0025] (9)，

[0026] (10)，

[0027] (11)，

[0028] (12)，

[0029] (13)，

[0030] (14)，

[0031] (15)，

[0032] (16)，

[0033] (17)，

[0034] (18)，

[0035] (19)，

[0036] (20)，

[0037] (21)，

[0038] (22)，

[0039] (23)，

[0040] (24)，

[0041] (25)，

[0042] (26)，

[0043] (27)，

[0044] (28)，

[0045] (29),

[0046] (30),

[0047] (31),

[0048] (32),

[0049] in: This represents the operating cost within time period T. This represents the carbon emission cost within time period T, where T represents the total number of optimization time slots. This indicates the cost reduction of photovoltaics in time slot t. This indicates the operating costs of electrolyzers and fuel cells. and These represent the costs of purchasing hydrogen and electricity, respectively. This represents the power purchased from the grid in time slot t. This indicates the amount of hydrogen purchased from the hydrogen market in time slot t. , These represent the carbon emission factor of grid electricity and the carbon emission factor of market hydrogen, respectively. This represents the cost reduction factor for photovoltaics. This indicates the photovoltaic power reduction in time slot t; , and Let represent the operating cost coefficient, startup cost coefficient, and shutdown cost coefficient of device x, respectively. and These represent the electrolyzer and the fuel cell, respectively. , and These represent the running state, startup state, and shutdown state of device x, respectively. , and These represent the operating status, start-up status, and shutdown status of the electrolytic cell, respectively. , and These represent the operating status, startup status, and shutdown status of the fuel cell, respectively. , These represent the electricity price and hydrogen market price in time slot t, respectively. Indicates the state of charge of the electrical storage in time slot t. and These represent the charging power and discharging power of the energy storage in time slot t, respectively. Indicates the time slot length. and These represent the charging efficiency and discharging efficiency of energy storage, respectively. Indicates the maximum state of charge of electrical energy storage; Indicates the maximum charging and discharging power of the electrical energy storage; This indicates the energy level of the time slot thermal storage system. and These represent the thermal energy storage charging power and thermal energy storage releasing power in time slot t, respectively. and These represent the thermal energy storage charging efficiency and the thermal energy storage releasing efficiency, respectively. This indicates the maximum charge and discharge power of thermal energy storage; Indicates the maximum energy level of the thermal storage system; This indicates the hydrogen storage capacity of the hydrogen tank in time slot t. This represents the input power of the electrolytic cell in time slot t. This indicates the amount of hydrogen input to the fuel cell in time slot t; Indicates the maximum hydrogen storage capacity of the hydrogen tank; This indicates the maximum input power of the electrolytic cell; This represents the electrical power output of the fuel cell in time slot t. This indicates the maximum output power of the fuel cell. and These represent the efficiency of hydrogen-to-electricity and hydrogen-to-heat conversion in fuel cells, respectively. This indicates the maximum amount of hydrogen that can be purchased from the hydrogen market; This represents the input power of the electric boiler in time slot t; This represents the photovoltaic power generation capacity in time slot t. This represents the electrical load demand in time slot t. This represents the heat load demand in time slot t. Indicates the efficiency of the electrolytic cell; Indicates the efficiency of the electric boiler; Let represent the maximum input power of the electric boiler, where the decision variable is . , , , , , , , , , , and .

[0050] Furthermore, the core framework of the third-generation non-dominated sorting genetic algorithm is as follows:

[0051] Step 2-1: Initialize the population size to First generation population , As a candidate solution, , , , , , , , , , , , ,in For T time slots, the power purchased from the grid For T time slots, the amount of hydrogen purchased from the hydrogen market, For the input power of T time-slot electrolytic cells, For the hydrogen input of the fuel cell in T time slots, For the input power of the electric boiler in T time slots, For the charging power of T time slots of electrical energy storage, For the energy storage discharge power of T time slots, For T time slots of thermal energy storage charging power, For T time slots of thermal energy storage heat release power, The operating status of T time-slot electrolytic cells. The operating states of a fuel cell over T time slots. For T time slots, the photovoltaic power reduction is achieved;

[0052] Step 2-2: Select a random parent generation Perform a crossover operation to generate offspring. The crossover operation expression is as follows:

[0053] (33),

[0054] in: Cross coefficient, For offspring;

[0055] Steps 2-3: For offspring individuals Perform mutation; the mutation operation expression is as follows:

[0056] (34),

[0057] in: As a variable intensity control factor, With a mean of 0 and a variance of Gaussian distribution;

[0058] Steps 2-4: Transfer the parent population With offspring population merged into Then layered into Perform a quick nondominated sort:

[0059] (35),

[0060] (36),

[0061] in: This is the first Pareto front. This represents the (k+1)th non-dominated frontier; For two solutions X and Y, X dominates Y, denoted as . If and only if: , ,

[0062] Among them and Let X represent the operating cost and carbon emissions corresponding to the solution X of the optimization problem, respectively. and Let Y represent the operating cost and carbon emission values ​​corresponding to the solution Y of the optimization problem, respectively.

[0063] Steps 2-5: Generate uniformly distributed reference points The distances from the normalized individual target values ​​to each reference point are calculated as follows:

[0064] (37),

[0065] Where: A represents the number of reference points; and These represent the normalized target values ​​related to operating costs and carbon emissions, respectively.

[0066] Steps 2-6: Select reference points The nearest solution is used to replenish the next generation of the population;

[0067] Step 2-7: Repeat steps 2-2 to 2-6 until the maximum number of iterations G is reached.

[0068] Furthermore, based on the traditional genetic algorithm framework, reinforcement learning is introduced to intelligently select genetic operations. The population evolution process is viewed as a Markov decision process, with its state space containing information such as the current iteration progress, the objective function value of the parent individuals, and dominance relationships, comprehensively reflecting the characteristics of population evolution. The action space is designed with various mutation and crossover operations with different parameters, allowing the agent to select different genetic strategies according to the evolutionary stage. The reward mechanism comprehensively considers whether the offspring dominates the parent, the degree of improvement in the objective value, and the distance from the existing solution set, guiding the agent to learn more effective genetic operation selection strategies. The environment state of the Markov decision process is as follows:

[0069] (38),

[0070] in: Let be the algebra that the current algorithm has iterated to; The total number of algebras required for the algorithm iterations; , , , These are the target values ​​for operating costs of parent generation 1, carbon emissions of parent generation 1, operating costs of parent generation 2, and carbon emissions of parent generation 2, respectively. , These are the normalization factors for operating cost target values ​​and carbon emission target values, respectively. As an indicator of dominance, when Dominate Take 1 at time. Dominate The time interval is 0; the action space for multiple genetic strategies is:

[0071] (39),

[0072] in: Genetic strategies for variation controlled by different variation intensity factors; A hybrid linear α-crossover genetic strategy controlled by different crossover coefficients α is proposed; each operation has differentiated effects based on different evolutionary stages and population characteristics; a multi-level reward mechanism considering dominance relationships and improvement rates is as follows:

[0073] (40)

[0074] in: This is the i-th objective value of the offspring; , These are the i-th target values ​​of parent generation 1 and parent generation 2, respectively; The improvement rate of the offspring relative to the parent generation. The minimum Euclidean distance is used; the improvement rate term quantifies the degree of optimization of the offspring relative to the parent. To adjust the parameters, where Controlling and controlling the intensity of rewards and improving rewards. Control the magnitude of similar punishments. Control the decay rate of the similarity penalty.

[0075] Furthermore, to realize this intelligent decision-making process, a dual-depth Q-network is constructed, comprising an online Q-network and a target Q-network, both of which are multi-layer neural networks. The online Q-network is a mapping from the environmental state to the Q-value, using... Parameterization is performed, aligning the number of input layer neurons to the dimension of the environment state and the number of output layer neurons to the dimension of the action space. The network uses Rectified Linear Units (ReLU) as the activation function for the hidden layers. A target Q-network is used to stabilize the learning process; its structure is identical to that of the online Q-network. It is parameterized, and its parameters are periodically updated from an online network.

[0076] Furthermore, the training process of the dual-deep Q-network is mainly implemented through two mechanisms: experience replay and target network update. The experience replay mechanism utilizes the transfer samples obtained from the agent's interaction with the environment. Stored in the playback buffer pool, where This represents the state of the algorithm at generation g. This represents the action selected in the g-th generation. Indicates the selection action in the g-th generation. The rewards received This indicates the next state after the transition.

[0077] During the action selection process, the agent adopts actions based on its current state. - Greedy strategy for decision-making: based on probability Random exploration, with a probability of 1- Choose the action with the highest Q value. This strategy balances exploration and utilization, ensuring sample diversity.

[0078] During training, a batch of samples is randomly sampled for learning. This mechanism breaks the correlation between samples, improving learning efficiency and stability. Network parameter updates employ the Temporal Difference (TD) algorithm, and its target Q-value is calculated as follows:

[0079] (41),

[0080] in: This is a discount factor used to balance the importance of immediate and future rewards. For the target network, It represents the set of all possible actions at the next moment. The loss function of an online network is defined as the mean square value of the TD error, calculated as follows:

[0081] (42),

[0082] in: For online networks, These are the network parameters. Parameter optimization uses the stochastic gradient descent algorithm, calculated by retrieving the loss function with respect to the parameters. Update the gradient ,in For learning rate, This indicates the loss function with respect to the parameters. The gradient. Parameters of the target network. Then it replicates from the online network via a soft update: ,in To maintain the update rate, this soft update mechanism slowly updates the target network parameters, ensuring the stability of the training process and avoiding drastic fluctuations in the value function estimation.

[0083] Furthermore, reinforcement learning and genetic algorithms are combined to construct a complete interactive optimization framework:

[0084] Step 6-1: Initialize the population Population size is Simultaneously initialize the parameters of the dual-depth Q network. and Set up the experience replay pool D;

[0085] Step 6-2: Select parent individuals Constructing the environmental state of a Markov decision process ;

[0086] Step 6-3: The dual-deep Q-network agent selects actions according to the ε-greedy strategy. Execute the corresponding mutation or crossover operator to generate offspring;

[0087] Step 6-4: Evaluate offspring fitness and calculate reward values. Transfer samples Store in experience replay pool D;

[0088] Step 6-5: Sample batch data from D and update the online network parameters of the dual deep Q network using the temporal difference algorithm. The target network is updated synchronously every fixed number of steps. ;

[0089] Step 6-6: Merge the parent and offspring populations, perform non-dominated sorting and reference point association, and select the next generation population;

[0090] Step 6-7: Repeat steps 6-2 to 6-6 until the maximum number of iterations G is reached.

[0091] Overall, this invention comprises three main modules in terms of system architecture: a data acquisition and processing module, an intelligent optimization decision-making module, and a system collaborative control module. The data acquisition and processing module is responsible for providing the necessary input information for optimization decisions; the intelligent optimization decision-making module includes a multi-objective optimization sub-module and a reinforcement learning sub-module, which are responsible for finding Pareto optimal solutions and adaptively adjusting optimization strategies, respectively; the system collaborative control module is responsible for receiving and executing optimization decisions to maintain the balanced operation of the multi-energy flow system.

[0092] Beneficial effects: A method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system, compared with existing technologies:

[0093] (1) By combining the dual-depth Q network with the third-generation non-dominated sorting genetic algorithm, adaptive selection of genetic operations is realized, overcoming the limitation of fixed strategies in traditional algorithms.

[0094] (2) The multi-level reward mechanism designed enables the agent to adaptively select the optimal strategy according to the evolutionary stage of the population, which significantly improves the convergence speed of the algorithm.

[0095] (3) By combining reference point association and intelligent genetic operations, a more uniform and better Pareto solution set is generated. Attached Figure Description

[0096] Figure 1 This is a flowchart of an optimization method for low-carbon economic operation of an integrated energy system based on electricity, hydrogen, and heat, as proposed in this invention.

[0097] Figure 2 This is a comparison chart of Pareto curves of the method of this invention and other methods;

[0098] Figure 3 This describes the economic performance of the method of this invention and other methods under the condition of a carbon emission constraint of 136,500 kg at the Pareto front. Detailed Implementation

[0099] To clearly illustrate the technical solution and innovative advantages of the present invention, specific embodiments of the present invention will now be described in conjunction with the accompanying drawings. It should be specifically noted that these embodiments are for illustrative purposes only and do not constitute a limitation on the claims of the present invention.

[0100] Example 1:

[0101] like Figure 1 As shown in the flowchart, this invention provides a method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system, which includes the following steps:

[0102] 1. Establish a model for minimizing the operating cost and carbon emissions of an integrated electricity-hydrogen-heat energy system, with the following expression:

[0103] (1),

[0104] (2),

[0105] (3),

[0106] (4),

[0107] (5)，

[0108] (6)，

[0109] (7)，

[0110] (8)，

[0111] (9)，

[0112] (10)，

[0113] (11)，

[0114] (12)，

[0115] (13)，

[0116] (14)，

[0117] (15)，

[0118] (16)，

[0119] (17)，

[0120] (18)，

[0121] (19)，

[0122] (20)，

[0123] (21)，

[0124] (22)，

[0125] (23)，

[0126] (24)，

[0127] (25)，

[0128] (26)，

[0129] (27),

[0130] (28),

[0131] (29),

[0132] (30),

[0133] (31),

[0134] (32),

[0135] in: This represents the operating cost within time period T. This represents the carbon emission cost within time period T, where T represents the total number of optimization time slots. This indicates the cost reduction of photovoltaics in time slot t. This indicates the operating costs of electrolyzers and fuel cells. and These represent the costs of purchasing hydrogen and electricity, respectively. This represents the power purchased from the grid in time slot t. This indicates the amount of hydrogen purchased from the hydrogen market in time slot t. , These represent the carbon emission factor of grid electricity and the carbon emission factor of market hydrogen, respectively. This represents the cost reduction factor for photovoltaics. This indicates the photovoltaic power reduction in time slot t; , and Let represent the operating cost coefficient, startup cost coefficient, and shutdown cost coefficient of device x, respectively. and These represent the electrolyzer and the fuel cell, respectively. , and These represent the running state, startup state, and shutdown state of device x, respectively. , and These represent the operating status, start-up status, and shutdown status of the electrolytic cell, respectively. , and These represent the operating status, startup status, and shutdown status of the fuel cell, respectively. , These represent the electricity price and hydrogen market price in time slot t, respectively. Indicates the state of charge of the electrical storage in time slot t. and These represent the charging power and discharging power of the energy storage in time slot t, respectively. Indicates the time slot length. and These represent the charging efficiency and discharging efficiency of energy storage, respectively. Indicates the maximum state of charge of electrical energy storage; Indicates the maximum charging and discharging power of the electrical energy storage; This indicates the energy level of the time slot thermal storage system. and These represent the thermal energy storage charging power and thermal energy storage releasing power in time slot t, respectively. and These represent the thermal energy storage charging efficiency and the thermal energy storage releasing efficiency, respectively. This indicates the maximum charge and discharge power of thermal energy storage; Indicates the maximum energy level of the thermal storage system; This indicates the hydrogen storage capacity of the hydrogen tank in time slot t. This represents the input power of the electrolytic cell in time slot t. This indicates the amount of hydrogen input to the fuel cell in time slot t; Indicates the maximum hydrogen storage capacity of the hydrogen tank; This indicates the maximum input power of the electrolytic cell; This represents the electrical power output of the fuel cell in time slot t. This indicates the maximum output power of the fuel cell. and These represent the efficiency of hydrogen-to-electricity and hydrogen-to-heat conversion in fuel cells, respectively. This indicates the maximum amount of hydrogen that can be purchased from the hydrogen market; This represents the input power of the electric boiler in time slot t; This represents the photovoltaic power generation capacity in time slot t. This represents the electrical load demand in time slot t. This represents the heat load demand in time slot t. Indicates the efficiency of the electrolytic cell; Indicates the efficiency of the electric boiler; Let represent the maximum input power of the electric boiler, where the decision variable is . , , , , , , , , , , and .

[0136] 2. Construct a bi-objective operational optimization problem-solving framework based on the third-generation non-dominated sorting genetic algorithm;

[0137] Furthermore, the core framework of the third-generation non-dominated sorting genetic algorithm is as follows:

[0138] Step 2-1: Initialize the population size to First generation population , As a candidate solution, , , , , , , , , , , , ,in For T time slots, the power purchased from the grid For T time slots, the amount of hydrogen purchased from the hydrogen market, For the input power of T time-slot electrolytic cells, For the hydrogen input of the fuel cell in T time slots, For the input power of the electric boiler in T time slots, For the charging power of T time slots of electrical energy storage, For the energy storage discharge power of T time slots, For T time slots of thermal energy storage charging power, For T time slots of thermal energy storage heat release power, The operating status of T time-slot electrolytic cells. The operating states of a fuel cell over T time slots. For T time slots, the photovoltaic power reduction is achieved;

[0139] Step 2-2: Select a random parent generation Perform a crossover operation to generate offspring. The crossover operation expression is as follows:

[0140] (33),

[0141] in: Cross coefficient, For offspring;

[0142] Steps 2-3: For offspring individuals Perform mutation; the mutation operation expression is as follows:

[0143] (34),

[0144] in: As a variable intensity control factor, With a mean of 0 and a variance of Gaussian distribution;

[0145] Steps 2-4: Transfer the parent population With offspring population merged into Then layered into Perform a quick nondominated sort:

[0146] (35),

[0147] (36),

[0148] in: This is the first Pareto front. This represents the (k+1)th non-dominated frontier; For two solutions X and Y, X dominates Y, denoted as . If and only if: , ,

[0149] Among them and Let X represent the operating cost and carbon emissions corresponding to the solution X of the optimization problem, respectively. and Let Y represent the operating cost and carbon emission values ​​corresponding to the solution Y of the optimization problem, respectively.

[0150] Steps 2-5: Generate uniformly distributed reference points The distances from the normalized individual target values ​​to each reference point are calculated as follows:

[0151] (37),

[0152] Where: A represents the number of reference points; and These represent the normalized target values ​​related to operating costs and carbon emissions, respectively.

[0153] Steps 2-6: Select reference points The nearest solution is used to replenish the next generation of the population;

[0154] Step 2-7: Repeat steps 2-2 to 2-6 until the maximum number of iterations G is reached.

[0155] 3. Model the population evolution decision problem in genetic algorithms as a Markov decision process;

[0156] In the embodiments of this invention, the environmental state of the Markov decision process is as follows:

[0157] (38),

[0158] in: Let be the algebra that the current algorithm has iterated to; The total number of algebras required for the algorithm iterations; , , , These are the target values ​​for operating costs of parent generation 1, carbon emissions of parent generation 1, operating costs of parent generation 2, and carbon emissions of parent generation 2, respectively. , These are the normalization factors for operating cost target values ​​and carbon emission target values, respectively. As an indicator of dominance, when Dominate Take 1 at time. Dominate The time interval is 0; the action space for multiple genetic strategies is:

[0159] (39),

[0160] in: Genetic strategies for variation controlled by different variation intensity factors; A hybrid linear α-crossover genetic strategy controlled by different crossover coefficients α is proposed; each operation has differentiated effects based on different evolutionary stages and population characteristics; a multi-level reward mechanism considering dominance relationships and improvement rates is as follows:

[0161] (40)

[0162] in: This is the i-th objective value of the offspring; , These are the i-th target values ​​of parent generation 1 and parent generation 2, respectively; The improvement rate of the offspring relative to the parent generation. The minimum Euclidean distance is used; the improvement rate term quantifies the degree of optimization of the offspring relative to the parent. To adjust the parameters, where Controlling and controlling the intensity of rewards and improving rewards. Control the magnitude of similar punishments. Control the decay rate of the similarity penalty.

[0163] 4. Establish an interaction mechanism between the agent and the optimization environment. The agent views the multi-objective optimization process of the integrated electricity-hydrogen-heat energy system as an interactive evolutionary environment. In each generation of population evolution, the agent, acting as a genetic strategy decision-maker, observes the state... The agent moves from an action space containing multiple genetic strategies. Choose one action After receiving the agent's action, the environment performs corresponding genetic operations to generate offspring individuals and calculates their operating costs and carbon emissions using a comprehensive energy system model. Subsequently, the environment calculates a reward value based on the improvement of the offspring compared to their parent. This forms a complete interactive feedback.

[0164] 5. By replaying experiences and updating network parameters, the agent is guided to learn the optimal genetic strategy, thereby achieving synergistic optimization of reinforcement learning and genetic algorithms;

[0165] In an embodiment of the present invention, the dual-depth Q-network structure includes an online Q-network and a target Q-network, both of which are multi-layer neural networks. The online Q-network is a mapping from the environmental state to the Q-value, using... Parameterization is performed, aligning the number of input layer neurons to the dimension of the environment state and the number of output layer neurons to the dimension of the action space. The network uses Rectified Linear Units (ReLU) as the activation function for the hidden layers. A target Q-network is used to stabilize the learning process; its structure is identical to that of the online Q-network. It is parameterized, and its parameters are periodically updated from an online network.

[0166] In the embodiments of this invention, the training process of the dual-deep Q-network is mainly implemented through two mechanisms: experience replay and target network update. The experience replay mechanism utilizes the transfer samples obtained from the interaction between the agent and the environment. Stored in the playback buffer pool, where This represents the state of the algorithm at generation g. This represents the action selected in the g-th generation. Indicates the selection action in the g-th generation. The rewards received This indicates the next state after the transition.

[0167] During the action selection process, the agent adopts actions based on its current state. - Greedy strategy for decision-making: based on probability Random exploration, with a probability of 1- Choose the action with the highest Q value. This strategy balances exploration and utilization, ensuring sample diversity.

[0168] During training, a batch of samples is randomly sampled for learning. This mechanism breaks the correlation between samples, improving learning efficiency and stability. Network parameter updates employ the Temporal Difference (TD) algorithm, and its target Q-value is calculated as follows:

[0169] (41),

[0170] in: This is a discount factor used to balance the importance of immediate and future rewards. For the target network, It represents the set of all possible actions at the next moment. The loss function of an online network is defined as the mean square value of the TD error, calculated as follows:

[0171] (42),

[0172] in: For online networks, These are the network parameters. Parameter optimization uses the stochastic gradient descent algorithm, calculated by retrieving the loss function with respect to the parameters. Update the gradient ,in For learning rate, This indicates the loss function with respect to the parameters. The gradient. Parameters of the target network. Then it replicates from the online network via a soft update: ,in This refers to the update rate.

[0173] 6. Combine reinforcement learning with genetic algorithms to build a complete interactive optimization framework;

[0174] Step 6-1: Initialize the population Population size is Simultaneously initialize the parameters of the dual-depth Q network. and Set up the experience replay pool D;

[0175] Step 6-2: Select parent individuals Constructing the environmental state of a Markov decision process ;

[0176] Step 6-3: The dual-deep Q-network agent selects actions according to the ε-greedy strategy. Execute the corresponding mutation or crossover operator to generate offspring;

[0177] Step 6-4: Evaluate offspring fitness and calculate reward values. Transfer samples Store in experience replay pool D;

[0178] Step 6-5: Sample batch data from D and update the online network parameters of the dual deep Q network using the temporal difference algorithm. The target network is updated synchronously every fixed number of steps. ;

[0179] Step 6-6: Merge the parent and offspring populations, perform non-dominated sorting and reference point association, and select the next generation population;

[0180] Step 6-7: Repeat steps 6-2 to 6-6 until the maximum number of iterations G is reached.

[0181] To demonstrate the effectiveness of the method of the present invention, four comparative schemes were introduced.

[0182] Scheme 1 employs a weighted sum method combined with a convex optimization solver. By assigning different weights to operating costs and carbon emission targets, the multi-objective optimization problem is transformed into a single-objective problem. The algorithm constructs a linearized mixed-integer programming model and uses the convex optimization Python package (CVXPY) to call a commercial solver (CPLEX) to solve it, ensuring optimality for the single-objective problem. After completing all weighted combination optimizations, non-dominated solutions are extracted from the solution set to form a Pareto front.

[0183] Scheme 2 employs a multi-objective particle swarm optimization algorithm, extending particle swarm optimization to the multi-objective domain. Each particle represents an energy system scheduling scheme, with velocity and position updates guided by individual and global optima. The algorithm maintains a capacity-constrained external archive of non-dominated solutions and uses crowding distance to maintain population diversity.

[0184] Scheme 3 employs a second-generation non-dominated sorting genetic algorithm, utilizing fast non-dominated sorting to classify individuals into different frontier levels. For individuals within the same frontier, crowding distance is used to maintain diversity. The algorithm uses tournament selection, where individual selection is based on a comprehensive evaluation of non-dominated level and crowding distance.

[0185] Scheme 4 employs a third-generation non-dominated sorting genetic algorithm. By introducing a reference point mechanism, it maintains population diversity and overcomes the limitations of the second-generation non-dominated sorting genetic algorithm in high-dimensional target spaces. The algorithm maintains the same fast non-dominated sorting method, but uses a reference point-based selection strategy instead of crowding distance to obtain a more evenly distributed non-dominated solution set.

[0186] Figure 2 and Figure 3 The performance comparison results of different optimization schemes in the integrated electricity-hydrogen-heat energy system after 600 iterations to full convergence are presented. For example... Figure 2 As shown, the degree of deviation of the curve towards the lower left reflects the quality of the algorithm's performance. The closer the curve is to the lower left corner, the lower the corresponding solution set can achieve both lower operating costs and less carbon emissions. The solution set generated by the scheme of this invention exhibits better distribution uniformity, providing decision-makers with a wealth of alternative solutions. Figure 3 Further quantitative analysis was conducted on the economic performance of each scheme under the condition of the same carbon emissions. The results show that the method of the present invention achieves the best cost control effect compared with all the comparative schemes, reducing operating costs by 3.97%-24.81% and improving the economic operating efficiency of the system. Figure 2 The data points in the image represent the Pareto optimal solutions found. Some solutions are relatively close to each other and overlap on the image.

Claims

1. A method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system, characterized in that, The steps include: Step 1: Establish a dual-objective optimization model for the low-carbon economic operation of the integrated electricity-hydrogen-heat energy system; Step 2: Construct a bi-objective operational optimization problem-solving framework based on the third-generation non-dominated sorting genetic algorithm; Step 3: Model the population evolution decision problem in the genetic algorithm as a Markov decision process, and construct an interaction mechanism between the deep reinforcement learning agent and the population evolution environment; Step 4: The agent is trained using a dual-deep Q-network algorithm and outputs genetic actions based on the environmental state; Step 5: Combine the selection strategy associated with the reference point and the genetic actions output by the agent to obtain new offspring; Step 6: Iterate through Steps 4 and 5 above until the maximum number of iterations is reached.

2. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 1, characterized in that, The expression for minimizing the operating cost and carbon emissions of the integrated electricity-hydrogen-heat energy system in step 1 is as follows: (1)， (2)， (3)， (4)， (5)， (6)， (7)， (8)， (9)， (10)， (11)， (12)， (13)， (14)， (15)， (16)， (17)， (18)， (19)， (20)， (21)， (22)， (23)， (24)， (25)， (26)， (27)， (28)， (29)， (30)， (31)， (32)， in: This represents the operating cost within time period T. This represents the carbon emission cost within time period T, where T represents the total number of optimization time slots. This indicates the cost reduction of photovoltaics in time slot t. This indicates the operating costs of electrolyzers and fuel cells. and These represent the costs of purchasing hydrogen and electricity, respectively. This represents the power purchased from the grid in time slot t. This indicates the amount of hydrogen purchased from the hydrogen market in time slot t. , These represent the carbon emission factor of grid electricity and the carbon emission factor of market hydrogen, respectively. This represents the cost reduction factor for photovoltaics. This indicates the photovoltaic power reduction in time slot t; , and Let represent the operating cost coefficient, startup cost coefficient, and shutdown cost coefficient of device x, respectively. and These represent the electrolyzer and the fuel cell, respectively. , and These represent the running state, startup state, and shutdown state of device x, respectively. , and These represent the operating status, start-up status, and shutdown status of the electrolytic cell, respectively. , and These represent the operating status, startup status, and shutdown status of the fuel cell, respectively. , These represent the electricity price and hydrogen market price in time slot t, respectively. Indicates the state of charge of the electrical storage in time slot t. and These represent the charging power and discharging power of the energy storage in time slot t, respectively. Indicates the time slot length. and These represent the charging efficiency and discharging efficiency of energy storage, respectively. Indicates the maximum state of charge of electrical energy storage; Indicates the maximum charging and discharging power of the electrical energy storage; This indicates the energy level of the time slot thermal storage system. and These represent the thermal energy storage charging power and thermal energy storage releasing power in time slot t, respectively. and These represent the thermal energy storage charging efficiency and the thermal energy storage releasing efficiency, respectively. This indicates the maximum charge and discharge power of thermal energy storage; Indicates the maximum energy level of the thermal storage system; This indicates the hydrogen storage capacity of the hydrogen tank in time slot t. This represents the input power of the electrolytic cell in time slot t. This indicates the amount of hydrogen input to the fuel cell in time slot t; Indicates the maximum hydrogen storage capacity of the hydrogen tank; This indicates the maximum input power of the electrolytic cell; This represents the electrical power output of the fuel cell in time slot t. This indicates the maximum output power of the fuel cell. and These represent the efficiency of hydrogen-to-electricity and hydrogen-to-heat conversion in fuel cells, respectively. This indicates the maximum amount of hydrogen that can be purchased from the hydrogen market; This represents the input power of the electric boiler in time slot t; This represents the photovoltaic power generation capacity in time slot t. This represents the electrical load demand in time slot t. This represents the heat load demand in time slot t. Indicates the efficiency of the electrolytic cell; Indicates the efficiency of the electric boiler; Let represent the maximum input power of the electric boiler, where the decision variable is . , , , , , , , , , , and .

3. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 2, characterized in that, Step 2: The core framework of the third-generation non-dominated sorting genetic algorithm is as follows: Step 2-1: Initialize the population size to First generation population , As a candidate solution, ,in For T time slots, the power purchased from the grid For T time slots, the amount of hydrogen purchased from the hydrogen market, For the input power of T time-slot electrolytic cells, For the hydrogen input of the fuel cell in T time slots, For the input power of the electric boiler in T time slots, For the charging power of T time slots of electrical energy storage, For the energy storage discharge power of T time slots, For T time slots of thermal energy storage charging power, For T time slots of thermal energy storage heat release power, The operating status of T time-slot electrolytic cells. The operating states of a fuel cell over T time slots. For T time slots, the photovoltaic power reduction is achieved; Step 2-2: Select a random parent generation Perform a crossover operation to generate offspring. The crossover operation expression is as follows: (33)， in: Cross coefficient, For offspring; Steps 2-3: For offspring individuals Perform mutation; the mutation operation expression is as follows: (34)， in: As a variable intensity control factor, With a mean of 0 and a variance of Gaussian distribution; Steps 2-4: Transfer the parent population With offspring population merged into Then layered into Perform a quick nondominated sort: (35)， (36)， in: This is the first Pareto front. This represents the (k+1)th non-dominated frontier; For two solutions X and Y, X dominates Y, denoted as . If and only if: , , among them and Let X represent the operating cost and carbon emissions corresponding to the solution X of the optimization problem, respectively. and Let Y represent the operating cost and carbon emission values ​​corresponding to the solution Y of the optimization problem, respectively. Steps 2-5: Generate uniformly distributed reference points The distances from the normalized individual target values ​​to each reference point are calculated as follows: (37)， Where: A represents the number of reference points; and These represent the normalized target values ​​related to operating costs and carbon emissions, respectively. Steps 2-6: Select reference points The nearest solution is used to replenish the next generation of the population; Step 2-7: Repeat steps 2-2 to 2-6 until the maximum number of iterations G is reached.

4. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 3, characterized in that, The environmental state of the Markov decision process in step 3 is as follows: (38)， in: Let be the algebra that the current algorithm has iterated to; The total number of algebras required for the algorithm iterations; , , , These are the target values ​​for operating costs of parent generation 1, carbon emissions of parent generation 1, operating costs of parent generation 2, and carbon emissions of parent generation 2, respectively. , These are the normalization factors for operating cost target values ​​and carbon emission target values, respectively. As an indicator of dominance, when Dominate Take 1 at time. Dominate Take 0 at the time; The action space containing multiple genetic strategies is: (39)， in: Genetic strategies for variation controlled by different variation intensity factors; A hybrid linear α-crossover genetic strategy controlled by different crossover coefficients α; The reward function is designed as follows: (40)， in: This is the i-th objective value of the offspring; , These are the i-th target values ​​of parent generation 1 and parent generation 2, respectively; The improvement rate of the offspring relative to the parent generation. The minimum Euclidean distance is used; the improvement rate term quantifies the degree of optimization of the offspring relative to the parent. To adjust the parameters, where Controlling and controlling the intensity of rewards and improving rewards. Control the magnitude of similar punishments. Control the decay rate of the similarity penalty.

5. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 4, characterized in that, The interaction mechanism between the deep reinforcement learning agent and the population evolution environment in step 3 is designed as follows: The intelligent agent views the multi-objective optimization process of the integrated electricity-hydrogen-heat energy system as an interactive evolutionary environment. In each generation of population evolution, the intelligent agent, as the genetic strategy decision-maker, observes the state... The agent moves from an action space containing multiple genetic strategies. Choose one action After receiving the agent's action, the environment performs corresponding genetic operations to generate offspring individuals, and calculates their operating costs and carbon emissions using a comprehensive energy system model. Subsequently, the environment calculates reward values ​​based on the improvement of the offspring compared to their parents. This forms a complete interactive feedback.

6. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 5, characterized in that, The dual-depth Q-network structure in step 4 includes an online Q-network and a target Q-network, both of which are multi-layer neural networks. The online Q-network is a mapping from the environment state to the Q-value, using... Parameterization is performed, aligning the number of input layer neurons to the dimension of the environment state and the number of output layer neurons to the dimension of the action space. The network uses Rectified Linear Units (ReLU) as the activation function for the hidden layers. A target Q-network is used to stabilize the learning process; its structure is identical to that of the online Q-network. It is parameterized, and its parameters are periodically updated from an online network.

7. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 6, characterized in that, Step 4, the dual-deep Q-network training process, is implemented through two mechanisms: experience replay and target network update. The experience replay mechanism uses the transfer samples obtained from the agent's interaction with the environment. Stored in the playback buffer pool, where This represents the state of the algorithm in the g-th generation. This represents the action selected in the g-th generation. Indicates the selection action in the g-th generation. The rewards received The next state after the transition is represented by the network parameter update algorithm, which uses a random batch of samples for learning during training. The target Q value is calculated as follows: (41)， in: This is a discount factor used to balance the importance of immediate and future rewards. For the target network, It is the set of all possible actions in the next moment. The loss function of the online network is defined as the mean square value of the TD error, and the calculation formula is as follows: (42)， in: For online networks, For the network parameters, stochastic gradient descent is used for parameter optimization, by calculating the loss function with respect to the parameters. Update the gradient ,in For learning rate, This indicates the loss function with respect to the parameters. The gradient, the parameters of the target network Then it replicates from the online network via a soft update: ,in This refers to the update rate.

8. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 7, characterized in that, Step 5: The agent outputs a genetic action to obtain new offspring. This is achieved through a combination of deep reinforcement learning and genetic algorithms. The dual-deep Q-network agent determines the offspring based on the current state. Choose appropriate genetic operations, employing a greedy strategy for action selection: based on probability. Random exploration, with a probability of 1- Choose the action with the highest Q value. After selecting an action, the corresponding genetic operator is executed according to the operation type to generate offspring individuals, and the reward value is calculated. .

9. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 8, characterized in that, Step 6, the optimization method for low-carbon and economical operation of the integrated electricity-hydrogen-heat energy system, is performed according to the following steps: Step 6-1: Initialize the population Population size is Simultaneously initialize the parameters of the dual-depth Q network. and Set up the experience replay pool D; Step 6-2: Select parent individuals Constructing the environmental state of a Markov decision process ; Step 6-3: The dual-deep Q-network agent selects actions according to the ε-greedy strategy. Execute the corresponding mutation or crossover operator to generate offspring; Step 6-4: Evaluate offspring fitness and calculate reward values. Transfer samples Store in experience replay pool D; Step 6-5: Sample batch data from D and update the online network parameters of the dual deep Q network using the temporal difference algorithm. The target network is updated synchronously every fixed number of steps. ; Step 6-6: Merge the parent and offspring populations, perform non-dominated sorting and reference point association, and select the next generation population; Step 6-7: Repeat steps 6-2 to 6-6 until the maximum number of iterations G is reached.

10. The method for optimizing the low-carbon and economical operation of an integrated electricity-hydrogen-heat energy system according to claim 1, characterized in that, The method includes a data acquisition and processing module, an intelligent optimization decision-making module, and a system collaborative control module; The data acquisition and processing module is responsible for collecting real-time operating data and forecast information of the integrated electricity-hydrogen-heat energy system, including photovoltaic power generation forecasts, load demand forecasts, and energy price information, and performing data preprocessing to provide necessary input information for optimization decisions. The intelligent optimization decision-making module includes a multi-objective optimization submodule and a reinforcement learning submodule; the multi-objective optimization submodule is responsible for finding the Pareto optimal solution between cost and carbon emissions; the reinforcement learning submodule is responsible for adaptively adjusting the optimization strategy. The system coordination control module is responsible for receiving and executing optimization decisions, maintaining power balance, thermal balance, and hydrogen energy balance, and realizing the coordinated operation of the multi-energy flow system.

Images