An integrated energy station operation method based on reward redistribution of multi-agent reinforcement learning

Abstract

The present invention discloses an integrated energy station operation method based on multi-agent reinforcement learning reward redistribution. Based on the deep reinforcement learning method, it solves the delayed reward problem in traffic scenarios through reward redistribution. The present invention establishes a hybrid dynamic user equilibrium model including electric vehicles, fuel cell vehicles and fuel vehicles, and designs an integrated energy station operation framework that integrates coupon dynamic decision-making and hydrogen production and storage methods; through the sequential Markov decision process, it describes the cross-period impact of pricing methods on the station-side revenue and traffic congestion, and redistributes the delayed reward into immediate rewards; aiming at the spatio-temporal coupling of the node marginal price of the distribution network and traffic flow, it constructs a collaborative decision-making model considering the operation utility of the distribution network, traffic network equilibrium and integrated energy station operation, and realizes the joint optimization of the spatio-temporal transfer of charging and hydrogen production demands, dynamic alleviation of traffic congestion and improvement of comprehensive energy efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of coordinated optimization of the electric-transportation network, and particularly relates to an operation method of an integrated energy station based on reward redistribution of multi-agent reinforcement learning. Background Art

[0002] The urgent need to reduce greenhouse gas emissions and dependence on fossil energy globally is driving the rapid development of electric vehicle and fuel cell vehicle technologies and accelerating the construction of a clean transportation system. However, the expansion of vehicle electrification scale has significantly deepened the coupling degree between the transportation network and the energy system, and problems such as road network congestion, spatio-temporal imbalance of energy demand, and power grid stability risks have become increasingly prominent. As a hub for energy supply to multiple types of vehicles, the integrated energy station needs to coordinately optimize traffic flow guidance, multi-energy combined supply scheduling, and power distribution network management.

[0003] There are two important gaps in the existing technologies for the current decision-making of integrated energy stations. First, although the existing deep reinforcement learning methods can alleviate information asymmetry, they cannot effectively handle the time deviation between pricing decisions and vehicle user responses. The combined effects of traffic propagation delay and the time-varying congestion of charging and hydrogen refueling facilities, which cause non-linear queuing effects, result in a significant lag in the impact of pricing methods on the station-side revenue and road network status. This delayed reward makes it impossible for the integrated energy station to accurately evaluate the long-term effects of methods such as coupon distribution and hydrogen production and storage. Second, the existing single-time period surface optimization paradigm is difficult to capture cross-time period dynamic correlations, severely restricting the economy and traffic guidance efficiency of the integrated energy station operation. Therefore, there is an urgent need to develop an operation method of an integrated energy station based on reward redistribution of multi-agent reinforcement learning to consider the delay of the integrated energy station reward and the coupling relationship between the transportation network and the integrated energy station in multiple time periods. Summary of the Invention

[0004] Object of the Invention. The technical problem to be solved by the present invention is to provide an operation method of an integrated energy station based on reward redistribution of multi-agent reinforcement learning in view of the deficiencies of the existing technologies. The present invention takes into account the efficient coordination among the power distribution network, multiple types of vehicle users, and the integrated energy station. The reinforcement learning algorithm for reward redistribution calculates the amount of coupons issued by the station and the hydrogen production and storage decisions based on the charging vehicle flow, and is used to handle the reward delay problem caused by the time deviation between the station decision and the vehicle response. By introducing traffic flow-related constraints, coordinated scheduling of the power-transportation-integrated energy station is carried out. The present invention can not only promote the coordination among traffic flow, power distribution network, and vehicles under information barriers, but also help the integrated energy station improve its own benefits based on the dynamic market.

[0005] Technical Solution. To solve the above technical problem, the present invention proposes an operation method of an integrated energy station based on reward redistribution of multi-agent reinforcement learning, and the method includes the following steps:

[0006] Step 1: Obtain the network coefficients and operation coefficients of the power grid model. The network coefficients include the power grid topology, line resistance, and impedance, and the operation coefficients include the power generation coefficient of the generator set, the charge and discharge coefficient of the energy storage system, the photovoltaic inverter coefficient, and the charging station parameters;

[0007] Step 2: Obtain the power grid load demand, photovoltaic output, vehicle travel demand, and historical demand scenario data for average charging and hydrogen refueling;

[0008] Step 3: Establish a traffic model based on dynamic user equilibrium with the state equations of road segments, flow propagation constraints, user proportion allocation constraints, node flow conservation constraints, and utility equations as constraints, and the minimum of user travel and energy charging utility in the transportation network as the objective function. Obtain the energy charging vehicle flow of the nodes where the integrated energy station is located according to this model;

[0009] Step 4: Regarding the integrated energy station as an agent, take the node energy charging vehicle flow obtained in Step 3 as the input of the agent state, and establish an integrated energy station operation model with the utility equations of electricity and hydrogen sales, hydrogen storage volume balance constraints, hydrogen production constraints, power balance constraints, and coupon utility equations as constraints, and the maximum comprehensive income of the integrated energy station as the objective function;

[0010] Step 5: Take the objective function in Step 4 as the reward function of the observation and evaluation network in the multi-agent reinforcement learning reward redistribution algorithm, calculate the reward function of the actual evaluation network through the reward redistribution formula. The reward redistribution formula consists of the reward function and value function of the observation and evaluation network, and maximize the reward function of the actual evaluation network to achieve the maximization of the integrated energy station income.

[0011] Further, in Step 3, the dynamic user equilibrium model is:

[0012] (A-1)

[0013] In the formula, represents the time, represents the road segment, represents the inflow rate of the road segment at time t, represents the passing utility of the road segment at time t.

[0014] Further, in Step 3, the state equation of the road segment is:

[0015] (A-2)

[0016] (A-3)

[0017] (A-4)

[0018] The flow propagation constraint is:

[0019] (A-5)

[0020] (A-6)

[0021] (A-7)

[0022] (A-8)

[0023] The user proportion allocation constraint:

[0024] (A-9)

[0025] (A-10)

[0026] (A-11)

[0027] (A-12)

[0028] The node flow conservation constraint

[0029] (A-13)

[0030] The utility equation:

[0031] (A-14)

[0032] (A-15)

[0033] (A-16)

[0034] (A-17)

[0035] Wherein, represents the inflow rate of vehicles of class at time along path between the origin-destination pair section ; represents the outflow rate of vehicles of class at time along path between the origin-destination pair section ; represents at time class Traffic flow of vehicle class between origin-destination along the path between road segments at time t, where represents the out-flow rate of road segment at time t, represents the traffic flow of road segment at time t, represents the percentage of vehicle class in the traffic flow, represents at time the in-flow rate of vehicle class between origin-destination along the path between road segments at time t, represents the travel demand between origin-destination at time t, represents the set of paths from which vehicles flow out of origin at time t, represents the set of charging paths for electric vehicles between origin-destination at time t, represents the set of charging paths for fuel cell vehicles between origin-destination at time t, represents the set of charging paths for fuel vehicles between origin-destination at time t, represents the set of paths into which vehicles flow into destination at time t, represents the traffic flow of vehicle class arriving at destination between origin-destination at time represents the traffic flow of vehicle class arriving at destination between origin-destination cumulatively at time represents the traffic flow of vehicle class arriving at destination between origin-destination cumulatively at time and represent the sets of paths flowing into and out of traffic node respectively, represents the travel time on road segment under the in-flow rate at time t, represents the free travel time on road segment at time t, represents the vehicle congestion capacity on road segment at time t, represents the set of ordinary road segments in the traffic network, represents the integrated energy station Average charging time of the vehicle Represents the integrated energy station Maximum allowable queuing time of electric vehicles Represents the integrated energy station Number of charging piles within Represents the set of charging sections in the transportation network Represents the integrated energy station Average hydrogenation time of the vehicle Represents the integrated energy station Maximum allowable queuing time of fuel cell vehicles Represents the integrated energy station Number of hydrogenation piles within Represents the set of hydrogenation sections in the transportation network Represents Traffic utility on the section at time Represents Unit time utility at time Represents Integrated energy station at time Unit electricity price of Represents Integrated energy station at time Unit hydrogen price of Represents Integrated energy station at time Average charging power of Represents Integrated energy station at time Average hydrogenation volume of Represents Integrated energy station at time Coupon amount set by Represents the set of virtual sections in the transportation network Represents Critical passing time of the section at time Represents the section Maximum passing vehicle capacity of Represents the section Maximum passing time of

[0036] Furthermore, in step 4, the operation model of the integrated energy station aiming at maximizing the profit is as follows:

[0037] (A-18)

[0038] Utility equation for electricity and hydrogen sales:

[0039] (A-19)​​

[0040] (A-20)

[0041] (A-21)

[0042] (A-22)

[0043] Hydrogen production constraint:

[0044] (A-23)

[0045] (A-24)

[0046] (A-25)

[0047] (A-26)

[0048] Hydrogen storage capacity balance constraint:

[0049] (A-27)

[0050] (A-28)

[0051] (A-29)

[0052] Electric energy balance constraint:

[0053] (A-30)

[0054] Coupon utility equation:

[0055] (A-31)

[0056] Wherein, represents the total revenue of the integrated energy station at time , represents the electricity sales revenue of the integrated energy station at time , represents the hydrogen sales revenue of the integrated energy station at time , represents the electricity purchase utility of the integrated energy station at time , represents the hydrogen production utility of the integrated energy station at time , represents Integrated Energy Station at a Certain Moment The utility of the issued coupon denotes Traffic volume of the road section at a certain moment of denotes Traffic volume of the road section at a certain moment of Denotes a binary variable. If the integrated energy station serves the road section it is 1, otherwise 0 Denotes a binary variable. If the integrated energy station serves the road section it is 1, otherwise 0 denotes The nodal marginal price of the power system node at a certain moment of denotes The total power demand of the power system node at a certain moment of Denotes the operation utility of power - to - hydrogen Denotes the operation utility of the compressor denotes Integrated Energy Station at a certain moment The power - to - hydrogen power denotes Integrated Energy Station at a certain moment The compressor power Denotes the operation efficiency of power - to - hydrogen Denotes the calorific value of hydrogen denotes Integrated Energy Station at a certain moment The hydrogen production volume and denotes Integrated Energy Station at a certain moment The minimum and maximum values of the power - to - hydrogen electric power Denotes the compression ratio of the compressor Denotes the isentropic exponent of hydrogen Denotes the operation parameters of the compressor denotes Integrated Energy Station at a certain moment The electric power of the compressor denotes Integrated Energy Station at a certain moment The maximum electric power of the compressor denotes Integrated Energy Station at a certain moment The hydrogen storage volume Denotes the time interval and denotes The minimum and maximum hydrogen storage capacities of the integrated energy station at a certain moment , and represent the hydrogen storage capacity of the integrated energy station at the initial moment 0 and the end moment , represents the maximum allowable hydrogen storage deviation represents a binary variable. If , belong to the same integrated energy station, it is 1; otherwise, it is 0 represents at the moment starting point - destination pair along the path section the inflow rate represents at the moment starting point - destination pair along the path section the inflow rate

[0057] Furthermore, in step 5, the reward function of the observation and evaluation network is as follows:

[0058] (A - 32)

[0059] The value function of the observation and evaluation network:

[0060] (A - 33)

[0061] The reward redistribution formula:

[0062] (A - 34)

[0063] Maximize the reward function of the actual evaluation network:

[0064] (A - 35)

[0065] In the formula, represents the reward function of the observation and evaluation network at the moment in state and action , represents the value function of the observation and evaluation network at the moment in state and action , represents the expected value under the policy , represents the discount factor at the moment Indicates the next state after performing an action and represents the next state the action selected next represents the next state at time and the action the value function of represents the reward function of the actual evaluation network

[0066] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0067] The present invention processes the delay characteristics of traffic propagation by introducing a reward redistribution method. Meanwhile, to capture the dynamic characteristics of traffic flow, a traffic model of dynamic user equilibrium is adopted. The method for capturing the dynamic characteristics of traffic flow in the present invention helps to improve the operation revenue of the integrated energy hydrogen station, improve the traffic node passing efficiency during peak hours, and achieve the dynamic matching of power - hydrogen energy supply and traffic demand in the time - space dimension compared with the existing methods. The pricing mechanism of the integrated energy station can guide users to preferentially select stations based on location advantages during low - load periods, and balance the regional traffic pressure and stabilize the comprehensive utilization rate of charging piles through dynamic premium methods during high - load periods Brief Description of the Drawings

[0068] Figure 1 is the flowchart of the method of the present invention

[0069] Figure 2 is the error situation of the road section flow under different methods Detailed Embodiments

[0070] The present invention will be further clarified below in conjunction with the drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art fall within the scope defined by the appended claims of this application

[0071] As Figure 1 shown, the present invention proposes an integrated energy station operation method based on reward redistribution of multi - agent reinforcement learning, and this method includes the following steps

[0072] Step 1: Obtain the network coefficients and operation coefficients of the power grid model. The network coefficients include the power grid topology, line resistance, and impedance, and the operation coefficients include the power generation coefficient of the generator set, the charge - discharge coefficient of the energy storage system, the photovoltaic inverter coefficient, and the charging station parameters

[0073] Step 2: Obtain the power grid load demand, photovoltaic output, vehicle travel demand, and average charging and hydrogen - adding historical demand scenario data

[0074] Step 3: Taking the road segment state equation, traffic flow propagation constraint, user proportion allocation constraint, node traffic conservation constraint, and utility equation as constraint conditions, and taking the minimum of user travel and charging utility in the transportation network as the objective function, a traffic model based on dynamic user equilibrium is established. According to this model, the charging vehicle flow of the node where the integrated energy station is located is obtained;

[0075] Step 4: Regarding the integrated energy station as an agent, taking the node charging vehicle flow obtained in Step 3 as the input of the agent state, and taking the utility equations of electricity and hydrogen sales, hydrogen storage volume balance constraint, hydrogen production constraint, power balance constraint, and coupon utility equation as constraint conditions, and taking the maximum comprehensive income of the integrated energy station as the objective function, an operation model of the integrated energy station is established;

[0076] Step 5: Taking the objective function in Step 4 as the reward function of the observation and evaluation network in the multi-agent reinforcement learning reward redistribution algorithm, the reward function of the actual evaluation network is calculated through the reward redistribution formula. The reward redistribution formula is composed of the reward function and value function of the observation and evaluation network. The maximum of the integrated energy station income is realized by maximizing the reward function of the actual evaluation network.

[0077] Further, in Step 3, the dynamic user equilibrium model is:

[0078] (A-1)

[0079] In the formula, represents the time, represents the road segment, represents the inflow rate of the road segment at time t, represents the road segment at time t

[0080] Further, in Step 3, the road segment state equation is:

[0081] (A-2)

[0082] (A-3)

[0083] (A-4)

[0084] The traffic flow propagation constraint is:

[0085] (A-5)

[0086] (A-6)

[0087] (A-7)

[0088] (A-8)

[0089] User proportion allocation constraint:

[0090] (A-9)

[0091] (A-10)

[0092] (A-11)

[0093] (A-12)

[0094] Node flow conservation constraint

[0095] (A-13)

[0096] Utility equation:

[0097] (A-14)

[0098] (A-15)

[0099] (A-16)

[0100] (A-17)

[0101] In the formula, represents the inflow rate of the th type of vehicle between the origin-destination pair along the path section ; represents the outflow rate of the th type of vehicle between the origin-destination pair along the path section ; represents the flow of the th type of vehicle between the origin-destination pair along the path section ; represents the outflow rate of section at time t; represents the flow of section at time t; represents the Percentage of vehicles of a certain type Indicating at the moment Origin-destination Interval along the path Section of the road Inflow rate on it Indicating the travel demand between the origin-destination Pairs at time t Indicating the set of paths flowing out of the origin Indicating the origin-destination Pairs of electric vehicles Set of charging paths for Indicating the origin-destination Pairs of fuel cell vehicles Set of charging paths for Indicating the origin-destination Pairs of fuel vehicles Set of charging paths for Indicating the set of paths flowing into the destination Indicating At the moment, the origin-destination Pairs of the Traffic flow of vehicles of a certain type arriving at the destination Indicating At the moment, the origin-destination Pairs of the Accumulated traffic flow of vehicles of a certain type arriving at the destination And Respectively indicate the sets of paths flowing into and out of the traffic node Indicating at the inflow rate On the road section Travel time on it Indicating on the road section Free travel time on it Indicating on the road section Vehicle congestion capacity of Indicating the set of ordinary road sections in the traffic network Indicating the integrated energy station Average charging time of vehicles Indicating the integrated energy station Maximum allowable queuing time for electric vehicles Indicating the integrated energy station Number of charging piles inside Indicating the set of charging road sections in the traffic network Indicating the integrated energy station Average hydrogen refueling time of vehicles​​​ Represents the integrated energy station The maximum allowable queuing time for fuel cell vehicles Represents the integrated energy station The number of hydrogen refueling piles inside Represents the set of hydrogen refueling sections in the transportation network Represents The traffic utility at time on the section Represents The unit time utility at time Represents The integrated energy station at time The unit electricity price Represents The integrated energy station at time The unit hydrogen price Represents The integrated energy station at time The average charging power Represents The integrated energy station at time The average hydrogen refueling volume Represents The integrated energy station at time The coupon amount set Represents the set of virtual sections in the transportation network Represents The critical passing time at time on the section Represents the section The maximum passing vehicle capacity Represents the section The maximum passing time

[0102] Furthermore, in step 4, the integrated energy station operation model with the goal of maximizing revenue is as follows:

[0103] (A-18)

[0104] The utility equations for electricity and hydrogen sales:

[0105] (A-19)

[0106] (A-20)

[0107] (A-21)

[0108] (A-22)

[0109] Hydrogen production constraint:​​

[0110] (A-23)

[0111] (A-24)

[0112] (A-25)

[0113] (A-26)

[0114] Hydrogen storage capacity balance constraint:

[0115] (A-27)

[0116] (A-28)

[0117] (A-29)

[0118] Electric energy balance constraint:

[0119] (A-30)

[0120] Coupon utility equation:

[0121] (A-31)

[0122] In the formula, represents the total revenue of the integrated energy station at time , represents the electricity sales revenue of the integrated energy station at time , represents the hydrogen sales revenue of the integrated energy station at time , represents the electricity purchase utility of the integrated energy station at time , represents the hydrogen production utility of the integrated energy station at time , represents the coupon utility issued by the integrated energy station at time , represents the traffic flow of road section at time represents the traffic flow of road section at time represents a binary variable. If the integrated energy station is a road section The service is 1, otherwise it is 0. represents a binary variable. If the integrated energy station is a road section The service is 1, otherwise it is 0. represents the nodal marginal price of the power system node at time represents the total power demand of the power system node at time represents the operating utility of power-to-hydrogen represents the operating utility of the compressor represents the power-to-hydrogen power of the integrated energy station at time represents the compressor power of the integrated energy station at time represents the operating efficiency of power-to-hydrogen represents the calorific value of hydrogen represents the hydrogen production of the integrated energy station at time and represents the minimum and maximum values of the power-to-hydrogen electric power of the integrated energy station at time represents the compression ratio of the compressor represents the isentropic exponent of hydrogen represents the operating parameters of the compressor represents the electric power of the compressor of the integrated energy station at time represents the maximum electric power of the compressor of the integrated energy station at time represents the hydrogen storage of the integrated energy station at time represents the time interval and represents the minimum and maximum hydrogen storage of the integrated energy station at time and represents the initial time 0 and the end the hydrogen storage of the integrated energy station at time represents the maximum allowable hydrogen storage deviation represents a binary variable. If​​​​​​​​​​​ , Belonging to the same integrated energy station, it is 1, otherwise it is 0. Indicates at time Starting point - destination The inflow rate along the path between Road section at Indicates at time Starting point - destination The inflow rate along the path between Road section at.

[0123] Furthermore, in step 5, the reward function of the observation and evaluation network is:

[0124] (A - 32)

[0125] The value function of the observation and evaluation network:

[0126] (A - 33)

[0127] The reward redistribution formula:

[0128] (A - 34)

[0129] Maximize the reward function of the actual evaluation network:

[0130] (A - 35)

[0131] In the formula, Indicates The reward function of the observation and evaluation network at time in state and action ; Indicates The value function of the observation and evaluation network at time in state and action ; Indicates the expected value under policy ; Indicates The discount factor at time Indicates the next state after executing action ; Indicates the next state The action selected under Indicates The value function of the next state at time, action ; Indicates the reward function of the actual evaluation network.

[0132] Case Study Analysis

[0133] The superiority of the integrated energy station pricing method based on multi-agent reinforcement learning reward redistribution described in the present invention will be illustrated by a case study below. The actual transportation network and distribution network in Nanjing are adopted in the present invention. To compare the superiority of the method proposed in the present invention, the integrated energy station pricing is respectively carried out by using the static traffic flow, semi-dynamic traffic flow and the dynamic user equilibrium method proposed in the present invention. The present invention is implemented through the Pytorch platform and the Gurobi solver is used to solve the non-linear programming problem.

[0134] Based on this case study, the error situations of the static traffic flow and semi-dynamic traffic flow models when calculating the traffic flow of road sections compared with the dynamic user equilibrium method of the present invention are shown in Figure 2 . Figure 2 Figure (a) in Figure 2 shows the error situation of the traffic flow of road sections changing with time and road sections when the static traffic flow model is compared with the dynamic user equilibrium method of the present invention.

[0135] Table 1 Comparison of Revenue Results of Different Methods

[0136] Integrated Energy Station Revenue Based on Static Traffic Flow Model (yuan) Revenue Based on Semi-Dynamic Traffic Flow Model (yuan) Revenue Based on Dynamic User Equilibrium Model (yuan) IEHS1 10352 11268 11891 IEHS2 9463 10172 10963 IEHS3 10177 10186 10183

[0137] As mentioned above, the above are only the specific implementation manners of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for operating an integrated energy station based on multi-agent reinforcement learning reward redistribution, characterized in that: The method comprises the following steps: Step 1: Obtain the network coefficient and operation coefficient of the power grid model, wherein the network coefficient includes the power grid topology, line resistance and impedance, and the operation coefficient includes the power generation coefficient of the generator set, the charging and discharging coefficient of the energy storage system, the photovoltaic inverter coefficient, and the charging station parameters; Step 2: Obtain grid load demand, photovoltaic output, vehicle travel demand, average charging and hydrogen refueling historical demand scenario data; Step 3: Taking the road state equation, flow propagation constraint, user ratio allocation constraint, node flow conservation constraint, and utility equation as constraint conditions, and taking the minimum travel and charging utility of users in the traffic network as the objective function, a traffic model based on dynamic user equilibrium is established, and the charging vehicle flow of the node to which the integrated energy station belongs is obtained according to the model; Step 4: The integrated energy station is used as an intelligent agent. The node charging vehicle flow obtained in step 3 is used as the input of the intelligent agent state. The utility equation of electricity and hydrogen sales, hydrogen storage balance constraint, hydrogen production constraint, power balance constraint, and coupon utility equation are used as constraints. The maximum comprehensive benefit of the integrated energy station is used as the objective function to establish an integrated energy station operation model. Step 5: Use the objective function in step 4 as the reward function of the observation evaluation network in the multi-agent reinforcement learning reward redistribution algorithm, and calculate the reward function of the actual evaluation network through the reward redistribution formula. The reward redistribution formula is composed of the reward function and value function of the observation evaluation network. The revenue of the integrated energy station is maximized by maximizing the reward function of the actual evaluation network. In step 3, the dynamic user equilibrium model is: (A-1) In the formula, Indicates the time, Indicates the road section, represents the road section at time t The inflow rate, represents the road section at time t The traffic utility; In step 4, the integrated energy station operation model with the goal of maximizing revenue is (A-18)； in, express Shike Integrated Energy Station Total revenue, express Shike Integrated Energy Station The revenue from electricity sales, express Shike Integrated Energy Station The revenue from hydrogen sales, express Shike Integrated Energy Station The utility of purchasing electricity, express Shike Integrated Energy Station The hydrogen production effect express Shike Integrated Energy Station The utility of the coupons issued; In step 5, the reward function of the observation evaluation network is: (A-32) Observe the value function of the evaluation network: (A-33) Reward redistribution formula: (A-34) Maximize the reward function of the actual evaluation network: (A-35) In the formula, express Always in the state and actions Next, observe the reward function of the evaluation network, express Always in the state and actions Next, observe the value function of the evaluation network, Indicated in strategy The expected value under express The discount factor at the moment, Indicates execution of an action The next state after Indicates the next state Next, select the action. express Next state at any time ,action The value function of represents the reward function that actually evaluates the network.

2. According to claim 1, a comprehensive energy station operation method based on multi-agent reinforcement learning reward redistribution is characterized in that: In step 3, the road segment state equation is: (A-2) (A-3) (A-4) The traffic propagation constraints are: (A-5) (A-6) (A-7) (A-8) User ratio allocation constraints: (A-9) (A-10) (A-11) (A-12) Node flow conservation constraints (A-13) Utility equation: (A-14) (A-15) (A-16) (A-17) In the formula, Indicates at time No. Type of vehicle in the origin-destination Along the path Road Section The inflow rate, Indicates at time No. Type of vehicle in the origin-destination Along the path Road Section The outflow rate on Indicates at time No. Type of vehicle in the origin-destination Along the path Road Section The flow rate on represents the road section at time t The outflow rate, represents the road section at time t of traffic, Indicates The percentage of vehicles in this category, Indicates at time Origin-Destination Along the path Road Section The inflow rate, Indicates origin-destination The travel demand at time t is Indicates the outflow starting point A collection of paths, Indicates origin-destination Electric Vehicles The charging path set, Indicates origin-destination Fuel cell vehicle The charging path set, Indicates origin-destination Fuel car The charging path set, Indicates the destination of inflow A collection of paths, express Time starting point-destination Pair The traffic flow of vehicles arriving at the destination, express Time starting point-destination Pair The total traffic volume of vehicles of this type arriving at the destination, and Represents the inflow and outflow traffic nodes respectively A collection of paths, Indicates the inflow rate Lower section The travel time on Indicates that on the road Free travel time on Indicates that on the road The vehicle congestion capacity, represents the set of common road sections in the traffic network, Indicates integrated energy station The average charging time of the vehicle, Indicates integrated energy station Maximum permissible queuing time for electric vehicles, Indicates integrated energy station The number of charging stations inside represents the set of charging sections in the traffic network, Indicates integrated energy station Average hydrogen refueling time for vehicles, Indicates integrated energy station Maximum permissible queue time for fuel cell vehicles, Indicates integrated energy station The number of internal hydrogenation piles, represents the set of hydrogen refueling sections in the transportation network, express Time section The utility of passage, express The utility per unit time, express Shike Integrated Energy Station The unit electricity price, express Shike Integrated Energy Station The unit hydrogen price, express Shike Integrated Energy Station The average charging power, express Shike Integrated Energy Station The average amount of hydrogenation, express Shike Integrated Energy Station The coupon amount you set, represents the set of virtual road segments in the traffic network, express Time section The critical travel time, Indicates road segment The maximum traffic capacity, Indicates road segment The maximum travel time.

3. The integrated energy station operation method based on multi-agent reinforcement learning reward redistribution according to claim 2 is characterized in that: In step 4, The utility equation for selling electricity and hydrogen is: (A-19) (A-20) (A-21) (A-22) Hydrogen production constraints: (A-23) (A-24) (A-25) (A-26) Hydrogen storage balance constraints: (A-27) (A-28) (A-29) Power balance constraints: (A-30) Coupon Utility Equation: (A-31) In the formula, express Time section of traffic, express Time section of traffic, Represents a binary variable. If the integrated energy station For road section 1 if service is enabled, 0 otherwise. Represents a binary variable. If the integrated energy station For road section 1 if service is enabled, 0 otherwise. express Power system node at time The node marginal electricity price is express Power system node at time The total power demand, Indicates the operational effectiveness of power-to-hydrogen conversion, Indicates the compressor operation efficiency. express Shike Integrated Energy Station The power of electricity to hydrogen, express Shike Integrated Energy Station The compressor power, Indicates the operating efficiency of power-to-hydrogen conversion, represents the calorific value of hydrogen, express Shike Integrated Energy Station The hydrogen production and express Shike Integrated Energy Station The minimum and maximum values ​​of the power conversion to hydrogen, Indicates the compression ratio of the compressor, represents the isentropic index of hydrogen, Indicates the operating parameters of the compressor. express Shike Integrated Energy Station The electrical power of the compressor, express Shike Integrated Energy Station The maximum electrical power of the compressor, express Shike Integrated Energy Station The hydrogen storage capacity, Indicates the time interval, and express Shike Integrated Energy Station The minimum and maximum hydrogen storage capacity, and Indicates the initial 0 time and the end Shike Integrated Energy Station The hydrogen storage capacity, Indicates the maximum allowable hydrogen storage deviation, represents a binary variable, if , Belong to the same integrated energy station, it is 1, otherwise it is 0, Indicates at time Origin-Destination Along the path Road Section The inflow rate, Indicates at time Origin-Destination Along the path Road Section The inflow rate on .

Images