Dynamic optimization method for port integrated energy system based on deep reinforcement learning

Abstract

The invention provides a port integrated energy system dynamic optimization method based on deep reinforcement learning, and the method comprises the steps: 1) constructing a port integrated energy system environment, and defining a system state space, an action space and a reward function; 2) constructing a deep reinforcement learning algorithm based on near-end strategy optimization, wherein the deep reinforcement learning algorithm comprises a strategy network, a value network, a trust region cutting function, a dominant function and a time difference error; 3) constructing a track set generated by interaction of a deep reinforcement learning algorithm and a port integrated energy system environment, wherein the track set is used for updating algorithm parameters; and 4) training the port integrated energy system environment by using the updated deep reinforcement learning algorithm to generate a dynamic scheduling strategy. The dynamic optimization operation of the port comprehensive energy system is realized through the deep reinforcement learning technology, the time sequence decision characteristics of the system are fully considered, the flexibility of the system and the utilization rate of renewable energy sources are improved, and the efficient and flexible operation of the port comprehensive energy system is realized.

Description

Technical Field

[0001] This application relates to the technical field of integrated energy operation, and particularly to a dynamic optimization method for a port integrated energy system based on deep reinforcement learning. Background Technique

[0002] As a trade and logistics hub, a port has a large energy consumption and a complex energy structure. Traditional ports rely on fossil energy, resulting in low efficiency and a large amount of emissions. A port integrated energy system consists of electricity, hydrogen energy, energy storage, and various energy-consuming devices. Such a multi-energy complementary system faces uncertain factors such as the volatility of renewable energy, the randomness of load demand, and changes in energy market prices, posing great challenges to the stable operation and economic dispatching of the system. Traditional optimization methods, such as linear programming and mixed-integer linear programming, can obtain accurate solutions in a deterministic environment but are difficult to handle complex systems with high dimensions, strong coupling, and strong randomness.

[0003] As a cutting-edge technology in artificial intelligence, deep reinforcement learning can effectively handle sequential decision-making problems, adapt to uncertain environments, and achieve self-learning and continuous optimization through environment perception, policy learning, and reward mechanisms, providing a new idea for the dynamic optimization operation of a port integrated energy system. Developing a dynamic optimization operation method for a port integrated energy system based on deep reinforcement learning can effectively improve the energy utilization efficiency of ports, reduce carbon emissions, and promote the green and intelligent development of ports. Summary of the Invention

[0004] Object of the Invention. Based on this, it is necessary to provide a dynamic optimization method for a port integrated energy system based on deep reinforcement learning in view of the above problems.

[0005] Technical Solution. To solve the above technical problems and achieve the above object of the invention, the present invention proposes a dynamic optimization method for a port integrated energy system based on deep reinforcement learning, and the method includes:

[0006] Step 1, constructing a port integrated energy system environment, including defining a system state space, an action space, and a reward function;

[0007] Step 2, constructing a deep reinforcement learning algorithm based on proximal policy optimization, including a policy network, a value network, a trust region clipping function, an advantage function, and a temporal difference error;

[0008] Step 3, constructing a trajectory set generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment for updating algorithm parameters;

[0009] Step 4, using the updated deep reinforcement learning algorithm to train the port integrated energy system environment to generate a dynamic scheduling strategy.

[0010] Further, the specific process of step (1) is as follows:

[0011] (101) Construct the environmental state space of the port integrated energy system required for the deep reinforcement learning algorithm:

[0012]

[0013] In the formula, i is the ship; j is the gas turbine unit; t is the operating time; s t represents the state observed by the deep reinforcement learning agent at the t-th moment; S i,t represents the in-port state of the i-th ship at the t-th moment, in-port is 1, otherwise it is 0; represents the ship load of the i-th ship at the t-th moment; A i represents the delay cost coefficient of the ship arriving at the port late; S j,t represents the operating state of the j-th gas turbine at the t-th moment, operating is 1, otherwise it is 0; represents the start-up cost coefficient of the j-th gas turbine; represents the shutdown cost coefficient of the j-th gas turbine; represents the operating cost coefficient of the j-th gas turbine; represents the power generation cost coefficient of the j-th gas turbine; represents the state of charge of the port electric energy storage at the t-th moment; represents the state of hydrogen of the port hydrogen energy storage at the t-th moment; represents the electric load of the port at the t-th moment; represents the power generation power of the new energy at the port at the t-th moment; C t represents the power purchase price of the superior power grid at the t-th moment;

[0014] (102) Construct the environmental action space of the port integrated energy system required for the deep reinforcement learning algorithm:

[0015]

[0016] In the formula, a t represents the action output by the deep reinforcement learning algorithm at the t-th moment; O i,t represents the in-port permit of the i-th ship at the t-th moment, in-port is 1, otherwise it is 0; P i,t represents the electric power consumed by the i-th ship at the t-th moment; O j,t represents the start-stop command of the j-th gas turbine at the t-th moment, start is 1, shutdown is 0; P j,t represents the power generation power of the j-th gas turbine at the t-th moment; represents the actual output power of the port electric energy storage at the t-th moment; represents the actual output power of the port hydrogen energy storage at the t-th moment;

[0017] (103) Construct the environmental reward function of the port integrated energy system required for the deep reinforcement learning algorithm:

[0018]

[0019] In the formula, N J is the total number of gas turbines; N I is the total number of ships; r t represents the reward obtained by the deep reinforcement learning algorithm at the t-th moment; y j,t represents the start state variable of the j-th gas turbine at the t-th moment, with start being 1 and otherwise 0; z j,t represents the shutdown state variable of the j-th gas turbine at the t-th moment, with shutdown being 1 and otherwise 0; x j,t represents the operating state variable of the j-th gas turbine at the t-th moment, with operation being 1 and otherwise 0; y i,t The in-port state variable of the i-th ship at the t-th moment, with in-port being 1 and otherwise 0; represents the expected arrival time of the i-th ship at the port; represents the electric power purchased from the superior power grid at the t-th moment; represents the delay time of the i-th ship arriving at the port at the t-th moment, and the negative delay time is set to 0.

[0020] Furthermore, the specific process of the step (2) is as follows:

[0021] (201) Construct the policy network of the deep reinforcement learning algorithm based on proximal policy optimization:

[0022]

[0023] θ←θ+α▽ θ L clip (θ)(6)

[0024] In the formula, θ represents the parameter of the policy network of the deep reinforcement learning algorithm; φ represents the parameter of the value network of the deep reinforcement learning algorithm; T represents the total number of steps when the task is completed; k represents the time step accumulation variable; γ represents the discount factor; λ represents the generalized advantage estimation smoothing coefficient; clip() represents the clipping function; ε represents the clipping range; α represents the learning rate; ▽ θ represents the gradient of θ; Policy(θ) represents the policy taken by the policy network θ; L clip (θ) represents the optimization objective function of the policy network θ; represents the expectation at the t-th moment; represents the policy network θ selecting the action in the state with probability;

[0025] Denote the old policy network as θ old At state The probability of selecting action ;

[0026] Denote the state value predicted by the value network φ at state ; Denote the state value predicted by the value network φ at state ; y j,t+k Denote the start state variable of the j - th gas turbine at the (t + k)-th moment, start is 1, otherwise 0; z j,t+k Denote the shutdown state variable of the j - th gas turbine at the (t + k)-th moment, shutdown is 1, otherwise 0; x j,t+k Denote the operating state variable of the j - th gas turbine at the (t + k)-th moment, operation is 1, otherwise 0; P j,t+k Denote the power generation of the j - th gas turbine at the (t + k)-th moment; y i,t+k The in - port state variable of the i - th ship at the (t + k)-th moment, in - port is 1, otherwise 0; C t+k Denote the power purchase price of the superior power grid at the (t + k)-th moment; Denote the electric power purchased from the superior power grid at the (t + k)-th moment; S i,t+1+k Denote the in - port state of the i - th ship at the (t + 1 + k)-th moment, in - port is 1, otherwise 0; Denote the ship load of the i - th ship at the (t + 1 + k)-th moment; S j,t+1+k Denote the operating state of the j - th gas turbine at the (t + 1 + k)-th moment, operation is 1, otherwise 0; Denote the state of charge of the port electric energy storage at the (t + 1 + k)-th moment; Denote the state of hydrogen charge of the port hydrogen energy storage at the (t + 1 + k)-th moment; Denote the electric load of the port at the (t + 1 + k)-th moment; Denote the power generation of the new energy at the port at the (t + 1 + k)-th moment; C t+1+k Denote the power purchase price of the superior power grid at the (t + 1 + k)-th moment; S i,t+k Denote the in - port state of the i - th ship at the (t + k)-th moment, in - port is 1, otherwise 0; Denote the ship load of the i - th ship at the (t + k)-th moment; S j,t+k Denote the operating state of the j - th gas turbine at the (t + k)-th moment, operation is 1, otherwise 0; Denote the state of charge of the port electric energy storage at the (t + k)-th moment; Denote the state of hydrogen charge of the port hydrogen energy storage at the (t + k)-th moment; Denote the electric load of the port at the (t + k)-th moment; Denote the power generation of the new energy at the port at the (t + k)-th moment;

[0027] (202) Construct the value network of the deep reinforcement learning algorithm based on Proximal Policy Optimization:

[0028]

[0029] φ ← φ - α▽ φ L value (φ)(9)

[0030] Where Value(φ) represents the state value predicted by the value network φ; L value (φ) represents the loss function of the value network φ; ▽ φ represents taking the gradient with respect to φ; represents the state value predicted by the value network φ at state ; represents the state value predicted by the value network φ at state ; S i,t+1 represents the in-port state of the i-th ship at the (t + 1)-th moment, 1 for in-port, otherwise 0; represents the ship load of the i-th ship at the (t + 1)-th moment; S j,t+1 represents the operating state of the j-th gas turbine at the (t + 1)-th moment, 1 for operating, otherwise 0; represents the state of charge of the port electrical energy storage at the (t + 1)-th moment; represents the state of hydrogen of the port hydrogen energy storage at the (t + 1)-th moment; represents the electrical load of the port at the (t + 1)-th moment; represents the power generation of new energy at the port at the (t + 1)-th moment; C t+1 represents the electricity purchase price from the superior power grid at the (t + 1)-th moment;

[0031] (203) Construct the trust region clipping function of the deep reinforcement learning algorithm based on Proximal Policy Optimization:

[0032]

[0033] Where r t (θ) represents the ratio of the action probability of the current policy θ to the action probability of the old policy of the deep reinforcement learning algorithm at the t-th moment;

[0034] (204) Construct the advantage function of the deep reinforcement learning algorithm based on Proximal Policy Optimization:

[0035]

[0036] Where: A t represents the advantage function of the deep reinforcement learning algorithm at the t-th moment;

[0037] (205)Construct the temporal difference error of the deep reinforcement learning algorithm based on proximal policy optimization:

[0038]

[0039] where δ t represents the temporal difference error of the deep reinforcement learning algorithm at the t-th moment.

[0040] Furthermore, in step (3), construct the trajectory set generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment for updating the algorithm parameters:

[0041]

[0042] where D t represents the trajectory set cumulatively generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment at the t-th moment; done t indicates whether the training at the t-th moment is over, 1 for over, otherwise 0.

[0043] Furthermore, in step (4), use the updated deep reinforcement learning algorithm to train the port integrated energy system environment to generate a dynamic scheduling strategy. The specific method is as follows:

[0044] (401) Initialize the time step t = 0, initialize the port integrated energy system environment parameters, and clear the trajectory set D t ;

[0045] (402) Input the environmental state s t into the algorithm policy network Policy(θ), generate the corresponding action probability and select the action a t ;

[0046] (403) Execute the action a t in the environment to obtain the feedback reward r t , the next state s t+1 and whether the current round of training is over done t ;

[0047] (404) Cumulatively update the trajectory set D t , add the state s t , the action a t , the reward r t , the next state s t+1 and the end state done t ;

[0048] (405) Judge whether the current round of training is over done t , if done t = 1, indicating the end, enter step (406); if donet = 0 indicates not ended, increment time t by 1, and return to step (402);

[0049] (406) Use the trajectory set D t and the value network Value(φ) to calculate the advantage function A t and the temporal difference error δ t ;

[0050] (407) Use the advantage function A t to update the policy network parameter θ, and use the temporal difference error δ t to update the value network parameter φ;

[0051] (408) Loop steps (401)-(407) until the model converges, and finally form a policy for effectively scheduling resources in a dynamic environment.

[0052] Beneficial effects. Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0053] The present invention considers a dynamic optimization method for a port integrated energy system based on deep reinforcement learning, which focuses on combining deep reinforcement learning technology with the operation optimization of the port integrated energy system. It can effectively handle the complex characteristics of high-dimensionality, non-linearity, and strong coupling in the port integrated energy system and adapt to the dynamic change process of the system; through interactive learning with the port environment, it continuously optimizes the decision-making strategy, has the ability of self-evolution and continuous learning, and can adapt to new operating environments and energy configurations; through end-to-end decision mapping, it does not require an accurate system mathematical model, reduces the modeling difficulty, and improves the generality of the method; at the same time, it considers multiple objectives such as energy utilization efficiency and carbon emission reduction to achieve the global optimization of the port integrated energy system. Description of the Drawings

[0054] Figure 1 is the flowchart of the method of the present invention;

[0055] Figure 2 is the framework diagram of the deep reinforcement learning algorithm of the present invention;

[0056] Figure 3 is the interaction diagram of the deep reinforcement learning and the port integrated energy system of the present invention. Detailed Embodiments

[0057] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In one embodiment, as Figure 1As shown in the figure, the present invention proposes a dynamic optimization method for a port integrated energy system based on deep reinforcement learning, and the method includes:

[0059] Step 1, construct the port integrated energy system environment, including defining the system state space, action space, and reward function;

[0060] Step 2, construct a deep reinforcement learning algorithm based on proximal policy optimization, including a policy network, a value network, a trust region clipping function, an advantage function, and a temporal difference error;

[0061] Step 3, construct a trajectory set generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment for updating the algorithm parameters;

[0062] Step 4, use the updated deep reinforcement learning algorithm to train the port integrated energy system environment to generate a dynamic scheduling strategy.

[0063] Further, the specific process of the step (1) is as follows:

[0064] (101) Construct the port integrated energy system environment state space required by the deep reinforcement learning algorithm:

[0065]

[0066] In the formula, i is the ship; j is the gas turbine unit; t is the operation time; s t represents the state observed by the deep reinforcement learning agent at the t-th moment; S i,t represents the in-port state of the i-th ship at the t-th moment, being in port is 1, otherwise it is 0; represents the ship load of the i-th ship at the t-th moment; A i represents the delay cost coefficient of the ship arriving at the port with delay; S j,t represents the operation state of the j-th gas turbine at the t-th moment, running is 1, otherwise it is 0; represents the start-up cost coefficient of the j-th gas turbine; represents the shutdown cost coefficient of the j-th gas turbine; represents the operation cost coefficient of the j-th gas turbine; represents the power generation cost coefficient of the j-th gas turbine; represents the state of charge of the port electrical energy storage at the t-th moment; represents the state of hydrogen charge of the port hydrogen energy storage at the t-th moment; represents the electrical load of the port at the t-th moment; represents the power generation power of the port new energy at the t-th moment; C t represents the power purchase price of the superior power grid at the t-th moment;

[0067] (102) Construct the environmental action space of the port integrated energy system required for the deep reinforcement learning algorithm:

[0068]

[0069] Wherein, a t represents the action output by the deep reinforcement learning algorithm at the t-th moment; O i,t represents the port entry permit of the i-th ship at the t-th moment, 1 for port entry, otherwise 0; P i,t represents the electric power consumed by the i-th ship at the t-th moment; O j,t represents the start-stop command of the j-th gas turbine at the t-th moment, 1 for start, 0 for shutdown; P j,t represents the power generation power of the j-th gas turbine at the t-th moment; P t e represents the actual output power of the port electric energy storage at the t-th moment; represents the actual output power of the port hydrogen energy storage at the t-th moment;

[0070] (103) Construct the environmental reward function of the port integrated energy system required for the deep reinforcement learning algorithm:

[0071]

[0072] Wherein, N J is the total number of gas turbines; N I is the total number of ships; r t represents the reward obtained by the deep reinforcement learning algorithm at the t-th moment; y j,t represents the start state variable of the j-th gas turbine at the t-th moment, 1 for start, otherwise 0; z j,t represents the shutdown state variable of the j-th gas turbine at the t-th moment, 1 for shutdown, otherwise 0; x j,t represents the operating state variable of the j-th gas turbine at the t-th moment, 1 for operation, otherwise 0; y i,t The port entry state variable of the i-th ship at the t-th moment, 1 for port entry, otherwise 0; represents the expected arrival time of the i-th ship at the port; represents the electric power purchased from the superior power grid at the t-th moment; represents the delay time of the i-th ship arriving at the port at the t-th moment, and the negative delay time is set to 0.

[0073] Furthermore, the specific process of the step (2) is as follows:

[0074] (201) Construct the policy network of the deep reinforcement learning algorithm based on proximal policy optimization:

[0075]

[0076]

[0077] θ ← θ + α▽ θ L clip (θ)(6)

[0078] Where θ represents the parameters of the policy network of the deep reinforcement learning algorithm; φ represents the parameters of the value network of the deep reinforcement learning algorithm; T represents the total number of steps for which the task has been completed; k represents the time step accumulation variable; γ represents the discount factor; λ represents the generalized advantage estimation smoothing coefficient; clip() represents the clipping function; ε represents the clipping range; α represents the learning rate; ▽ θ represents the gradient with respect to θ; Policy(θ) represents the policy taken by the policy network θ; L clip (θ) represents the optimization objective function of the policy network θ; represents the expectation at the t-th moment; represents that the policy network θ is in the state and selects the action with probability;

[0079] represents the old policy network θ old in the state and selects the action with probability;

[0080] represents the state value predicted by the value network φ in the state ; represents the state value predicted by the value network φ in the state ; y j,t+k represents the start state variable of the j-th gas turbine at the t + k-th moment, with start being 1 and otherwise 0; z j,t+k represents the shutdown state variable of the j-th gas turbine at the t + k-th moment, with shutdown being 1 and otherwise 0; x j,t+k represents the operating state variable of the j-th gas turbine at the t + k-th moment, with operation being 1 and otherwise 0; P j,t+k represents the power generation of the j-th gas turbine at the t + k-th moment; y i,t+k The in-port state variable of the i-th ship at the t + k-th moment, with in-port being 1 and otherwise 0; C t+k represents the electricity purchase price of the superior power grid at the t + k-th moment; represents the electric power purchased from the superior power grid at the t + k-th moment; S i,t+1+k represents the in-port state of the i-th ship at the t + 1 + k-th moment, with in-port being 1 and otherwise 0; represents the ship load of the i-th ship at the t + 1 + k-th moment; S j,t+1+kIndicates the operating state of the j-th gas turbine at the (t + 1 + k)-th moment, with operation being 1 and otherwise 0; Indicates the state of charge of the port's electrical energy storage at the (t + 1 + k)-th moment; Indicates the state of hydrogen charge of the port's hydrogen energy storage at the (t + 1 + k)-th moment; Indicates the electrical load of the port at the (t + 1 + k)-th moment; Indicates the power generation of new energy at the port at the (t + 1 + k)-th moment; C t+1+k Indicates the purchase price of electricity from the superior power grid at the (t + 1 + k)-th moment; S i,t+k Indicates the in-port state of the i-th ship at the (t + k)-th moment, with being in port being 1 and otherwise 0; Indicates the ship load of the i-th ship at the (t + k)-th moment; S j,t+k Indicates the operating state of the j-th gas turbine at the (t + k)-th moment, with operation being 1 and otherwise 0; Indicates the state of charge of the port's electrical energy storage at the (t + k)-th moment; Indicates the state of hydrogen charge of the port's hydrogen energy storage at the (t + k)-th moment; Indicates the electrical load of the port at the (t + k)-th moment; Indicates the power generation of new energy at the port at the (t + k)-th moment;

[0081] (202) Construct the value network of the deep reinforcement learning algorithm based on proximal policy optimization:

[0082]

[0083] φ ← φ - α▽ φ L value (φ)(9)

[0084] In the formula, Value(φ) represents the state value predicted by the value network φ; L value (φ) represents the loss function of the value network φ; ▽ φ represents taking the gradient with respect to φ; represents the state value predicted by the value network φ in the state ; represents the state value predicted by the value network φ in the state ; S i,t+1 Indicates the in-port state of the i-th ship at the (t + 1)-th moment, with being in port being 1 and otherwise 0; Indicates the ship load of the i-th ship at the (t + 1)-th moment; S j,t+1 Indicates the operating state of the j-th gas turbine at the (t + 1)-th moment, with operation being 1 and otherwise 0; Indicates the state of charge of the port's electrical energy storage at the (t + 1)-th moment; Indicates the state of hydrogen charge of the port's hydrogen energy storage at the (t + 1)-th moment; denotes the electrical load of the port at the (t + 1)-th moment; denotes the power generation of new energy at the port at the (t + 1)-th moment; C t+1 denotes the power purchase price of the superior power grid at the (t + 1)-th moment;

[0085] (203) Construct a trust region clipping function for the proximal policy optimization-based deep reinforcement learning algorithm:

[0086]

[0087] where r t (θ) represents the ratio of the action probability of the current policy θ to the action probability of the old policy of the deep reinforcement learning algorithm at the t-th moment;

[0088] (204) Construct an advantage function for the proximal policy optimization-based deep reinforcement learning algorithm:

[0089]

[0090] where: A t represents the advantage function of the deep reinforcement learning algorithm at the t-th moment;

[0091] (205) Construct a temporal difference error for the proximal policy optimization-based deep reinforcement learning algorithm:

[0092]

[0093] where δ t represents the temporal difference error of the deep reinforcement learning algorithm at the t-th moment.

[0094] Furthermore, in step (3), construct a trajectory set generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment for updating the algorithm parameters:

[0095]

[0096] where D t represents the trajectory set cumulatively generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment at the t-th moment; done t represents whether the training ends at the t-th moment, 1 for end, otherwise 0.

[0097] Furthermore, in step (4), use the updated deep reinforcement learning algorithm to train the port integrated energy system environment to generate a dynamic scheduling strategy, and the specific method is as follows:

[0098] (401) Initialize the time step t = 0, initialize the port integrated energy system environment parameters, and clear the trajectory set D t ;

[0099] (402) Input the environmental state s t into the algorithmic policy network Policy(θ), generate the corresponding action probability, and select the action a t ;

[0100] (403) Execute the action a in the environment t , and obtain the feedback reward r t , the next state s t+1 and whether the current round of training is over done t ;

[0101] (404) Accumulate the trajectory set D t , add the state s t , the action a t , the reward r t , the next state s t+1 and the end state done t ;

[0102] (405) Judge whether the current round of training is over done t , if done t = 1, it means the end, and enter step (406); if done t = 0, it means not over, add 1 to the time t, and return to step (402);

[0103] (406) Use the trajectory set D t and the value network Value(φ) to calculate the advantage function A t and the temporal difference error δ t ;

[0104] (407) Use the advantage function A t to update the policy network parameter θ, and use the temporal difference error δ t to update the value network parameter φ;

[0105] (408) Loop steps (401)-(407) until the model converges, and finally form a policy for effectively scheduling resources in a dynamic environment.

[0106] Take a typical port integrated energy system as an example. The port integrated energy system includes wind power generation, photovoltaic power generation, gas turbines, electric energy storage systems, electrolyzers, hydrogen energy storage devices, and various port energy-consuming equipment. The system conducts optimization scheduling on a daily basis with a time resolution of 1 hour. Model training and testing are implemented in a Python environment. A neural network is constructed using the PyTorch deep learning framework, and a deep reinforcement learning method is used for solving.

[0107] To verify the effectiveness of the dynamic optimization method for port integrated energy systems based on deep reinforcement learning proposed in the present invention, the following three comparison schemes are designed:

[0108] Scheme 1: A deterministic optimization method based on mixed integer linear programming (MILP);

[0109] Scheme 2: A rolling optimization method based on model predictive control (MPC);

[0110] Scheme 3: The dynamic optimization method based on deep reinforcement learning proposed in the present invention.

[0111] The comparison results of the three schemes are shown in Table 1. In Scheme 1, although the global optimal solution can be theoretically obtained, due to the strong uncertainty in the actual system, there is a large deviation between the actual operating cost and the theoretical cost. In Scheme 2, the influence of uncertainty is partially overcome through rolling optimization, but the computational complexity is high and the real-time performance is poor. In Scheme 3, that is, the method based on deep reinforcement learning proposed in the present invention, through continuous interaction and learning with the environment, adaptively adjusts the decision-making strategy, and can still obtain a better operating cost considering uncertainty, while having high computational efficiency and being suitable for real-time decision-making. Compared with Scheme 1 and Scheme 2, Scheme 3 has advantages in terms of total cost, carbon emissions, and renewable energy utilization rate, reflecting the effectiveness of the method of the present invention.

[0112] Table 1 Comparison of different algorithms for solving port problems

[0113]

[0114] The above-described embodiments merely represent several implementation manners of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A dynamic optimization method for a port integrated energy system based on deep reinforcement learning, characterized in that The method includes the following steps: Step 1: Construct the environment of the port integrated energy system, including defining the system state space, action space, and reward function; Step 2: Construct a deep reinforcement learning algorithm based on proximal policy optimization, including a policy network, a value network, a trust region clipping function, an advantage function, and a temporal difference error; Step 3: Construct a trajectory set generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment for updating the algorithm parameters; Step 4: Use the updated deep reinforcement learning algorithm to train the port integrated energy system environment to generate a dynamic scheduling policy.

2. The dynamic optimization method of a port integrated energy system based on deep reinforcement learning according to claim 1, wherein, The specific process of step (1) is as follows: (101) Construct the state space of the port integrated energy system environment required by the deep reinforcement learning algorithm; Where, i represents a ship; j represents a gas turbine unit; t represents the operating time; s t represents the state observed by the deep reinforcement learning agent at the t-th moment; S i,t represents the in-port state of the i-th ship at the t-th moment, where in-port is 1, otherwise 0; represents the ship load of the i-th ship at the t-th moment; A i represents the delay cost coefficient for the ship to arrive at the port late; S j,t represents the operating state of the j-th gas turbine at the t-th moment, where operating is 1, otherwise 0; represents the start-up cost coefficient of the j-th gas turbine; represents the shutdown cost coefficient of the j-th gas turbine; represents the operating cost coefficient of the j-th gas turbine; represents the power generation cost coefficient of the j-th gas turbine; represents the state of charge of the port's electrical energy storage at the t-th moment; represents the state of hydrogen of the port's hydrogen energy storage at the t-th moment; P t l represents the electrical load of the port at the t-th moment; P t r represents the power generation power of new energy at the port at the t-th moment; C t represents the purchase price of electricity from the superior power grid at the t-th moment; (102) Construct the action space of the port integrated energy system environment required by the deep reinforcement learning algorithm; where a t represents the action output by the deep reinforcement learning algorithm at the t-th moment; O i,t represents the inbound permission of the i-th ship at the t-th moment, 1 for inbound, otherwise 0; P i,t represents the electric power consumed by the i-th ship at the t-th moment; O j,t represents the start / stop command of the j-th gas turbine at the t-th moment, 1 for start, 0 for shutdown; P j,t represents the power generation of the j-th gas turbine at the t-th moment; P t e represents the actual output power of the port electric energy storage at the t-th moment; represents the actual output power of the port hydrogen energy storage at the t-th moment; (103) Construct the reward function of the port integrated energy system environment required by the deep reinforcement learning algorithm; Where, N J is the total number of gas turbines; N I is the total number of ships; r t represents the reward obtained by the deep reinforcement learning algorithm at the t-th moment; y j,t represents the start state variable of the j-th gas turbine at the t-th moment, 1 for start, otherwise 0; z j,t represents the shutdown state variable of the j-th gas turbine at the t-th moment, 1 for shutdown, otherwise 0; x j,t represents the operating state variable of the j-th gas turbine at the t-th moment, 1 for operation, otherwise 0; y i,t The in-port state variable of the i-th ship at the t-th moment, 1 for in-port, otherwise 0; T i a represents the estimated arrival time of the i-th ship at the port; P t g represents the electric power purchased from the superior power grid at the t-th moment; (ty i,t -T i a ) + represents the delay time of the i-th ship arriving at the port at the t-th moment, and the negative delay time is set to 0.

3. The dynamic optimization method for a port integrated energy system based on deep reinforcement learning according to claim 2, wherein The specific process of step (2) is as follows: (201) Construct the policy network of the deep reinforcement learning algorithm based on proximal policy optimization; Where, θ represents the parameters of the policy network of the deep reinforcement learning algorithm; φ represents the parameters of the value network of the deep reinforcement learning algorithm; T represents the total number of steps for which the task has been completed; k represents the time step accumulation variable; γ represents the discount factor; λ represents the generalized advantage estimation smoothing coefficient; clip() represents the clipping function; ε represents the clipping range; α represents the learning rate; represents taking the gradient with respect to θ; Policy(θ) represents the policy adopted by the policy network θ; L clip (θ) represents the optimization objective function of the policy network θ; represents the expectation at the t-th moment; represents that the policy network θ selects an action in the state and is the probability of; Denote the old policy network as θ old At state The probability of selecting action ; Denotes the state value predicted by the value network φ at state ; Denotes the state value predicted by the value network φ at state ; y j,t+k Denotes the start state variable of the j-th gas turbine at the (t + k)-th moment, with start being 1 and otherwise 0; z j,t+k Denotes the shutdown state variable of the j-th gas turbine at the (t + k)-th moment, with shutdown being 1 and otherwise 0; x j,t+k Denotes the operating state variable of the j-th gas turbine at the (t + k)-th moment, with operation being 1 and otherwise 0; P j,t+k Denotes the power generation of the j-th gas turbine at the (t + k)-th moment; y i,t+k The in-port state variable of the i-th ship at the (t + k)-th moment, with in-port being 1 and otherwise 0; C t+k Denotes the power purchase price of the superior power grid at the (t + k)-th moment; Denotes the electric power purchased from the superior power grid at the (t + k)-th moment; S i,t+1+k Denotes the in-port state of the i-th ship at the (t + 1 + k)-th moment, with in-port being 1 and otherwise 0; Denotes the ship load of the i-th ship at the (t + 1 + k)-th moment; S j,t+1+k Denotes the operating state of the j-th gas turbine at the (t + 1 + k)-th moment, with operation being 1 and otherwise 0; Denotes the state of charge of the port electric energy storage at the (t + 1 + k)-th moment; Denotes the state of hydrogen charge of the port hydrogen energy storage at the (t + 1 + k)-th moment; Denotes the electric load of the port at the (t + 1 + k)-th moment; Denotes the power generation of the new energy at the port at the (t + 1 + k)-th moment; C t+1+k Denotes the power purchase price of the superior power grid at the (t + 1 + k)-th moment; S i,t+k Denotes the in-port state of the i-th ship at the (t + k)-th moment, with in-port being 1 and otherwise 0; Denotes the ship load of the i-th ship at the (t + k)-th moment; S j,t+k Denotes the operating state of the j-th gas turbine at the (t + k)-th moment, with operation being 1 and otherwise 0; Denotes the state of charge of the port electric energy storage at the (t + k)-th moment; Denotes the state of hydrogen charge of the port hydrogen energy storage at the (t + k)-th moment; Denotes the electric load of the port at the (t + k)-th moment; Denotes the power generation of the new energy at the port at the (t + k)-th moment; (202) Construct the value network of the deep reinforcement learning algorithm based on proximal policy optimization; Wherein, Value(φ) represents the state value predicted by the value network φ; L value (φ) represents the loss function of the value network φ; denotes taking the gradient with respect to φ; represents the state value predicted by the value network φ at state ; represents the state value predicted by the value network φ at state ; S i,t+1 represents the in-port state of the i-th ship at the (t + 1)-th moment, 1 for in port and 0 otherwise; represents the ship load of the i-th ship at the (t + 1)-th moment; S j,t+1 represents the operating state of the j-th gas turbine at the (t + 1)-th moment, 1 for operating and 0 otherwise; represents the state of charge of the port electrical energy storage at the (t + 1)-th moment; represents the state of hydrogen charge of the port hydrogen energy storage at the (t + 1)-th moment; represents the electrical load of the port at the (t + 1)-th moment; represents the power generation of new energy at the port at the (t + 1)-th moment; C t+1 represents the electricity purchase price from the superior power grid at the (t + 1)-th moment; (203) Construct the trust region clipping function of the deep reinforcement learning algorithm based on proximal policy optimization; where r t (θ) represents the ratio of the current policy θ action probability to the old policy action probability of the deep reinforcement learning algorithm at the t-th moment; (204) Construct the advantage function of the deep reinforcement learning algorithm based on proximal policy optimization; where: A t represents the advantage function of the deep reinforcement learning algorithm at the t-th moment; (205) Construct the temporal difference error of the deep reinforcement learning algorithm based on proximal policy optimization; where δ t represents the temporal difference error of the deep reinforcement learning algorithm at the t-th moment.

4. The dynamic optimization method of a port integrated energy system based on deep reinforcement learning according to claim 3, characterized in that In step (3), construct a trajectory set generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment for updating the algorithm parameters: where D t represents the trajectory set cumulatively generated by the interaction between the deep reinforcement learning algorithm and the port integrated energy system environment at the t-th moment; done t indicates whether the training ends at the t-th moment, with 1 for end and 0 otherwise.

5. The dynamic optimization method for a port integrated energy system based on deep reinforcement learning according to claim 4, wherein In step (4), use the updated deep reinforcement learning algorithm to train the port integrated energy system environment to generate a dynamic scheduling policy. The specific method is as follows: (401) Initialize the time step \(t = 0\), initialize the environmental parameters of the port integrated energy system, and clear the trajectory set \(D\) t ; (402) Input the environmental state s t into the algorithmic policy network Policy(θ), generate the corresponding action probability and select the action a t ; (403) Execute action a in the environment t , and obtain the feedback reward r t , the next state s t+1 and whether the current round of training has ended, done t ; (404) Cumulative trajectory set D t , add state s t , action a t , reward r t , next state s t+1 and end state done t ; (405) Determine whether the current round of training is over, done t , if done t = 1, indicating the end, proceed to step (406); if done t = 0, indicating not over, increment time t by 1, and return to step (402); (406) Utilize the trajectory set D t and the value network Value(φ) to calculate the advantage function A t and the temporal difference error δ t ; (407) Use the advantage function A t Update the policy network parameters θ, using the temporal difference error δ t Update the value network parameters φ; (408) Loop through steps (401)-(407) until the model converges, and finally form a policy for effectively scheduling resources in a dynamic environment.

Images