A port micro-grid dispatching method and system based on deep reinforcement learning

Abstract

The present application relates to the field of energy scheduling, and provides a port micro-grid scheduling method and system based on deep reinforcement learning, comprising: constructing a gas turbine action space and an electric ship action space; calculating a gas turbine environment penalty and an energy storage reset penalty, and constructing a comprehensive reward function; constructing a proximal optimization strategy algorithm comprising a first action network, a second action network and an evaluation network, calculating an action strategy gradient, calculating an evaluation strategy gradient of the evaluation network, training the first action network and the second action network, training the evaluation network, and obtaining a target prediction algorithm; obtaining gas turbine energy parameters and electric ship energy parameters, calculating an energy self-consistency rate through electric ship charging and discharging characteristics and the electric ship energy parameters, constructing an electric ship iteration equation and iterating to obtain electric ship charging and discharging output; and scheduling the micro-grid through the gas turbine energy parameters, the electric ship energy parameters and the electric ship charging and discharging output.

Description

Technical Field

[0001] This invention relates to the field of energy dispatching technology, and in particular to a port microgrid dispatching method and system based on deep reinforcement learning. Background Technology

[0002] Against the backdrop of increasing renewable energy utilization and optimizing the energy structure, microgrids incorporating a high proportion of renewable energy generation have emerged. Due to the abundance of natural resources such as sunlight, wind, and tides in offshore areas, and the fact that most ports are far from the main urban power grid, the construction of microgrids in port areas has a natural advantage and necessity. With the actual operation of more port microgrids, their energy optimization and dispatching has become one of the key issues in developing new productive forces and constructing new power systems. Currently, the dispatching of port microgrids only includes the port's own gas turbines and energy storage equipment, with less consideration given to the consumption of renewable energy such as wind turbines and photovoltaics, as well as the energy interaction process involving shipping traffic. The solution strategy usually adopts centralized dispatching, making it difficult to achieve optimal control of each part of the port microgrid. With the growth of port load and the continuous integration of electrified ships, the single energy supply method of the port power grid can no longer meet the ever-increasing energy demand of the port area. In summary, current research on low-carbon dispatching technology for power generation, grid, load, and storage still faces the following challenges. First, most studies focus on promoting the construction of distributed renewable energy generation, often neglecting the economic viability of self-operated regional microgrids. Second, due to the far greater prevalence and industry development speed of electric vehicles compared to electric ships, most research teams prioritize or only consider electric vehicles as research targets when studying orderly charging and discharging. However, the capacity, transport capacity, range, and charging / discharging modes of electric ships differ significantly from those of electric vehicles, making the conclusions of some studies inapplicable to the charging dispatching of electric ships and the construction of port microgrids. Furthermore, current research largely focuses on seeking single-objective optimal solutions, making it difficult to simultaneously and effectively address multiple issues such as the numerous hazards arising from renewable energy grid connection, load fluctuations ("flat-peak-valley"), and the economical operation of microgrids. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a port microgrid scheduling method and system based on deep reinforcement learning, enabling effective scheduling of the port's microgrid system.

[0004] This invention provides a port microgrid scheduling method based on deep reinforcement learning, comprising: S1: Construct intelligent agents for gas turbines and electric ships, and construct the action space for gas turbines and the action space for electric ships based on the intelligent agents for gas turbines and electric ships. S2: Calculate the gas turbine environmental penalty and energy storage reset penalty through the gas turbine action space, and construct a comprehensive reward function through the gas turbine environmental penalty and energy storage reset penalty; S3: Construct a proximal optimization policy algorithm based on the comprehensive reward function, including a first action network, a second action network, and an evaluation network. Calculate the action policy gradients of the first and second action networks, calculate the evaluation policy gradient of the evaluation network, train the first and second action networks using the action policy gradients, and train the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. S4: Use the target prediction algorithm, gas turbine action space and electric ship action space to obtain gas turbine energy parameters and electric ship energy parameters, calculate electric ship charging and discharging characteristics, calculate energy self-consistency rate through electric ship charging and discharging characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charging and discharging output. S5: The microgrid is scheduled using the energy parameters of the gas turbine, the energy parameters of the electric boat, and the charging and discharging output of the electric boat.

[0005] According to the port microgrid scheduling method based on deep reinforcement learning provided by the present invention, step S1 further includes; S11: Determine the state variables of the gas turbine and the state variables of the electric ship, construct the gas turbine intelligent agent based on the gas turbine state variables, and construct the electric ship intelligent agent based on the electric ship state variables; S12: Extract energy storage power and gas turbine power from the gas turbine intelligent agent, and construct the gas turbine operation space using the energy storage power and gas turbine power; extract electric ship user load and electric ship power from the electric ship intelligent agent, and construct the electric ship operation space using the electric ship user load and electric ship power.

[0006] According to the port microgrid scheduling method based on deep reinforcement learning provided by the present invention, step S2 further includes: S21: Determine the turbine ramp penalty coefficient and the energy storage penalty coefficient, calculate the gas turbine ramp penalty using the gas turbine operating space and the turbine ramp penalty coefficient, calculate the energy storage penalty using the energy storage penalty coefficient and the gas turbine operating space, and obtain the gas turbine environmental penalty including the gas turbine ramp penalty and the energy storage penalty; S22: Determine the reset penalty coefficient, calculate the energy storage reset penalty using the reset penalty coefficient, determine the operating cost parameters, and construct a comprehensive reward function using the operating cost parameters, gas turbine environmental penalty, and energy storage reset penalty.

[0007] According to the port microgrid scheduling method based on deep reinforcement learning provided by the present invention, in step S3, the near-end optimization strategy algorithm including a first action network, a second action network and an evaluation network is constructed with the objective of minimizing the comprehensive reward function. Calculate the first output action probability of the first action network, calculate the second output action probability of the second action network, and calculate the action policy gradient using the first output action probability and the second output action probability.

[0008] According to the port microgrid scheduling method based on deep reinforcement learning provided by the present invention, in step S3, the evaluation network obtains a state evaluation, calculates the evaluation policy gradient based on the state evaluation, updates the action network parameters of the first action network and the second action network using the gradient ascent method with the objective of maximizing the action policy gradient, updates the evaluation network parameters of the evaluation network using the gradient descent method with the objective of minimizing the evaluation policy gradient, and obtains the target prediction algorithm.

[0009] According to the port microgrid scheduling method based on deep reinforcement learning provided by the present invention, step S4 further includes: S41: Using the gas turbine operating space and the electric ship operating space as constraints, the target prediction algorithm is used to obtain the energy parameters of the gas turbine and the energy parameters of the electric ship. S42: Calculate the charging and discharging characteristics of the electric boat, and calculate the energy self-consistency rate using the energy parameters of the electric boat, the charging and discharging characteristics; S43: Obtain the matrix coefficients, construct the electric ship iterative equation through the energy self-consistency rate and the matrix coefficients, iterate the matrix coefficients in the electric ship iterative equation to obtain the target coefficient matrix, and obtain the electric ship charging and discharging output based on the target coefficient matrix.

[0010] This invention also provides a port microgrid scheduling system based on deep reinforcement learning, comprising: Action Space Module: Used to construct gas turbine intelligent agents and electric ship intelligent agents, and to construct gas turbine action spaces and electric ship action spaces based on gas turbine intelligent agents and electric ship intelligent agents; The integrated reward function module is used to calculate the gas turbine environmental penalty and the energy storage reset penalty through the gas turbine action space, and to construct the integrated reward function through the gas turbine environmental penalty and the energy storage reset penalty; The target prediction algorithm module is used to construct a proximal optimization policy algorithm including a first action network, a second action network, and an evaluation network based on the comprehensive reward function. It calculates the action policy gradients of the first action network and the second action network, calculates the evaluation policy gradient of the evaluation network, trains the first action network and the second action network using the action policy gradients, and trains the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. Charge and discharge output module: Used to obtain gas turbine energy parameters and electric ship energy parameters using target prediction algorithm, gas turbine action space and electric ship action space, calculate electric ship charge and discharge characteristics, calculate energy self-consistency rate through electric ship charge and discharge characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charge and discharge output. Grid dispatching module: Used to dispatch the microgrid based on gas turbine energy parameters, electric boat energy parameters, and electric boat charging and discharging output.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a port microgrid scheduling method based on deep reinforcement learning as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a port microgrid scheduling method based on deep reinforcement learning as described above.

[0013] The present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer is able to perform the steps of a port microgrid scheduling method based on deep reinforcement learning as described above.

[0014] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: This invention provides a port microgrid scheduling method and system based on deep reinforcement learning. By constructing a near-end optimization strategy algorithm and training the algorithm to minimize the comprehensive reward function, a target prediction algorithm is obtained, thereby obtaining the operating parameters of the gas turbine in the microgrid. The charging and discharging output of the electric ship is calculated through the charging and discharging characteristics of the electric ship, effectively scheduling the port microgrid and improving the energy utilization efficiency and grid stability of the port.

[0015] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a port microgrid scheduling method based on deep reinforcement learning provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of a port microgrid scheduling system based on deep reinforcement learning provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of a port microgrid scheduling device based on deep reinforcement learning provided by the present invention.

[0020] Figure label: 100. Action Space Module; 200. Comprehensive Reward Function Module; 300. Target Prediction Algorithm Module; 400. Charge / Discharge Output Module; 500. Power Grid Dispatching Module; 810. Processor; 820. Communication Interface; 830. Memory; 840. Communication Bus. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0022] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second" and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0024] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0025] The following is combined with Figures 1 to 3 Specific embodiments of the present invention are described below. Figure 1 This is a flowchart illustrating a port microgrid scheduling method based on deep reinforcement learning provided by the present invention. It includes: S1: Construct intelligent agents for gas turbines and electric ships, and construct the action space for gas turbines and the action space for electric ships based on the intelligent agents for gas turbines and electric ships. Furthermore, the objective of this stage is to construct the operational space for gas turbines and electric ships. Specifically, step S1 further includes; S11: Determine the state variables of the gas turbine and the state variables of the electric ship, construct the gas turbine intelligent agent based on the gas turbine state variables, and construct the electric ship intelligent agent based on the electric ship state variables; S12: Extract energy storage power and gas turbine power from the gas turbine intelligent agent, and construct the gas turbine operation space using the energy storage power and gas turbine power; extract electric ship user load and electric ship power from the electric ship intelligent agent, and construct the electric ship operation space using the electric ship user load and electric ship power.

[0026] The specific implementation method for the above steps in this embodiment is as follows: First, determine the photovoltaic power generation, including the current time t and time t. Wind power generation at time t Gas turbine power generation at time t Battery energy storage power at time t The energy storage device's power at time t The state variables of the gas turbine are used to construct a gas turbine intelligent agent. It is also necessary to determine the total load power of the electric boat, including time t. Total port user load at time t Port power grid bus power at time t Peak electricity price at time t And off-peak electricity prices The state variables of the electric ship are used to construct the intelligent agent of the electric ship. : This allows us to extract values ​​from it and construct the gas turbine's motion space. and the movement space of electric boats : in, Let t represent the load of electric boat users connected to the microgrid. Let t be the electric power of all electric boats at time t. The gas turbine motion space and electric boat motion space also constrain the range of values ​​of the parameters therein.

[0027] S2: Calculate the gas turbine environmental penalty and energy storage reset penalty through the gas turbine action space, and construct a comprehensive reward function through the gas turbine environmental penalty and energy storage reset penalty; Furthermore, the objective of this stage is to construct a comprehensive reward function using gas turbine environmental penalties and energy storage reset penalties. Specifically, step S2 further includes: S21: Determine the turbine ramp penalty coefficient and the energy storage penalty coefficient, calculate the gas turbine ramp penalty using the gas turbine operating space and the turbine ramp penalty coefficient, calculate the energy storage penalty using the energy storage penalty coefficient and the gas turbine operating space, and obtain the gas turbine environmental penalty including the gas turbine ramp penalty and the energy storage penalty; S22: Determine the reset penalty coefficient, calculate the energy storage reset penalty using the reset penalty coefficient, determine the operating cost parameters, and construct a comprehensive reward function using the operating cost parameters, gas turbine environmental penalty, and energy storage reset penalty.

[0028] The specific implementation method for the above steps in this embodiment is as follows: First, it is necessary to determine the engine ramp penalty coefficient. and energy storage penalty coefficient This allows us to calculate the gas turbine ramp penalty at time t using the gas turbine operating space and the turbine ramp penalty coefficient. : In this context, `max{}` represents the maximum value within the parentheses, and `min{}` represents the minimum value within the parentheses. Let t-1 be the power output of the gas turbine. This represents the upper limit of the rate of change of gas turbine power, which is the maximum allowable change in gas turbine power over the time interval between two moments. This represents the lower limit of the gas turbine power change rate, which is the minimum allowable change in gas turbine power over the time interval between two moments. Similarly, the energy storage penalty at time t is calculated using the energy storage penalty coefficient and the gas turbine operating space. : in, This represents the maximum allowable electrical capacity of the energy storage device. This is the minimum allowable electrical capacity for energy storage devices. This represents the rated capacity of the energy storage device. This yields the gas turbine environmental penalty, which includes both the gas turbine ramp penalty and the energy storage penalty.

[0029] The reset penalty coefficient was then determined. The energy storage reset penalty at time t is calculated using the reset penalty coefficient. : in, The exponential coefficient for the reset penalty. The initial charge of the energy storage device is the power level at the start of a scheduling cycle. This means that in the initial stage of a scheduling cycle, the reset penalty is small, but it increases over time, reaching its maximum at the end of the cycle. This ensures that the energy storage device's power level at the end of each scheduling cycle is as equal as possible to its initial charge level.

[0030] Determine the operating cost parameters of the gas turbine and energy storage equipment at time t. This allows us to construct a comprehensive reward function R using operating cost parameters, gas turbine environmental penalties, and energy storage reset penalties: Where T is the total number of moments in a scheduling cycle.

[0031] S3: Construct a proximal optimization policy algorithm based on the comprehensive reward function, including a first action network, a second action network, and an evaluation network. Calculate the action policy gradients of the first and second action networks, calculate the evaluation policy gradient of the evaluation network, train the first and second action networks using the action policy gradients, and train the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. Furthermore, the objective of this stage is to calculate the action policy gradients of the first action network and the second action network, thereby training the first action network and the second action network, and training the evaluation network to obtain the target prediction algorithm. Specifically, in step S3, with the objective of minimizing the comprehensive reward function, the proximal optimization policy algorithm comprising the first action network, the second action network, and the evaluation network is constructed. Calculate the first output action probability of the first action network, calculate the second output action probability of the second action network, and calculate the action policy gradient using the first output action probability and the second output action probability.

[0032] In step S3, the evaluation network obtains a state evaluation, calculates the evaluation policy gradient based on the state evaluation, updates the action network parameters of the first action network and the second action network using gradient ascent with the goal of maximizing the action policy gradient, and updates the evaluation network parameters of the evaluation network using gradient descent with the goal of minimizing the evaluation policy gradient, thus obtaining the target prediction algorithm.

[0033] The specific implementation method for the above steps in this embodiment is as follows: First, a proximal optimization strategy algorithm is constructed with the objective of minimizing the overall reward function. This algorithm includes a first action network, a second action network, and an evaluation network. The first action network is used to evaluate the input... This outputs the probability distribution for each action, and then performs probability sampling to generate the action. The immediate reward for this action is determined. Here, the action refers to the current power of the gas turbine, the power of the energy storage device, etc., and the input refers to the various operating parameters of the microgrid at this moment. The immediate reward is the value of the comprehensive reward function. The first action network is the latest action network used for interactive decision-making with the environment, using the latest network parameters θ. The second action network controls the update magnitude of the network parameters of the first action network, using the network parameters of the first action network from a previous time step. The evaluation network is used to evaluate the input and obtain a state evaluation. The instant rewards from the First Action Network are scored to obtain a reward score. The sum of the input evaluation and reward scores will be used as the evaluation target value at time t. .

[0034] Then, the probability of the first output action of the first action network is calculated. Calculate the probability of the second output action of the second action network. Therefore, the intermediate action parameters at time t are calculated first. : The intermediate action function at time t : in, The action coefficient is determined empirically, and for the clip function ,have: This allows us to calculate the action policy gradient of the first action network. : Where B is the amount of parameters that the action pool of the first action network can hold. This represents the gradient of the computation function with respect to the network parameters.

[0035] Subsequently, the evaluation network obtains a state evaluation, and the evaluation policy gradient is calculated using the state evaluation and the evaluation target value. : in, To calculate the gradient of the content within the parentheses, the gradient ascent method is then used to update the action network parameters of the first and second action networks, aiming to maximize the gradient of the action policy. Gradient descent is then used to update the evaluation network parameters, aiming to minimize the gradient of the evaluation policy. This process is repeated multiple times to train the first, second, and evaluation networks, thus obtaining the target prediction algorithm.

[0036] S4: Use the target prediction algorithm, gas turbine action space and electric ship action space to obtain gas turbine energy parameters and electric ship energy parameters, calculate electric ship charging and discharging characteristics, calculate energy self-consistency rate through electric ship charging and discharging characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charging and discharging output. Furthermore, the objective of this stage is to calculate the charging and discharging characteristics of the electric boat, and to calculate the energy self-consistency rate using these characteristics and energy parameters. This allows for the construction and iteration of the electric boat's iterative equations to obtain the charging and discharging output. Specifically, step S4 further includes: S41: Using the gas turbine operating space and the electric ship operating space as constraints, the target prediction algorithm is used to obtain the energy parameters of the gas turbine and the energy parameters of the electric ship. S42: Calculate the charging and discharging characteristics of the electric boat, and calculate the energy self-consistency rate using the energy parameters of the electric boat, the charging and discharging characteristics; S43: Obtain the matrix coefficients, construct the electric ship iterative equation through the energy self-consistency rate and the matrix coefficients, iterate the matrix coefficients in the electric ship iterative equation to obtain the target coefficient matrix, and obtain the electric ship charging and discharging output based on the target coefficient matrix.

[0037] The specific implementation method for the above steps in this embodiment is as follows: First, constraints on the value ranges within the gas turbine and electric ship operation spaces are used as limitations. A target prediction algorithm is then employed to predict the energy parameters of the gas turbine and electric ship. The gas turbine energy parameters include predictions of several values ​​within the gas turbine operation space, and the electric ship energy parameters include predictions of several values ​​within the electric ship operation space.

[0038] For electric boats, we need to perform further calculations. Based on the current charging plan of the electric boat obtained from its energy parameters, we need to calculate the charging characteristics of the i-th electric boat at time t. The discharge characteristics of the i-th electric boat at time t : in, Let the remaining power of the i-th electric boat being charged be the power at time t. The amount of electricity that the i-th electric boat, which is currently charging, plans to consume on its next voyage. For the i-th electric boat at the start of charging Time and end of charging The charging and discharging power between moments Let be the charging efficiency of the i-th electric boat. Let be the battery capacity of the i-th electric boat.

[0039] The charging and discharging characteristics are selected based on whether the electric boat is currently in a charging or discharging state, thus obtaining the charging and discharging characteristics of the i-th electric boat. Then, the charging power of the i-th electric boat at time t is calculated. : in, Let N be the fluctuating power of the port's microgrid system, and N be the number of electric ships participating in charging and discharging in the port.

[0040] Next, the energy self-consistency rate of the i-th electric boat at time t is calculated using the electric boat's energy parameters, charging characteristics, and discharging characteristics. : in, This represents the power consumption of all infrastructure in the port other than electric ships at time t. Let be the total power generation of the microgrid at time t. Let t be the total electricity consumption of all loads at the port at time t.

[0041] The matrix coefficients of the i-th electric boat are then determined. Thus, the iterative equations for the electric ship can be constructed using the energy self-consistency rate and matrix coefficients: in, Let be the energy self-consistency rate of the i-th electric ship at time t+1. The convergence speed step size is determined empirically. Let be the charge / discharge difference of the i-th electric ship at time t. The first iteration coefficient for the i-th electric boat is determined empirically. The second iteration coefficient for the i-th electric boat is determined empirically. Let be the charging power of the i-th electric boat at time t+1. Let be the charge / discharge difference of the i-th electric ship at time t.

[0042] By using iterative equations for electric boats, relevant parameters of the charging and discharging process of each electric boat can be predicted. During this process, the actual changes in these data are continuously monitored. The matrix coefficients are iterated based on the difference between the predicted and actual values, adjusting them to make the predictions closer to reality. This process continues until the error between the predicted and actual values ​​is consistently less than an empirically determined threshold. The iteratively obtained matrix coefficients for each electric boat over a future period are then used to form a target coefficient matrix. This target coefficient matrix is ​​then used to predict the energy self-sufficiency and charging power of each boat. Based on the prediction results, parameters such as the charging and discharging power of the electric boats are corrected and predicted again for continuous optimization, improving the overall energy self-sufficiency of the electric boats. The optimized charging and discharging power and other parameters are then used as the charging and discharging output of the electric boats.

[0043] S5: The microgrid is scheduled using the energy parameters of the gas turbine, the energy parameters of the electric boat, and the charging and discharging output of the electric boat.

[0044] Furthermore, after obtaining the gas turbine energy parameters, the gas turbine in the microgrid can be driven according to the gas turbine energy parameters; the charging and discharging process of the electric ship can be controlled by the electric ship energy parameters and the electric ship charging and discharging output, thereby scheduling the port's microgrid.

[0045] The present invention provides a port microgrid scheduling method based on deep reinforcement learning, which can schedule the port microgrid, thereby improving the port's energy utilization efficiency and grid stability.

[0046] The following describes a port microgrid scheduling device based on deep reinforcement learning provided by the present invention. The port microgrid scheduling device based on deep reinforcement learning described below and the port microgrid scheduling method based on deep reinforcement learning described above can be referred to and correspond to each other.

[0047] Figure 2 An example is a schematic diagram of a port microgrid scheduling system based on deep reinforcement learning, as shown below. Figure 2 As shown, a port microgrid scheduling method based on deep reinforcement learning, as described above, includes: Action Space Module 100: Used to construct gas turbine intelligent agents and electric ship intelligent agents, and to construct gas turbine action spaces and electric ship action spaces based on gas turbine intelligent agents and electric ship intelligent agents; The comprehensive reward function module 200 is used to calculate the gas turbine environmental penalty and the energy storage reset penalty through the gas turbine action space, and to construct the comprehensive reward function through the gas turbine environmental penalty and the energy storage reset penalty; Target prediction algorithm module 300: is used to construct a proximal optimization policy algorithm including a first action network, a second action network and an evaluation network based on the comprehensive reward function, calculate the action policy gradient of the first action network and the second action network, calculate the evaluation policy gradient of the evaluation network, train the first action network and the second action network through the action policy gradient, train the evaluation network through the evaluation policy gradient, and obtain the target prediction algorithm. Charge and discharge output module 400: Used to obtain gas turbine energy parameters and electric ship energy parameters by using target prediction algorithm, gas turbine action space and electric ship action space, calculate electric ship charge and discharge characteristics, calculate energy self-consistency rate by electric ship charge and discharge characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation by energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charge and discharge output; Grid dispatching module 500: Used to dispatch the microgrid based on gas turbine energy parameters, electric boat energy parameters, and electric boat charging and discharging output.

[0048] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call a computer program in the memory 830 to execute a port microgrid scheduling method based on deep reinforcement learning, the method including: S1: Construct intelligent agents for gas turbines and electric ships, and construct the action space for gas turbines and the action space for electric ships based on the intelligent agents for gas turbines and electric ships. S2: Calculate the gas turbine environmental penalty and energy storage reset penalty through the gas turbine action space, and construct a comprehensive reward function through the gas turbine environmental penalty and energy storage reset penalty; S3: Construct a proximal optimization policy algorithm based on the comprehensive reward function, including a first action network, a second action network, and an evaluation network. Calculate the action policy gradients of the first and second action networks, calculate the evaluation policy gradient of the evaluation network, train the first and second action networks using the action policy gradients, and train the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. S4: Use the target prediction algorithm, gas turbine action space and electric ship action space to obtain gas turbine energy parameters and electric ship energy parameters, calculate electric ship charging and discharging characteristics, calculate energy self-consistency rate through electric ship charging and discharging characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charging and discharging output. S5: The microgrid is scheduled using the energy parameters of the gas turbine, the energy parameters of the electric boat, and the charging and discharging output of the electric boat.

[0049] Furthermore, when the computer program in the aforementioned memory 830 can be implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0050] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute a port microgrid scheduling method based on deep reinforcement learning provided by the above methods, the method comprising: S1: Construct intelligent agents for gas turbines and electric ships, and construct the action space for gas turbines and the action space for electric ships based on the intelligent agents for gas turbines and electric ships. S2: Calculate the gas turbine environmental penalty and energy storage reset penalty through the gas turbine action space, and construct a comprehensive reward function through the gas turbine environmental penalty and energy storage reset penalty; S3: Construct a proximal optimization policy algorithm based on the comprehensive reward function, including a first action network, a second action network, and an evaluation network. Calculate the action policy gradients of the first and second action networks, calculate the evaluation policy gradient of the evaluation network, train the first and second action networks using the action policy gradients, and train the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. S4: Use the target prediction algorithm, gas turbine action space and electric ship action space to obtain gas turbine energy parameters and electric ship energy parameters, calculate electric ship charging and discharging characteristics, calculate energy self-consistency rate through electric ship charging and discharging characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charging and discharging output. S5: The microgrid is scheduled using the energy parameters of the gas turbine, the energy parameters of the electric boat, and the charging and discharging output of the electric boat.

[0051] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the aforementioned port microgrid scheduling method based on deep reinforcement learning, the method comprising: S1: Construct intelligent agents for gas turbines and electric ships, and construct the action space for gas turbines and the action space for electric ships based on the intelligent agents for gas turbines and electric ships. S2: Calculate the gas turbine environmental penalty and energy storage reset penalty through the gas turbine action space, and construct a comprehensive reward function through the gas turbine environmental penalty and energy storage reset penalty; S3: Construct a proximal optimization policy algorithm based on the comprehensive reward function, including a first action network, a second action network, and an evaluation network. Calculate the action policy gradients of the first and second action networks, calculate the evaluation policy gradient of the evaluation network, train the first and second action networks using the action policy gradients, and train the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. S4: Use the target prediction algorithm, gas turbine action space and electric ship action space to obtain gas turbine energy parameters and electric ship energy parameters, calculate electric ship charging and discharging characteristics, calculate energy self-consistency rate through electric ship charging and discharging characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charging and discharging output. S5: The microgrid is scheduled using the energy parameters of the gas turbine, the energy parameters of the electric boat, and the charging and discharging output of the electric boat.

[0052] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0053] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A port microgrid scheduling method based on deep reinforcement learning, characterized in that, include: S1: Construct intelligent agents for gas turbines and electric ships, and construct the action space for gas turbines and the action space for electric ships based on the intelligent agents for gas turbines and electric ships. S2: Calculate the gas turbine environmental penalty and energy storage reset penalty through the gas turbine action space, and construct a comprehensive reward function through the gas turbine environmental penalty and energy storage reset penalty; S3: Construct a proximal optimization policy algorithm based on the comprehensive reward function, including a first action network, a second action network, and an evaluation network. Calculate the action policy gradients of the first and second action networks, calculate the evaluation policy gradient of the evaluation network, train the first and second action networks using the action policy gradients, and train the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. S4: Use the target prediction algorithm, gas turbine action space and electric ship action space to obtain gas turbine energy parameters and electric ship energy parameters, calculate electric ship charging and discharging characteristics, calculate energy self-consistency rate through electric ship charging and discharging characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charging and discharging output. S5: The microgrid is scheduled using the energy parameters of the gas turbine, the energy parameters of the electric boat, and the charging and discharging output of the electric boat.

2. The port microgrid scheduling method based on deep reinforcement learning according to claim 1, characterized in that, Step S1 further includes; S11: Determine the state variables of the gas turbine and the state variables of the electric ship, construct the gas turbine intelligent agent based on the gas turbine state variables, and construct the electric ship intelligent agent based on the electric ship state variables; S12: Extract energy storage power and gas turbine power from the gas turbine intelligent agent, and construct the gas turbine operation space using the energy storage power and gas turbine power; extract electric ship user load and electric ship power from the electric ship intelligent agent, and construct the electric ship operation space using the electric ship user load and electric ship power.

3. The port microgrid scheduling method based on deep reinforcement learning according to claim 1, characterized in that, Step S2 further includes: S21: Determine the turbine ramp penalty coefficient and the energy storage penalty coefficient, calculate the gas turbine ramp penalty using the gas turbine operating space and the turbine ramp penalty coefficient, calculate the energy storage penalty using the energy storage penalty coefficient and the gas turbine operating space, and obtain the gas turbine environmental penalty including the gas turbine ramp penalty and the energy storage penalty; S22: Determine the reset penalty coefficient, calculate the energy storage reset penalty using the reset penalty coefficient, determine the operating cost parameters, and construct a comprehensive reward function using the operating cost parameters, gas turbine environmental penalty, and energy storage reset penalty.

4. The port microgrid scheduling method based on deep reinforcement learning according to claim 1, characterized in that, In step S3, with the objective of minimizing the comprehensive reward function, the proximal optimization strategy algorithm, which includes a first action network, a second action network, and an evaluation network, is constructed. Calculate the first output action probability of the first action network, calculate the second output action probability of the second action network, and calculate the action policy gradient using the first output action probability and the second output action probability.

5. The port microgrid scheduling method based on deep reinforcement learning according to claim 1, characterized in that, In step S3, the evaluation network obtains a state evaluation, calculates the evaluation policy gradient based on the state evaluation, updates the action network parameters of the first action network and the second action network using gradient ascent with the goal of maximizing the action policy gradient, and updates the evaluation network parameters of the evaluation network using gradient descent with the goal of minimizing the evaluation policy gradient, thus obtaining the target prediction algorithm.

6. The port microgrid scheduling method based on deep reinforcement learning according to claim 1, characterized in that, Step S4 further includes: S41: Using the gas turbine operating space and the electric ship operating space as constraints, the target prediction algorithm is used to obtain the energy parameters of the gas turbine and the energy parameters of the electric ship. S42: Calculate the charging and discharging characteristics of the electric boat, and calculate the energy self-consistency rate using the energy parameters of the electric boat, the charging and discharging characteristics; S43: Obtain the matrix coefficients, construct the electric ship iterative equation through the energy self-consistency rate and the matrix coefficients, iterate the matrix coefficients in the electric ship iterative equation to obtain the target coefficient matrix, and obtain the electric ship charging and discharging output based on the target coefficient matrix.

7. A port microgrid scheduling system based on deep reinforcement learning, used to execute a port microgrid scheduling method based on deep reinforcement learning as described in any one of claims 1 to 6, characterized in that, include: Action Space Module: Used to construct gas turbine intelligent agents and electric ship intelligent agents, and to construct gas turbine action spaces and electric ship action spaces based on gas turbine intelligent agents and electric ship intelligent agents; The integrated reward function module is used to calculate the gas turbine environmental penalty and the energy storage reset penalty through the gas turbine action space, and to construct the integrated reward function through the gas turbine environmental penalty and the energy storage reset penalty; The target prediction algorithm module is used to construct a proximal optimization policy algorithm including a first action network, a second action network, and an evaluation network based on the comprehensive reward function. It calculates the action policy gradients of the first action network and the second action network, calculates the evaluation policy gradient of the evaluation network, trains the first action network and the second action network using the action policy gradients, and trains the evaluation network using the evaluation policy gradients to obtain the target prediction algorithm. Charge and discharge output module: Used to obtain gas turbine energy parameters and electric ship energy parameters using target prediction algorithm, gas turbine action space and electric ship action space, calculate electric ship charge and discharge characteristics, calculate energy self-consistency rate through electric ship charge and discharge characteristics and electric ship energy parameters, construct matrix coefficients, construct electric ship iterative equation through energy self-consistency rate and matrix coefficients, iterate the electric ship iterative equation to obtain electric ship charge and discharge output. Grid dispatching module: Used to dispatch the microgrid based on gas turbine energy parameters, electric boat energy parameters, and electric boat charging and discharging output.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a port microgrid scheduling method based on deep reinforcement learning as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of a port microgrid scheduling method based on deep reinforcement learning as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that, When the program instructions are executed by the computer, the computer is able to perform the steps of the port microgrid scheduling method based on deep reinforcement learning as described in any one of claims 1 to 6.

Images