Energy storage ship energy transportation scheduling method based on deep reinforcement learning

Abstract

The invention provides an energy storage ship energy transportation scheduling method based on deep reinforcement learning, and the scheduling method is based on an island-fish raft-coast energy transportation model containing an AES, and carries out the energy scheduling through transporting an energy storage battery at a coast battery swap station or an island to a fish raft through a full electric power ship AES. Solving by using a deep reinforcement learning method of a parameterized double-depth Q network based on multi-batch forward propagation to obtain an energy transportation strategy required by the energy transportation model; according to the method, an effective approximate optimal solution can be provided in the case of time and resource limitation in practical application, and the method can be suitable for solving a larger-scale energy transportation problem.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning and energy transportation scheduling, and in particular to a method for energy transportation scheduling of energy storage ships based on deep reinforcement learning, and specifically to a large-scale island-fishing raft-coast energy transportation scheduling method containing all-electric ships based on deep reinforcement learning in a hybrid action space. Background Art

[0002] As global energy demand continues to grow, traditional fossil fuel resources are becoming increasingly depleted and environmental pollution problems are becoming more serious. In order to meet this severe challenge, offshore new energy has gradually become an important alternative energy source. However, offshore new energy is intermittent and unstable.

[0003] At the same time, fish rafts face many challenges in terms of energy supply, and they mostly rely on diesel generators to provide electricity. In order to cope with the energy dilemma of fish rafts, energy interconnection between fish rafts and nearby islands is considered to be one of the potential solutions. Energy transmission methods based on mobile energy storage transfer have gradually attracted widespread attention and research in recent years.

[0004] At present, the methods for solving the range optimization problem of all-electric ships with energy transportation attributes can be divided into two categories: ① traditional methods; ② heuristic algorithms.

[0005] In traditional methods, the energy transportation process is mainly modeled as a mixed integer optimization model, and the optimal scheduling strategy is obtained using, for example, a CPLEX solver.

[0006] The representative algorithms in the ship voyage planning problem based on heuristic algorithms are the chaotic gravitational search algorithm and the improved ant colony algorithm.

[0007] In recent years, Deep Reinforcement Learning (DRL) has become an important tool for solving complex decision-making problems. Similar to heuristic algorithms, DRL does not rely on detailed modeling of the system, but compared to heuristic algorithms, DRL's advantage is that it can make real-time decisions after training and provide the best decision based on the current system state, so it performs better in dealing with uncertainty in the system. However, traditional methods are difficult to adapt to the increase in the scale of the above problems and the heuristic algorithms need to simplify the model to a certain extent, resulting in inaccurate models.

[0008] Traditional DRL methods are usually only applicable to MDPs containing a single action type. However, there are many problems in reality that require making decisions on both discrete and continuous variables. The traditional PDQN algorithm for mixed action spaces has a large number of invalid gradients generated during the network update process, which has a negative impact on the algorithm effect and overestimates the Q value.

[0009] Based on this, the present invention proposes a large-scale island-fishing raft-coast energy transportation scheduling method containing energy storage ships based on deep reinforcement learning. Summary of the invention

[0010] The present invention proposes a method for energy transportation scheduling of energy storage ships based on deep reinforcement learning, which can be applicable to large-scale island-fishing raft-coast energy transportation scheduling containing energy storage ships. By establishing an island-fishing raft group energy transportation model containing energy storage ships, the energy transportation problem is modeled as a Markov decision process, and a parameterized double-depth Q network algorithm based on multi-batch forward propagation is used to train the intelligent agent. Multi-batch forward propagation is introduced in the PDQN algorithm update process, and the algorithm hyperparameters and various coefficients are adjusted. The algorithm is trained until convergence to obtain the algorithm result. When faced with time and resource constraints in practical applications, the present invention can provide an effective approximate optimal solution and can be applicable to solving larger-scale energy transportation problems, which has important practical significance.

[0011] The present invention adopts the following technical solutions.

[0012] A method for energy transportation scheduling of energy storage ships based on deep reinforcement learning, the scheduling method is based on an island-fishing raft-coast energy transportation model including AES, and energy scheduling is performed by transporting energy storage batteries from a battery swap station at the coast or an island to the fishing raft using an all-electric ship AES. The method is solved using a deep reinforcement learning method based on a multi-batch forward propagation parameterized double deep Q-network (MBF-PDDQN) to obtain the energy transportation strategy required by the energy transportation model.

[0013] The energy storage battery is a mobile energy storage battery, i.e., a power battery. The method solves an island-offshore fishing raft-coastal energy transportation model containing an all-electric ship (AES) by using a deep reinforcement learning method of a parameterized double-depth Q network based on multi-batch forward propagation; the steps included are as follows:

[0014] Step S1: Establish an island-fishing raft energy transportation model with an all-electric ship AES containing energy storage. Step S2: Model the energy transportation problem as a Markov decision process, and clearly define the state space, action space and reward function in the Markov decision process;

[0015] Step S3: Adjust the algorithm hyperparameters, train the agent using a parameterized dual-depth Q network algorithm based on multi-batch forward propagation, and train the algorithm until convergence.

[0016] Step S1 includes the following steps:

[0017] Step S11: Design the AES model and establish the relationship between the actual propulsion power, speed and displacement of the AES as follows: s =A×V 3 ×(M 0 +MN s ) 2 / 3

[0018] Among them, A is the inherent parameter of the ship; V is the AES sailing speed, which is a certain value; M 0 is the displacement of AES when it is empty, i.e. the mass of water displaced when the ship is empty; M is the displacement caused by AES loading cargo; N s Load AES with mobile energy storage batteries quantity AES;

[0019]

[0020] Among them, E s,b The maximum capacity of a single marine power battery; is the floor rounding symbol; Indicates the number of fully charged power batteries in AES at time t; Indicates the number of unloaded power batteries in AES at time t; Indicates the charge of the marine power battery in the AES under non-full charge and no-load conditions at time t;

[0021] Step S12: designing a mobile energy storage device model;

[0022] The mathematical model of mobile energy storage battery is as follows:

[0023]

[0024] 0≤E i,t ≤E i,max

[0025] in, are the charging and discharging power of a single mobile energy storage battery i; E i,max P is the maximum charge capacity of a single mobile energy storage battery; i,max is the maximum charge and discharge power of a single mobile energy storage battery; k i,t is a 0-1 variable that marks the charge and discharge status of the energy storage battery. i,t When the value is 1, it means that the energy storage battery is in charging state. Both are set to 1.

[0026] Step S2 includes the design of the scheduling algorithm, specifically: for the established algorithm to be applied to the energy transportation problem, three quantities need to be designed first: state space, action space, and reward function; the execution process of the algorithm is: the state space is used as the input of the algorithm, and the algorithm parameters are updated in a gradient optimization manner based on the feedback signal provided by the reward function, and the action space is used as the output scheduling strategy of the algorithm; the specific definition is:

[0027] The state space is determined according to the scenario and used as the input of the algorithm: the islands, fishing rafts, and coasts at the dispatching locations are numbered with numbers such as 0, 1, and 2 to represent the location mark; the state space is designed as: dispatching time t, the number of fully charged mobile energy storage batteries loaded on the AES at time t, the location mark of the AES at time t, the charge of the marine power battery in the AES under non-fully charged and no-load conditions at time t, the number of fully charged mobile energy storage batteries at all locations at time t, the charge of incompletely charged mobile energy storage batteries at all locations at time t, and the number of fully discharged mobile energy storage batteries at all locations at time t.

[0028] Step S2 comprises the following steps:

[0029] Step S21: Design the state space, which is the output of the algorithm, that is, the specific dispatching action to be performed is as follows: First, AES obtains the target location to be traveled to according to the decision of the algorithm: AES obtains the target location to be traveled to according to the output decision of the algorithm. There are three types of locations in the model, namely, fishing rafts, islands with rich wind and solar energy resources, and shores; Then, when AES chooses to travel to the above locations, it needs to make further decisions based on the location type: it is divided into three situations;

[0030] In the first case, if the AES reaches the ship's AES battery resource replenishment location (i.e., the shore), the algorithm outputs the ratio of the fully charged ship power battery to the empty power battery on board, thereby achieving the purpose of charging the AES power battery;

[0031] In the second case, if the AES reaches the fishing raft, the algorithm outputs the ratio of the battery quantity exchanged with the fishing raft to the total battery quantity that can be exchanged, thus realizing the battery exchange between the AES and the fishing raft;

[0032] The third type is that when the AES reaches an island rich in wind and solar energy resources, the algorithm outputs the ratio of the amount of batteries exchanged with the island to the total amount of batteries that can be exchanged, thus realizing the battery exchange between the AES and the island;

[0033] During the entire scheduling cycle, AES can plan the navigation route through the action space given by the algorithm: when the number of empty mobile batteries stored in the fishing raft accounts for more than 60% of the total number of batteries at a certain moment, the fishing raft will have a large demand for energy transfer, and AES can go to the fishing raft to exchange the fully charged mobile batteries loaded by AES with the empty mobile batteries in the fishing raft, so that the fishing raft can avoid using its diesel generator for power replenishment as much as possible; in other time periods, AES will stay moored as much as possible to avoid unnecessary energy loss and navigation costs during navigation; when the island will be abandoned according to the energy transfer demand, AES can go to the island to exchange the empty mobile batteries loaded by AES with the fully charged mobile batteries in the island to increase the island's ability to absorb renewable energy; when AES does not have enough power for navigation every time it decides on the next target location, AES will go to the battery exchange point. During the entire scheduling cycle, AES can avoid unnecessary navigation as much as possible and only go to places with greater energy demand, thereby reducing navigation costs;

[0034] This step establishes the following state space:

[0035]

[0036] in, represents the number of fully charged mobile energy storage batteries loaded by AES at time t; p s,t Indicates the position mark of AES at time t; s i,p,t represents the information in location i when AES is at location p; Indicates the number of fully charged mobile energy storage batteries at location i; E i,t The charge of the partially charged mobile energy storage battery at the representative location i; The number of fully-charged mobile energy storage batteries at the representative location i;

[0037] Step S22: designing the action space;

[0038] Establish a mixed action space A containing discrete and continuous actions; expressed in the formula:

[0039] N={1,2,…n}

[0040] The values ​​1-n are the location numbers; there are three types of locations in the model, namely: fishing rafts {n raft}, islands rich in scenery and light resources {n island} and the shore {n shore}:

[0041] A={(p,a p )|p∈N,a p ∈χ p}

[0042]

[0043] 0≤a p ≤1

[0044] Where χ is a set of selectable continuous actions, where a s It indicates the ratio of the fully charged marine power battery to the empty power battery on board when the AES reaches the ship's AES resource replenishment location; a c,r It indicates the ratio of the number of batteries to be replaced when AES reaches the fishing raft to the total number of batteries that can be replaced; c,i It indicates the proportion of batteries that can be exchanged when AES reaches an island rich in wind and solar resources;

[0045] Step S23: designing a reward function;

[0046] In this step, the reward function is designed reasonably to enable the agent to learn the action strategy that meets the goal more quickly. The reward function contains system costs and some guiding items, specifically:

[0047] First, suppose that the penalty for failure to operate normally is that the AES does not have enough power to sail each time it decides on the next target location. Then the reward function is set as follows: the penalty for the AES failure to operate normally and the penalty for the AES remaining power being insufficient to ensure that it goes to the power exchange location, the energy transfer demand of various places including islands, shores and fish rafts, and the total cost of the system. The total cost includes the cost of wind and solar abandonment on islands and fish rafts, the operating cost of the fish raft diesel generator, the operating cost of the AES and the sum of the labor cost; the operating cost of the AES specifically refers to the power exchange cost and the unit kilometer navigation cost;

[0048] The reward function used to update the agent's policy network and value network parameters is set as follows:

[0049]

[0050] Among them, fa r,e ,fa start,e They represent the penalty for AES failure to operate normally and the penalty for insufficient remaining power of AES to ensure that it goes to the battery replacement place. If the AES navigation time constraint is not met, a penalty is incurred, otherwise it is set to 0; α, β, γ are coefficients used to balance the order of magnitude of penalty, reward and cost; r p,e The energy transfer demand of various regions; F e represents the total cost of the system during the e-th operation of AES; where r p,e Different calculation methods are used depending on the location type:

[0051]

[0052]

[0053] First, calculate the total system cost as shown below:

[0054]

[0055] C s (k) = p s ΔE s,k +ε s ΔS k

[0056] C l (k) = ε l N tras,k

[0057] Among them, C rn (k),C pg (k),C s (k) are the total wind and solar curtailment costs of the system, the operating costs of the system diesel generators, and the operating costs of the AES; is the predicted output of new energy at location i at time k; P rn,i (k) is the actual output of renewable energy at location i at time k; ε rn is the unit energy abandonment cost; P pg,i (k) is the output of the diesel generator set at location i at time k; a 0 ,a 1 ,a 2 is the diesel generator operating cost coefficient; the AES operating cost includes the battery replacement cost and the unit kilometer navigation cost, ΔE s,k is the power battery capacity of AES for battery replacement at time k; p s Indicates the battery replacement cost, in yuan / kWh; ΔS k is the AES navigation distance in the [k-1, k] period; ε s C is the unit distance cost of AES navigation; l (k) is the labor cost; ε l To transfer the cost of a single mobile energy storage battery; N tras,k is the number of movable energy storage batteries transferred in the system at time k.

[0058] In step S23, when establishing the reward function, the constraints are as follows:

[0059] Constraints 1. Power balance constraints:

[0060]

[0061] Where i is the location number, represents the output of the diesel generator cluster at location i in the microgrid at time t; It represents the actual output of renewable energy at microgrid i at time t; represents the total charging and discharging power of the mobile energy storage cluster at microgrid i at time t; represents the electrical load at microgrid i at time t;

[0062] Constraint 2: AES flight time constraint:

[0063] T i n-1 ≥T i,j j∈N

[0064] Among them, T i n represents the remaining navigable time when AES is at position i after making the n-1th decision; T i,j It represents the time required for AES to travel from location i to location j when making the nth decision. The above AES navigation time constraint formula indicates that the remaining navigation time of AES should be greater than the time required for AES to travel from location i to the next decision location each time it makes a decision.

[0065] In addition, it is necessary to ensure that the remaining power of AES is sufficient to reach the battery replacement site each time it decides on the next target location, which can be expressed as:

[0066] T i n-1 ≥T i,0 .

[0067] Step S3 is as follows: adjust the algorithm hyperparameters, introduce multi-batch forward propagation based on the PDDQN algorithm, use the neural network established by Python's Pytorch library to forward batch parallel data at the same time, and separate unrelated discrete actions from continuous actions without changing the algorithm network framework; p As input, a forward pass is performed for each action p, where e p is a standard basis vector of dimension n, then ae p for:

[0068] ae p =(0,…,0,a p ,0,…,0)

[0069] e p =(0,…,0,1,0,…,0)

[0070] Set up a batch of size n for a single forward pass through the batch-parallel data, thus:

[0071]

[0072] So we can get all the Q values: The Q value is used to update the neural network parameters and the algorithm is trained until convergence.

[0073] The AES is only used to load and transport mobile energy storage batteries (the number of batteries to be loaded depends on the specific scenario); the AES is powered by only one power battery at any given moment, and the AES is replenished with power through the battery swap station at the coast. Every time the AES reaches the shore, it swaps the unloaded power battery with the fully charged power battery at the battery swap station to achieve rapid energy supply for the AES;

[0074] The AES navigation speed of the all-electric ship is a predetermined value that matches the general ship specifications.

[0075] The island is modeled as an island rich in wind and solar energy resources, and is equipped with power generation equipment including wind turbines and photovoltaic units for outputting wind and solar energy and charging mobile energy storage batteries. The coast is equipped with a shore battery swap station, and the all-electric ship AES can exchange power batteries in an unloaded state with power batteries in a fully charged state at the shore battery swap station.

[0076] The power generation equipment equipped on the fish cage includes wind turbines, photovoltaic units, and diesel generators (due to the concentrated load on the fish cage, the fish cage's ability to utilize renewable energy is relatively limited); in order to meet the electricity demand of the fish cage, an all-electric ship AES is used to achieve energy transfer, allowing the fish cage to utilize surplus electricity in external wind and light energy resource-rich areas.

[0077] The method described in the present invention establishes an island-fishing raft energy transportation model containing energy storage ships, models the energy transportation problem as a Markov decision process, uses a parameterized double-depth Q network algorithm based on multi-batch forward propagation to train an intelligent agent, introduces multi-batch forward propagation in the PDQN algorithm update process, adjusts algorithm hyperparameters and various coefficients, trains the algorithm until convergence, and obtains algorithm results; the method can be applicable to large-scale island-fishing raft-coast energy transportation scheduling containing energy storage ships.

[0078] The present invention can provide an effective approximate optimal solution when faced with time and resource constraints in practical applications, and can be applied to solving larger-scale energy transportation problems, which has important practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0080] Attached Figure 1 It is a schematic diagram of the principle of the PDDQN algorithm of the present invention;

[0081] Attached Figure 2 This is a schematic diagram of the principle of action selection of the MBF-PDDQN algorithm of the present invention;

[0082] Attached Figure 3 A schematic diagram of the relative positions and numbers of various locations in an embodiment of the present invention;

[0083] Attached Figure 4 This is a schematic diagram of the day-ahead predicted power load at various locations in an embodiment of the present invention;

[0084] Attached Figure 5 This is a schematic diagram of the day-ahead prediction of new energy output at various locations in an embodiment of the present invention;

[0085] Attached Figure 6 It is a schematic diagram of the net load curve of each location considering the source load fluctuation in the embodiment of the present invention;

[0086] Attached Figure 7 Schematic diagram of reward change curve during the training process of each algorithm in the embodiment of the present invention;

[0087] Attached Figure 8 Schematic diagram of penalty change curves during the training process of each algorithm in the embodiments of the present invention;

[0088] Attached Fig. 9 Schematic diagram of Q network gradient during the training process of MBF-PDDQN and PDDQN in an embodiment of the present invention;

[0089] Attached Fig.10 A schematic diagram of the relationship between AES path selection and demand value changes at various locations in an embodiment of the present invention;

[0090] Attached Fig.11 Schematic diagram of the number of onboard batteries and SOC changes of AES power batteries at various locations in an embodiment of the present invention;

[0091] Attached Fig.12 Schematic diagram of cost values ​​obtained by MBF-PDDQN in various scenarios in an embodiment of the present invention;

[0092] Attached Fig.13 Schematic diagram of penalty values ​​obtained by MBF-PDDQN in various scenarios in an embodiment of the present invention;

[0093] Attached Fig.14 Schematic diagram of information solution using the traditional method and MBF-PDDQN in different node scenarios in an embodiment of the present invention. DETAILED DESCRIPTION

[0094] As shown in the figure, a method for energy transportation scheduling of energy storage ships based on deep reinforcement learning is provided. The scheduling method is based on an island-fishing raft-coast energy transportation model including AES. Energy scheduling is performed by transporting energy storage batteries from battery swap stations on the coast or islands to fishing rafts using an all-electric ship AES. The method is solved using a deep reinforcement learning method based on a multi-batch forward propagation parameterized double deep Q network (Multi-Batch Forward-Parameterized Double Deep Q-network, MBF-PDDQN) to obtain the energy transportation strategy required by the energy transportation model.

[0095] The energy storage battery is a mobile energy storage battery, i.e., a power battery. The method solves an island-offshore fishing raft-coastal energy transportation model containing an all-electric ship (AES) by using a deep reinforcement learning method of a parameterized double-depth Q network based on multi-batch forward propagation; the steps included are as follows:

[0096] Step S1: Establish an island-fishing raft energy transportation model with an all-electric ship AES containing energy storage. Step S2: Model the energy transportation problem as a Markov decision process, and clearly define the state space, action space and reward function in the Markov decision process;

[0097] Step S3: Adjust the algorithm hyperparameters, train the agent using a parameterized dual-depth Q network algorithm based on multi-batch forward propagation, and train the algorithm until convergence.

[0098] Step S1 includes the following steps:

[0099] Step S11: Design the AES model and establish the relationship between the actual propulsion power, speed and displacement of the AES as follows: s =A×V 3 ×(M 0 +MN s ) 2 / 3

[0100] Among them, A is the inherent parameter of the ship; V is the AES sailing speed, which is a certain value; M 0 is the displacement of AES when it is empty, i.e. the mass of water displaced when the ship is empty; M is the displacement caused by AES loading cargo; N s Load AES with mobile energy storage batteries quantity AES;

[0101]

[0102] Among them, E s,b The maximum capacity of a single marine power battery; is the floor rounding symbol; Indicates the number of fully charged power batteries in AES at time t; Indicates the number of unloaded power batteries in AES at time t; Indicates the charge of the marine power battery in the AES under non-full charge and no-load conditions at time t;

[0103] Step S12: designing a mobile energy storage device model;

[0104] The mathematical model of mobile energy storage battery is as follows:

[0105]

[0106] 0≤E i,t ≤E i,max

[0107] in, are the charging and discharging power of a single mobile energy storage battery i; E i,max P is the maximum charge capacity of a single mobile energy storage battery; i,max is the maximum charge and discharge power of a single mobile energy storage battery; k i,t is a 0-1 variable that marks the charge and discharge status of the energy storage battery. i,t When the value is 1, it means that the energy storage battery is in charging state. Both are set to 1.

[0108] Step S2 includes the design of the scheduling algorithm, specifically: for the established algorithm to be applied to the energy transportation problem, three quantities need to be designed first: state space, action space, and reward function; the execution process of the algorithm is: the state space is used as the input of the algorithm, and the algorithm parameters are updated in a gradient optimization manner based on the feedback signal provided by the reward function, and the action space is used as the output scheduling strategy of the algorithm; the specific definition is:

[0109] The state space is determined according to the scenario and used as the input of the algorithm: the islands, fishing rafts, and coasts at the dispatching locations are numbered with numbers such as 0, 1, and 2 to represent the location mark; the state space is designed as: dispatching time t, the number of fully charged mobile energy storage batteries loaded on the AES at time t, the location mark of the AES at time t, the charge of the marine power battery in the AES under non-fully charged and no-load conditions at time t, the number of fully charged mobile energy storage batteries at all locations at time t, the charge of incompletely charged mobile energy storage batteries at all locations at time t, and the number of fully discharged mobile energy storage batteries at all locations at time t.

[0110] Step S2 comprises the following steps:

[0111] Step S21: Design the state space, which is the output of the algorithm, that is, the specific dispatching action to be performed is as follows: First, AES obtains the target location to be traveled to according to the decision of the algorithm: AES obtains the target location to be traveled to according to the output decision of the algorithm. There are three types of locations in the model, namely, fishing rafts, islands with rich wind and solar energy resources, and shores; Then, when AES chooses to travel to the above locations, it needs to make further decisions based on the location type: it is divided into three situations;

[0112] In the first case, if the AES reaches the ship's AES battery resource replenishment location (i.e., the shore), the algorithm outputs the ratio of the fully charged ship power battery to the empty power battery on board, thereby achieving the purpose of charging the AES power battery;

[0113] In the second case, if the AES reaches the fishing raft, the algorithm outputs the ratio of the battery quantity exchanged with the fishing raft to the total battery quantity that can be exchanged, thus realizing the battery exchange between the AES and the fishing raft;

[0114] The third type is that when the AES reaches an island rich in wind and solar energy resources, the algorithm outputs the ratio of the amount of batteries exchanged with the island to the total amount of batteries that can be exchanged, thus realizing the battery exchange between the AES and the island;

[0115] During the entire scheduling cycle, AES can plan the navigation route through the action space given by the algorithm: when the number of empty mobile batteries stored in the fishing raft accounts for more than 60% of the total number of batteries at a certain moment, the fishing raft will have a large demand for energy transfer, and AES can go to the fishing raft to exchange the fully charged mobile batteries loaded by AES with the empty mobile batteries in the fishing raft, so that the fishing raft can avoid using its diesel generator for power replenishment as much as possible; in other time periods, AES will stay moored as much as possible to avoid unnecessary energy loss and navigation costs during navigation; when the island will be abandoned according to the energy transfer demand, AES can go to the island to exchange the empty mobile batteries loaded by AES with the fully charged mobile batteries in the island to increase the island's ability to absorb renewable energy; when AES does not have enough power for navigation every time it decides on the next target location, AES will go to the battery exchange point. During the entire scheduling cycle, AES can avoid unnecessary navigation as much as possible and only go to places with greater energy demand, thereby reducing navigation costs;

[0116] This step establishes the following state space:

[0117]

[0118]

[0119] in, represents the number of fully charged mobile energy storage batteries loaded by AES at time t; p s,t Indicates the position mark of AES at time t; si,p,t represents the information in location i when AES is at location p; Indicates the number of fully charged mobile energy storage batteries at location i; E i,t The charge of the partially charged mobile energy storage battery at the representative location i; The number of fully-charged mobile energy storage batteries at the representative location i;

[0120] Step S22: designing the action space;

[0121] Establish a mixed action space A containing discrete and continuous actions; expressed in the formula:

[0122] N={1,2,…n}

[0123] The values ​​1-n are the location numbers; there are three types of locations in the model, namely: fishing rafts {n raft}, islands rich in scenery and light resources {n island} and the shore {n shore}:

[0124] A={(p,a p )|p∈N,a p ∈χ p}

[0125]

[0126] 0≤a p ≤1

[0127] Where χ is a set of selectable continuous actions, where a s It indicates the ratio of the fully charged marine power battery to the empty power battery on board when the AES reaches the ship's AES resource replenishment location; a c,r It indicates the ratio of the number of batteries to be replaced when AES reaches the fishing raft to the total number of batteries that can be replaced; c,i It indicates the proportion of batteries that can be exchanged when AES reaches an island rich in wind and solar resources;

[0128] Step S23: designing a reward function;

[0129] In this step, the reward function is designed reasonably to enable the agent to learn the action strategy that meets the goal more quickly. The reward function contains system costs and some guiding items, specifically:

[0130] First, suppose that the penalty for failure to operate normally is that the AES does not have enough power to sail each time it decides on the next target location. Then the reward function is set as follows: the penalty for the AES failure to operate normally and the penalty for the AES remaining power being insufficient to ensure that it goes to the power exchange location, the energy transfer demand of various places including islands, shores and fish rafts, and the total cost of the system. The total cost includes the cost of wind and solar abandonment on islands and fish rafts, the operating cost of the fish raft diesel generator, the operating cost of the AES and the sum of the labor cost; the operating cost of the AES specifically refers to the power exchange cost and the unit kilometer navigation cost;

[0131] The reward function used to update the agent's policy network and value network parameters is set as follows:

[0132]

[0133] Among them, fa r,e ,fa start,e They represent the penalty for AES failure to operate normally and the penalty for insufficient remaining power of AES to ensure that it goes to the battery replacement place. If the AES navigation time constraint is not met, a penalty is incurred, otherwise it is set to 0; α, β, γ are coefficients used to balance the order of magnitude of penalty, reward and cost; r p,e The energy transfer demand of various regions; F e represents the total cost of the system during the e-th operation of AES; where r p,e Different calculation methods are used depending on the location type:

[0134]

[0135] First, calculate the total system cost as shown below:

[0136]

[0137]

[0138] C s (k) = p s ΔE s,k +ε s ΔS k

[0139] C l (k) = ε l N tras,k

[0140] Among them, C rn (k),C pg (k),C s (k) are the total wind and solar curtailment costs of the system, the operating costs of the system diesel generators, and the operating costs of the AES; is the predicted output of new energy at location i at time k; Prn,i (k) is the actual output of renewable energy at location i at time k; ε rn is the unit energy abandonment cost; P pg,i (k) is the output of the diesel generator set at location i at time k; a 0 ,a 1 ,a 2 is the diesel generator operating cost coefficient; the AES operating cost includes the battery replacement cost and the unit kilometer navigation cost, ΔE s,k is the power battery capacity of AES for battery replacement at time k; p s Indicates the battery replacement cost, in yuan / kWh; ΔS k is the AES navigation distance in the [k-1, k] period; ε s C is the unit distance cost of AES navigation; l (k) is the labor cost; ε l To transfer the cost of a single mobile energy storage battery; N tras,k is the number of movable energy storage batteries transferred in the system at time k.

[0141] In step S23, when establishing the reward function, the constraints are as follows:

[0142] Constraints 1. Power balance constraints:

[0143]

[0144] Where i is the location number, represents the output of the diesel generator cluster at location i in the microgrid at time t; It represents the actual output of renewable energy at microgrid i at time t; represents the total charging and discharging power of the mobile energy storage cluster at microgrid i at time t; represents the electrical load at microgrid i at time t;

[0145] Constraint 2: AES flight time constraint:

[0146] T i n-1 ≥T i,j j∈N

[0147] Among them, T i n represents the remaining navigable time when AES is at position i after making the n-1th decision; T i,j It represents the time required for AES to travel from location i to location j when making the nth decision. The above AES navigation time constraint formula indicates that the remaining navigation time of AES should be greater than the time required for AES to travel from location i to the next decision location each time it makes a decision.

[0148] In addition, it is necessary to ensure that the remaining power of AES is sufficient to reach the battery replacement site each time it decides on the next target location, which can be expressed as:

[0149] T i n-1 ≥T i,0 .

[0150] Step S3 is as follows: adjust the algorithm hyperparameters, introduce multi-batch forward propagation based on the PDDQN algorithm, use the neural network established by Python's Pytorch library to forward batch parallel data at the same time, and separate unrelated discrete actions from continuous actions without changing the algorithm network framework; p As input, a forward pass is performed for each action p, where e p is a standard basis vector of dimension n, then ae p for:

[0151] ae p =(0,…,0,a p ,0,…,0)

[0152] e p =(0,…,0,1,0,…,0)

[0153] Set up a batch of size n for a single forward pass through the batch-parallel data, thus:

[0154]

[0155] So we can get all the Q values: The Q value is used to update the neural network parameters and the algorithm is trained until convergence.

[0156] The AES is only used to load and transport mobile energy storage batteries (the number of batteries to be loaded depends on the specific scenario); the AES is powered by only one power battery at any given moment, and the AES is replenished with power through the battery swap station at the coast. Every time the AES reaches the shore, it swaps the unloaded power battery with the fully charged power battery at the battery swap station to achieve rapid energy supply for the AES;

[0157] The AES navigation speed of the all-electric ship is a predetermined value that matches the general ship specifications.

[0158] The island is modeled as an island rich in wind and solar energy resources, and is equipped with power generation equipment including wind turbines and photovoltaic units for outputting wind and solar energy and charging mobile energy storage batteries. The coast is equipped with a shore battery swap station, and the all-electric ship AES can exchange power batteries in an unloaded state with power batteries in a fully charged state at the shore battery swap station.

[0159] The power generation equipment equipped on the fish cage includes wind turbines, photovoltaic units, and diesel generators (due to the concentrated load on the fish cage, the fish cage's ability to utilize renewable energy is relatively limited); in order to meet the electricity demand of the fish cage, an all-electric ship AES is used to achieve energy transfer, allowing the fish cage to utilize surplus electricity in external wind and light energy resource-rich areas.

[0160] Example:

[0161] The following are specific examples of the present invention.

[0162] This example uses an island-fishing raft group in Sanduao, Ningde City, Fujian Province as the research object for example analysis. The relative positions and numbers of each location are as follows: Figure 3 As shown in the figure, the constructed island-fishing raft group includes 3 fishing rafts of different sizes, 1 resource-rich island and 1 AES departure and energy replenishment point. The location and actual distance of each point (with the AES departure and energy replenishment point as the initial location) are shown in Table 1 below:

[0163] Table 1 Coordinates and real distances of various locations

[0164]

[0165] In the coordinates (x, y) of each location in Table 1, x is the horizontal distance compared to the origin, and y is the vertical distance compared to the origin. Since each location is distributed near the coast, there are many mountains and islands nearby, which makes it impossible for AES to sail between the locations in a straight line. For AES, the actual distance between the locations is the sailing distance of AES between the locations. The installed capacity of wind and solar power generation equipment and the number of mobile energy storage configurations in each fishing raft and island are shown in Table 2 below:

[0166] Table 2 Wind and solar installed capacity and mobile energy storage configuration at various locations

[0167]

[0168] The day-ahead forecast of electricity load and renewable energy output at each location is as follows: Figure 4 and Figure 5 Based on the above forecast data, taking into account the actual situation of drastic changes in sea surface weather, the net load curves of various locations are obtained by considering the source load fluctuations as shown in Figure 6 shown.

[0169] The above algorithm was built using python 3.9 with PyTorch as the neural network framework. The hardware platform used was AMD Ryzen 74800H with Radeon Graphics 2.90GHz CPU and NVIDIA GeForce RTX3060 Laptop GPU. During the training process, the training cycle was set to 2000 times, the exploration process episilon step was set to 500 times, the initial exploration rate and the final exploration rate were set to 1 and 0.01 respectively, and the experience replay pool capacity was set to 1e6. In order to ensure the adaptability of this algorithm to uncertain quantities, different source load scenarios with uncertain fluctuations were selected for model training in each training cycle, and the reward and penalty change curves during the training process of each algorithm were obtained as shown in the following figure. Figure 7 and Figure 8 The AES navigation scheduling is implemented based on four deep reinforcement learning algorithms: DQN, PDQN, PDDQN, and MBF-PDDQN. The relevant parameter settings of each algorithm are shown in Table 3. The relevant parameter settings of the model are shown in Table 4.

[0170] Table 3 Description of hyper parameters of each algorithm

[0171]

[0172]

[0173] Table 4 Model related parameters

[0174]

[0175] Table 5 shows the cost and training time of each algorithm tested in a typical day scenario, where the AES operating cost includes AES navigation cost, AES power battery replacement cost, and mobile energy storage battery transfer cost. It can be seen from Table 5 that the method proposed in the present invention can obtain a better AES navigation strategy than other comparison algorithms, and the total cost is reduced by 8.32%, 11.71% and 18.25% respectively compared with other algorithms. At the same time, by comparing the AES operating costs obtained by each algorithm, it can be seen that since the DQN algorithm is only applicable to deep reinforcement learning in discrete action space, its flexibility is limited when realizing AES navigation scheduling, and the number of transfers and replacements of mobile energy storage batteries cannot be effectively adjusted according to real-time conditions, resulting in a large increase in the AES operating cost of the DQN algorithm compared with other algorithms, which increases by 12.72%, 15.58% and 18.75% respectively. Finally, by comparing the time required for training each algorithm, it can be seen that although the method proposed in the present invention requires more training time than the DQN algorithm and the PDQN algorithm, the method in this example is an improved algorithm based on the PDDQN algorithm. Based on the training time of the PDDQN algorithm of 558.24s, the training time of the algorithm of the present invention is reduced to 472.57s.

[0176] Table 5 Cost of each algorithm and time required for training

[0177]

[0178] Compared with other algorithms, the MBF-PDDQN algorithm proposed in this example obtains higher reward values ​​and reduces the burden of computational resources by eliminating irrelevant gradients and avoiding overestimation of Q values.

[0179] This example further visualizes the gradients of each layer in the Q network update process of the MBF-PDDQN algorithm and the PDDQN algorithm, such as Fig. 9 As shown in . Since this example performs an iterative update (Iteration) on the algorithm every certain number of steps in each training cycle, the actual number of iterations of the neural network parameters in the algorithm is much larger than the number of training cycles. Fig. 9It can be seen that the method proposed in this example has smaller norms and smaller fluctuations in the gradients of each layer of the Q network compared to the PDDQN algorithm during training. This is because in the PDDQN algorithm, many irrelevant continuous and discrete action couplings are involved in the Q network update, which leads to the generation of invalid gradients, making the gradient of the Q network in the PDDQN algorithm larger. In addition, the gradients of each layer of the Q network in this example method steadily decrease during the Q network update process, while the gradients of each layer of the Q network of the PDDQN algorithm gradually increase from 0 to 3000 iterations, and then begin to decrease, and continue to decrease at the end of the iteration, failing to converge, which may lead to the emergence of suboptimal solutions. Therefore, the MBF-PDDQN algorithm proposed in the present invention effectively avoids the invalid gradients generated by irrelevant continuous and discrete action coupling in the Q network back propagation through multi-step small batch forward propagation, thereby improving the stability and solution performance of the algorithm.

[0180] The cruising speed of AES and the onboard energy storage parameters are set according to the actual situation of the scene. AES moves at a constant speed of 25km / h during operation, and the AES onboard energy storage system consists of 4 ternary lithium battery packs, each with a capacity of 200kWh. In order to fully verify the effectiveness of the energy transfer strategy adopted by the present invention, the present invention sets the following three scenarios for analysis

[0181] Scenario 1: Each location operates independently.

[0182] Scenario 2: AES is set up in the system for transferring mobile energy storage batteries between locations, but it is not guided.

[0183] Scenario 3: AES is set up in the system to transfer mobile energy storage batteries between locations and guide them to obtain the optimal energy transfer path. However, each transfer uses the maximum number of transferable batteries.

[0184] Scenario 4: AES is set up in the system to transfer mobile energy storage batteries between locations and guide them to obtain the optimal energy transfer path. At the same time, AES can autonomously decide the amount of energy to transfer through the hybrid action space.

[0185] First, the total operating cost and energy abandonment rate of the system under each scenario are compared, as shown in Table 6. The total operating cost consists of the operating cost of each location and the operating cost of AES. The energy abandonment rate is defined as the ratio of the abandoned energy of all locations in the system to the total day-ahead predicted new energy output. Since the new energy of each fish raft is not enough to fully meet the load demand, the overall energy abandonment rate of the system is low. For the total operating cost of each scenario, each location in Scenario 1 operates independently, relying only on local power generation and energy storage equipment for power balance and new energy consumption, and cannot achieve energy interaction between locations, resulting in high operating costs; in contrast, Scenario 2 introduces AES to achieve energy interaction between locations. Although AES operates randomly in Scenario 2 and the energy abandonment rate is still a high value of 6.49%, the total operating cost is reduced by 56.45% compared with Scenario 1. Therefore, due to the random operation of AES, the energy transfer demand was partially alleviated, but the overall optimal operation of the system was not achieved; Scenario 3 guided AES to obtain the optimal cruise path, reducing the operating cost to 4,319.32 yuan and the system energy abandonment rate to 0.8%, which were reduced by 34.18% and 87.7% respectively compared with Scenario 2. However, since AES always transferred energy at the maximum energy transfer amount in Scenario 3, the energy transfer cost increased, so the operating cost of AES increased by 4.27% compared with Scenario 2; in Scenario 4, AES was able to optimize the decision on the energy transfer amount. Compared with Scenario 3, the operation of AES was more flexible, which ultimately reduced the total operating cost of Scenario 4 and the operating cost of AES by 9.68% and 10.61% respectively.

[0186] Table 6 System operation costs under different scenarios

[0187]

[0188]

[0189] Take scenario 4 as an example to verify the effectiveness of the energy transfer strategy proposed in this invention. First, the energy transfer demand at each location at each time is normalized to the minimum-maximum value, and the AES navigation path and energy transfer demand value map of each location are obtained, as well as the number of mobile batteries stored at each location and AES during the scheduling process. Fig.10 and Fig.11 shown.

[0190] Depend on Fig.10It can be seen that during the entire scheduling cycle, AES can avoid unnecessary navigation as much as possible and only go to places with greater energy demand, thereby reducing navigation costs. Taking Fish Raft 2 as an example, when the number of unloaded movable batteries stored in Fish Raft 2 accounts for more than 60% of the total number of batteries at a certain moment, and if energy is not transferred in the next period of time, the proportion of unloaded batteries will increase to occupy most of all batteries stored in Fish Raft 2, thus requiring diesel generators to supplement electricity. In this case, Fish Raft 2 will have a greater demand for energy transfer. In other time periods, AES will stay moored as much as possible to avoid unnecessary energy loss and navigation costs during navigation. Fig.11 It can be seen that during the dispatching process, when the fishing raft may be short of electricity, AES can go to the fishing raft to exchange the fully charged mobile battery loaded by AES with the unloaded mobile battery in the fishing raft, so that the fishing raft can avoid using diesel generators to supplement electricity as much as possible. Taking fishing raft 1 as an example, the net load of fishing raft 1 fluctuates within the range of -75-80kW in the 0-15h period on a typical day. During this period, fishing raft 1 can basically use internal new energy power generation combined with mobile energy storage battery packs to achieve internal energy balance, and the status of fully charged, unloaded, and partially charged mobile energy storage batteries in the mobile energy storage battery pack does not change much. However, the net load gradually increases to between 50-200kW during the 16h-24h period. The new energy power generation of fishing raft 1 is not enough to meet the power demand, and mobile energy storage battery packs are needed to provide electricity. During the 16-20h period, the number of fully charged batteries in the mobile energy storage battery pack gradually decreases, and the number of unloaded batteries gradually increases, indicating that fishing raft 1 supplies energy to the system through mobile energy storage equipment during this period. At 21h, fishing raft 1 has a large demand for energy transfer. At this time, the mobile energy storage battery packs in fishing rafts 3 and 2 are in a power shortage state. Since fishing rafts 1 and 2 are geographically close, AES first chooses to go to fishing raft 3 at 20h and replenish its energy, then go to fishing raft 2 at 21h, and go to fishing raft 1 at 22h, and finally return to the shore to prepare for the next day's energy transportation; when the island is about to abandon energy, AES can go to the island to exchange the unloaded mobile batteries loaded by AES with the fully charged mobile batteries in the island to increase the island's ability to absorb renewable energy.

[0191] In order to ensure the robustness of the proposed method to uncertain disturbances, Figure 6 500 source-load scenarios are randomly selected from the net load curves of various locations considering source-load fluctuations for model testing. Ten of them are used as benchmark scenarios to obtain rewards and penalties during the test. Fig.12 and Fig.13 shown.

[0192] Depend on Fig.12It can be seen that the average cost of testing 500 randomly selected scenarios using the trained model is 4292.64 yuan, which is only 10.01% higher than the minimum cost converged to for a typical day; secondly, Fig.13 It can be seen that for the 500 randomly selected scenarios, the average penalty of the method proposed in the present invention during the test process is only -0.0051, which shows that the method proposed in the present invention can well meet the system operation constraints in the wind and solar load uncertainty scenario, and only 1-2 times of over-limit will occur in very few severe scenarios; the above test process takes 19.01s, and the average agent takes 0.00158s for each decision, which shows that the method proposed in the present invention can be used for real-time decision-making of AES navigation paths, and AES can make appropriate decisions in a very short time according to the information in the system at the current moment. Therefore, the method proposed in this example can train the model according to the wind and solar load scenarios predicted at various locations with a large number of additional uncertainty disturbances in the day-ahead stage to ensure the generalization ability of the model. In addition, due to the real-time decision-making characteristics of deep reinforcement learning, the trained model can be used in the intra-day stage to make millisecond-level real-time decisions according to the real-time wind and solar load conditions observed by AES. When the climate changes and the system wind and solar load output is different from the day-ahead forecast, AES can accurately capture this change and adjust the scheduling decision, so as to better cope with the challenge of source load uncertainty.

[0193] In order to verify the necessity of the proposed method in this example for solving the AES range optimization problem in large-scale sea scenarios with obvious uncertainties, the model of the present invention is further modeled as a mixed integer programming problem and solved using the commercial solver GUROBI. The solution termination condition is MinGap = 0.1%, and the solution termination time is 24h. The simulation scenario is gradually expanded from the basic 3-node scenario to the 5-node scenario in the model of the present invention. For the mixed integer programming model solution information such as Fig.14 As shown: Fig.14 In the figure, the bar graph corresponds to the operating costs obtained by solving the two methods in each scenario, and the line graph represents the time required for the solution. Since the solution time limit was set to 24h when the traditional method was used for the solution in this example, the traditional method obtained the optimal solution within the time limit in all test scenarios. Compared with the method proposed in this invention, the cost was reduced by 1.99%, 5.14% and 6.17%, respectively. However, the solution time increased by 311.96%, 683.61% and 1764.73%, respectively, and as the model scale expanded, the solution time of the traditional method increased by 114.32% and 460.18%, respectively, compared with the 3-node model. At the same time, when the model scale is further expanded, the increase in the solution time of the traditional method will be even greater. Fig.14It can be seen that the time required for solving the 5-node model by the traditional method increases by 161.37% compared to the time required for solving the 4-node model. This shows that as the scale of the model increases, the solution time of the traditional method will increase significantly. It is worth noting that the solution time of the method proposed in this example remains at a low level in the above three scenarios. As the scale of the model increases, the solution time only increases by 12.67% and 9.84%. In order to further verify the adaptability of the method proposed in the present invention to the expansion of model scale and the sensitivity of the traditional method to the expansion of model scale, the present invention uses a 5-node scenario with a time accuracy of 15 minutes for testing. When the time accuracy is changed from 1 hour to 15 minutes, the number of variables and constraints in the model can be approximately considered to have increased three times. Fig.14 It can be seen that when the solution time reaches the set upper limit, the traditional method cannot obtain the optimal solution, while the method proposed in this example can still reach an approximate optimal solution in a relatively short time. In summary, although the method of the present invention fails to strictly reach the optimal solution in some cases, the difference between the obtained suboptimal solution and the optimal solution is small, and the solution time can be significantly shortened. Therefore, in practical applications, when faced with time and resource constraints, the method of this example can provide an effective approximate optimal solution, and compared with traditional methods, the method of the present invention can be applied to the solution of larger-scale energy transportation problems, which has important practical significance.

[0194] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A method for energy transportation scheduling of energy storage ships based on deep reinforcement learning, characterized in that: The scheduling method is based on an island-fishing raft-coast energy transportation model including AES. Energy scheduling is performed by transporting energy storage batteries from battery swap stations on the coast or islands to fishing rafts using an all-electric ship AES. The method is solved using a deep reinforcement learning method of a parameterized dual-depth Q network based on multi-batch forward propagation to obtain the energy transportation strategy required by the energy transportation model.

2. According to claim 1, a method for energy storage ship energy transportation scheduling based on deep reinforcement learning is characterized in that: The energy storage battery is a mobile energy storage battery, i.e., a power battery. The method solves the island-offshore fishing raft-coastal energy transportation model containing an all-electric ship AES by using a deep reinforcement learning method of a parameterized double-depth Q network based on multi-batch forward propagation; the steps included are as follows: Step S1: Establishing an island-fishing raft energy transportation model including an all-electric energy storage vessel AES; Step S2: Model the energy transportation problem as a Markov decision process, and clearly define the state space, action space, and reward function in the Markov decision process; Step S3: Adjust the algorithm hyperparameters, train the agent using a parameterized dual-depth Q network algorithm based on multi-batch forward propagation, and train the algorithm until convergence.

3. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 2 is characterized in that: Step S1 includes the following steps: Step S11: Design the AES model and establish the relationship between the actual propulsion power, speed and displacement of AES as follows: s =A×V 3 ×(M0+MN s ) 2 / 3 Among them, A is the inherent parameter of the ship; V is the navigation speed of AES, which is a certain value; M0 is the displacement of AES when it is unloaded, that is, the mass of water displaced when the ship is unloaded; M is the displacement caused by AES loading cargo; N s Load AES with mobile energy storage batteries quantity AES; Among them, E s,b The maximum capacity of a single marine power battery; is the floor rounding symbol; Indicates the number of fully charged power batteries in AES at time t; Indicates the number of unloaded power batteries in AES at time t; Indicates the charge of the marine power battery in the AES under non-full charge and no-load conditions at time t; Step S12: designing a mobile energy storage device model; The mathematical model of mobile energy storage battery is as follows: 0≤E i,t ≤E i,max in, are the charging and discharging power of a single mobile energy storage battery i; E i,max P is the maximum charge capacity of a single mobile energy storage battery; i,max is the maximum charge and discharge power of a single mobile energy storage battery; k i,t is a 0-1 variable that marks the charge and discharge status of the energy storage battery. i,t When the value is 1, it means that the energy storage battery is in charging state. Both are set to 1.

4. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 2 is characterized in that: Step S2 includes the design of the scheduling algorithm, specifically: for the application of the established algorithm to the energy transportation problem, three quantities need to be designed first: state space, action space, and reward function; the execution process of the algorithm is: the state space is used as the input of the algorithm, and the algorithm parameters are updated in a gradient optimization manner based on the feedback signal provided by the reward function, and the action space is used as the output scheduling strategy of the algorithm; the specific definition is: determine the state space according to the scenario, which is used as the input of the algorithm: number the islands, fishing rafts, and coasts at the scheduling location as numbers such as 0, 1, and 2 to represent the location mark quantity; the state space is designed as: scheduling time t, the number of fully charged mobile energy storage batteries loaded by AES at time t, the position mark quantity of AES at time t, the charge of the marine power battery in the AES under non-full charge and no-load conditions at time t, the number of fully charged mobile energy storage batteries at all locations at time t, the charge of incompletely charged mobile energy storage batteries at all locations at time t, and the number of fully discharged mobile energy storage batteries at all locations at time t.

5. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 4 is characterized in that: Step S2 comprises the following steps: Step S21: Design the state space, which is the output of the algorithm, i.e. the scheduling action to be executed is: First, AES obtains the target location according to the decision of the algorithm: AES obtains the target location according to the output of the algorithm. There are three types of locations in the model: fishing rafts, islands with rich wind and solar energy resources, and shores; then, when AES chooses to go to the above locations, it needs to make further decisions based on the location type: there are three situations; In the first case, if the AES reaches the ship's AES battery resource replenishment location, the algorithm outputs the ratio of the fully charged ship power battery to the empty power battery on board, thereby achieving the purpose of charging the AES power battery; In the second case, if the AES reaches the fishing raft, the algorithm outputs the ratio of the battery quantity exchanged with the fishing raft to the total battery quantity that can be exchanged, thus realizing the battery exchange between the AES and the fishing raft; The third type is that when the AES reaches an island rich in wind and solar energy resources, the algorithm outputs the ratio of the amount of batteries exchanged with the island to the total amount of batteries that can be exchanged, thus realizing the battery exchange between the AES and the island; During the entire scheduling cycle, AES can plan the navigation route through the action space given by the algorithm: when the number of empty mobile batteries stored in the fishing raft accounts for more than 60% of the total number of batteries at a certain moment, the fishing raft will have a large demand for energy transfer, and AES can go to the fishing raft to exchange the fully charged mobile batteries loaded by AES with the empty mobile batteries in the fishing raft, so that the fishing raft can avoid using its diesel generator for power replenishment as much as possible; in other time periods, AES will stay at anchor as much as possible to avoid unnecessary energy loss and navigation costs during navigation; when the island is about to abandon energy according to the energy transfer demand, AES can go to the island to exchange the empty mobile batteries loaded by AES with the fully charged mobile batteries in the island, so as to increase the island's ability to absorb renewable energy; When AES does not have enough power to sail each time it decides on the next destination, it will go to the power exchange station. During the entire scheduling cycle, AES can avoid unnecessary sailing as much as possible and only go to places with greater energy demand, thereby reducing sailing costs; This step establishes the following state space: in, represents the number of fully charged mobile energy storage batteries loaded by AES at time t; p s,t Indicates the position mark of AES at time t; s i,p,t represents the information in location i when AES is at location p; Indicates the number of fully charged mobile energy storage batteries at location i; E i,t The charge of the partially charged mobile energy storage battery at the representative location i; The number of fully-charged mobile energy storage batteries at the representative location i; Step S22: designing the action space; Establish a mixed action space A containing discrete and continuous actions; expressed in the formula: N={1,2,…n} The values ​​1-n are the location numbers; there are three types of locations in the model, namely: fishing rafts {n raft }, islands rich in scenery and light resources {n island } and the shore {n shore }: A={(p,a p )|p∈N,a p ∈χ p } 0≤a p ≤1 Where χ is a set of selectable continuous actions, where a s It indicates the ratio of the fully charged marine power battery to the empty power battery on board when the AES reaches the ship's AES resource replenishment location; a c,r It indicates the ratio of the number of batteries to be replaced when AES reaches the fishing raft to the total number of batteries that can be replaced; c,i It indicates the proportion of batteries that can be exchanged when AES reaches an island rich in wind and solar resources; Step S23: designing a reward function; In this step, the reward function is designed reasonably to enable the agent to learn the action strategy that meets the goal more quickly. The reward function contains system costs and some guiding items, specifically: First, suppose that the penalty for failure to operate normally is that the AES does not have enough power to sail each time it decides on the next target location. Then the reward function is set as follows: the penalty for the AES failure to operate normally and the penalty for the AES remaining power being insufficient to ensure that it goes to the power exchange location, the energy transfer demand of various places including islands, shores and fish rafts, and the total cost of the system. The total cost includes the cost of wind and solar abandonment on islands and fish rafts, the operating cost of the fish raft diesel generator, the operating cost of the AES and the sum of the labor cost; the operating cost of the AES specifically refers to the power exchange cost and the unit kilometer navigation cost; The reward function used to update the agent's policy network and value network parameters is set as follows: Among them, fa r,e ,fa start,e They represent the penalty for AES failure to operate normally and the penalty for insufficient remaining power of AES to ensure that it goes to the battery replacement place. If the AES navigation time constraint is not met, a penalty is incurred, otherwise it is set to 0; α, β, γ are coefficients used to balance the order of magnitude of penalty, reward and cost; r p,e The energy transfer demand of various regions; F e represents the total cost of the system during the e-th operation of AES; where r p,e Different calculation methods are used depending on the location type: First, calculate the total system cost as shown below: C s (k)=p s DE s,k +e s ΔS k C l (k)=e l N tras,k Among them, C rn (k),C pg (k),C s (k) are the total wind and solar curtailment costs of the system, the operating costs of the system diesel generators, and the operating costs of the AES; is the predicted output of new energy at location i at time k; P rn,i (k) is the actual output of renewable energy at location i at time k; ε rn is the unit energy abandonment cost; P pg,i (k) is the output of the diesel generator set at time k and location i; a0, a1, a2 are the diesel generator operating cost coefficients; the AES operating cost includes the battery replacement cost and the unit kilometer navigation cost, ΔE s,k is the power battery capacity of AES for battery replacement at time k; p s Indicates the battery replacement cost, in yuan / kWh; ΔS k is the AES navigation distance in the [k-1, k] period; ε s C is the unit distance cost of AES navigation; l (k) is the labor cost; ε l To transfer the cost of a single mobile energy storage battery; N tras,k is the number of movable energy storage batteries transferred in the system at time k.

6. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 5 is characterized in that: In step S23, when establishing the reward function, the constraints are as follows: Constraints 1. Power balance constraints: Where i is the location number, represents the output of the diesel generator cluster at location i in the microgrid at time t; It represents the actual output of renewable energy at microgrid i at time t; represents the total charging and discharging power of the mobile energy storage cluster at microgrid i at time t; represents the electrical load at microgrid i at time t; Constraint 2: AES flight time constraint: in, represents the remaining navigable time when AES is at position i after making the n-1th decision; T i,j It represents the time required for AES to travel from location i to location j when making the nth decision. The above AES navigation time constraint formula indicates that the remaining navigation time of AES should be greater than the time required for AES to travel from location i to the next decision location each time it makes a decision. In addition, it is necessary to ensure that the remaining power of AES is sufficient to reach the battery replacement site each time it decides on the next target location, which can be expressed as:

7. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 2 is characterized in that: Step S3 is as follows: adjust the algorithm hyperparameters, introduce multi-batch forward propagation based on the PDDQN algorithm, use the neural network established by Python's Pytorch library to forward batch parallel data at the same time, and separate unrelated discrete actions from continuous actions without changing the algorithm network framework; p As input, a forward pass is performed for each action p, where e p is a standard basis vector of dimension n, then ae p for: ae p =(0,…,0,a p ,0,…,0) e p ＝(0,…,0,1,0,…,0) Set up a batch of size n for a single forward pass through the batch-parallel data, thus: So we can get all the Q values: The Q value is used to update the neural network parameters and the algorithm is trained until convergence.

8. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The AES is only used to load and transport mobile energy storage batteries; the AES is powered by only one power battery at any time, and the AES is replenished through the battery swap station at the coast. Every time the AES reaches the shore, the unloaded power battery is exchanged with the fully charged power battery at the battery swap station to achieve rapid energy supply for the AES; The AES navigation speed of the all-electric ship is a predetermined value that matches the general ship specifications.

9. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The island is modeled as an island rich in wind and solar energy resources, and the power generation equipment equipped includes wind turbines and photovoltaic units, which are used to generate wind and solar energy and to charge mobile energy storage batteries; The coast is equipped with a shore battery swap station, and the all-electric ship AES can exchange the power batteries in the no-load state with the power batteries in the fully charged state at the shore battery swap station.

10. The energy storage ship energy transportation scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The power generation equipment equipped on the fish cage includes wind turbines, photovoltaic units, and diesel generators. In order to meet the electricity demand of the fish cage, an all-electric ship AES is used to achieve energy transfer, so that the fish cage can utilize surplus electricity in areas rich in external wind and light energy resources.

Images