Joint optimization method for all-electric ship power generation and navigation scheduling based on deep reinforcement learning

Abstract

The application discloses a kind of full electric ship power generation and navigation scheduling joint optimization method based on deep reinforcement learning, including steps: establishing the full electric ship speed model considering navigation environment, generator and battery operation model, and battery capacity attenuation model;Establish the joint optimization model of ship power system power generation and navigation scheduling with the optimization goal of minimizing generator operation cost and battery attenuation capacity;The joint optimization model is modeled by Markov decision process, the improved double-layer MSD3QN algorithm is trained using historical data set, and the model parameters are saved;Load the trained agent model, input current state variable to obtain the corresponding generator and battery power, speed of action sequence.The application can realize end-to-end control from measurement data to generator, battery and speed action, with good dynamic performance, globality and robustness.Meanwhile, by improving and optimizing the DQN agent architecture, the training efficiency and convergence of the agent are improved.

Description

Technical Field

[0001] This invention belongs to the field of energy management technology for all-electric ship power systems, specifically relating to a joint optimization method for power generation and navigation scheduling of all-electric ships based on deep reinforcement learning. Background Technology

[0002] In recent years, to alleviate the crises of resource scarcity and climate change, electrification has become an inevitable trend in the shipping industry. Against this backdrop, all-electric ships, characterized by integrated electric power systems, use electricity to power all propulsion loads, navigation, communication, and other auxiliary equipment, achieving comprehensive utilization of the ship's energy and aligning with the trend of improving energy conversion efficiency and reducing pollution in ship electrical systems.

[0003] Unlike traditional ships that use fixed mechanical transmissions, electric propulsion technology makes all-electric ships more flexible and controllable. However, during navigation, they must not only comply with the ship's navigation constraints but also meet the operational constraints of the microgrid, greatly increasing the complexity of all-electric ship operation. On the one hand, as a "mobile independent microgrid," the all-electric ship has weak disturbance resistance, but uncertainties such as wave height and wind speed in its navigation environment, as well as the randomness of crew electricity consumption and ship mission requirements, all pose challenges to the stable operation of the ship's power system. On the other hand, although the integration of energy storage systems greatly improves the flexibility of the ship's power system operation, frequent charging and discharging during ship navigation will cause the battery life to decline rapidly, thus affecting the response performance of the energy storage system.

[0004] Existing methods for solving the joint optimization problem of power generation and navigation scheduling of all-electric ships with uncertainties mainly consist of a prediction process and a decision process, which are separated from each other. The prediction process uses statistical methods or artificial intelligence techniques to predict and estimate uncertainties, while the decision process uses real-time optimization algorithms and uncertainty optimization algorithms to schedule and control the power generation units and demand-side loads in the established single-objective or multi-objective model. Although these methods have good optimization performance and robustness, the description of uncertainties in the prediction process has a significant impact on the performance of subsequent algorithms, and they are difficult to handle scenarios where uncertainties are difficult to predict.

[0005] With the development of sequential decision-making technologies such as reinforcement learning and deep reinforcement learning, and the intelligentization of equipment measurement devices, data-driven methods have significant advantages in solving the joint optimization problem of power generation and navigation scheduling of all-electric ships with uncertainties. However, due to the large number of devices in the power system of all-electric ships and the complexity of operational constraints, the corresponding agent model contains high-dimensional state variables and high-dimensional action variables. If existing deep reinforcement learning algorithms are directly used for training, the training efficiency of the agent is low and the convergence is poor, which cannot meet the optimization requirements of joint scheduling of power generation and navigation of all-electric ships.

[0006] Therefore, given the uncertainties in the navigation environment and electricity consumption behavior of all-electric ships, a suitable joint optimization strategy for power generation and navigation scheduling of all-electric ships is necessary in order to better control the safe, stable and efficient operation of the ship's power system. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the aforementioned background technology by providing a joint optimization method for power generation and navigation scheduling of all-electric ships based on deep reinforcement learning. This method is directly data-driven and improves and optimizes the value function update method and action space of the classic DQN agent. It considers and realizes the uncertainty of the all-electric ship's navigation environment and load demand, as well as the flexibility of generator, battery, and speed adjustment, greatly improving the training efficiency and convergence performance of the agent. This method enables dynamic optimal control of power generation and navigation of all-electric ships while meeting system operation constraints.

[0008] The technical solution adopted in this invention is: a joint optimization method for power generation and navigation scheduling of all-electric ships based on deep reinforcement learning, comprising the following steps:

[0009] S1. Considering the uncertainties of the navigation environment, establish the speed model, generator output model, battery SOC model, and battery capacity decay model of the all-electric ship.

[0010] S2. Based on the constraints of the speed model, generator output model, battery SOC model and battery capacity decay model, a joint optimization model for ship power system power generation and navigation scheduling with uncertainty is established with the optimization objective of minimizing generator operating cost and battery capacity decay.

[0011] S3 will model the Markov decision process using the joint optimization model and train the parameters of the two-layer multi-step competitive dual-depth Q agent model using the historical dataset of the state variables of the all-electric ship.

[0012] S4. Deploy the trained two-layer multi-step competitive dual-depth Q agent model, input the state variables of the current moment in the scheduling cycle to obtain the agent's actions at the current moment, including the adjustment of the generator and battery output power of the ship's power system and the speed of the all-electric ship; the state variables of the two-layer multi-step competitive dual-depth Q network agent are then transferred to the next moment according to the response actions of the two-layer multi-step competitive dual-depth Q network agent.

[0013] S5: Determine if the current time is the maximum time of the scheduling cycle. If the condition is met, it means that the current scheduling cycle has ended. Output the action sequence of the two-layer multi-step competitive dual-depth Q network agent. Otherwise, return to S4 to continue execution.

[0014] In the above technical solution, step S3 includes the following steps:

[0015] S31, Define the state space, action space and reward function of the two-layer multi-step competitive dual-depth Q network agent based on the state variables, optimization variables and optimization objectives obtained from the joint optimization model in step S2;

[0016] S32, the two-layer multi-step competitive dual-depth Q-network agent includes an upper-layer multi-step competitive dual-depth Q-network agent and a lower-layer solver; the actions contained in the agent's action space are divided into two categories according to the response: fluctuating actions and steady-state actions. Fluctuating actions are responded to by the upper-layer multi-step competitive dual-depth Q-network agent; steady-state actions are responded to by the lower-layer solver.

[0017] S33: Obtain state variables from the historical dataset of state variables of the all-electric ship, train the model of the two-layer multi-step competitive dual-depth Q network agent until the maximum number of iterations is reached, and fix the model parameters of the two-layer multi-step competitive dual-depth Q network agent.

[0018] In the above technical solution, in step S2, the generator operating cost includes fuel cost and start-up cost, and the battery capacity degradation is represented by the equivalent battery capacity degradation cost; the system operation constraints included in the joint optimization model mainly include: overall system power balance constraints and spinning reserve constraints; generator operation constraints, including power output constraints, ramping constraints, and minimum start-stop time constraints; battery operation constraints, including power output constraints, SOC constraints, and temperature constraints; and all-electric ship navigation constraints, including speed and range constraints.

[0019] In the above technical solution, the uncertainties included in the joint optimization model in step S2 include the uncertainties brought about by the navigation environment to the propulsion load of the all-electric ship and the uncertainties brought about by the power consumption behavior of the ship's users to the daily service load.

[0020] In the above technical solution, in step S31, the state space S of the two-layer multi-step competitive dual-depth Q-network agent is as follows:

[0021]

[0022] Where s(t)∈S; It is the daily load power with uncertainties. The additional drag coefficient for navigation considering environmental factors, SOC(t) is the battery's state of charge, and T bat (t) is the battery operating temperature, ΔQ loss (t) represents the battery degradation capacity; v min (t) and v max (t) represents the upper and lower limits of the speed, Dis n Dis(t) represents the planned voyage, Dis(t) represents the actual voyage, and t represents the time of voyage.

[0023] The action space A of a two-layer, multi-step competitive, dual-depth Q-network agent is as follows:

[0024]

[0025] Among them, a t ∈A;ΔP i,DG (t), P bat v(t) and v(t) represent the discrete actions corresponding to the change in power of the i-th generator, battery power, and speed at time t in a two-layer, multi-step competitive dual-depth Q-network agent, respectively. Let i be the maximum ramp power of the i-th generator. This is the minimum output power of the battery. v is the battery's maximum output power. min The minimum speed of an all-electric ship, v max N is the maximum speed of the all-electric ship. i,DG N bat N v These represent the discrete number of actions for the i-th generator power, battery power, and speed, respectively.

[0026] The dimensions of the action space are as follows:

[0027]

[0028] The reward function r of a two-layer, multi-step competitive, dual-depth Q-network agent is as follows:

[0029] r t = -β·[FC(t)+SC(t)+ρΔQ loss (t)+penalty_all(t)]

[0030] Where FC(t) is the fuel cost, SC(t) is the startup cost, and ρΔQ loss (t) represents the cost of equivalent battery capacity decay, β is the scaling factor of the reward function, and penalty_all(t) is the penalty value for violating the running constraints in step S1.

[0031] In the above technical solution, in step S32, the fluctuating action is the change in battery output and speed, which is used to respond to the uncertainty brought about by changes in the navigation environment and the randomness of user behavior; the steady-state action is the output of the generator, which is used to bear the steady-state power of the ship's power system.

[0032] In the above technical solution, in step S32, the upper-layer multi-step competitive dual-deep Q-network intelligent agent A... upper The design is as follows:

[0033]

[0034] Lower-level optimization solver A lower，t The optimization action is solved by establishing a unit combination model, and its design is as follows:

[0035] A lower,t =A s ={P i,DG (t),δ i,DG (t)}, i=1,2,...,N

[0036] Where, δ i,DG (t) represents the operating state of the i-th generator at time t, P i,DG (t) represents the power of the i-th generator at time t; A f A represents the action space of wave-like actions; s The action space that represents steady-state or stable actions.

[0037] In the above technical solution, the training process of the two-layer multi-step competitive dual-depth Q-network agent in S33 is as follows:

[0038] 1) Read historical data information from the database; the historical data information includes: daily load history data of the all-electric ship. Battery initial operating temperature data T bat,0 Using navigation log data, the impact of the navigation environment on ship propulsion resistance is equivalently converted into an environmentally added drag coefficient. The historical data information read is used to construct a training set for the intelligent agent;

[0039] 2) Based on the physical parameters of each device in the all-electric ship's power system, determine the action dimension dimA of the upper-level intelligent agent. f =N bat ×N v ;

[0040] 3) Set up two neural networks with the same architecture: an online Q-network and a target Q-network; initialize the agent's neural network hyperparameters and important reinforcement learning parameters, including the number of input and output layers, the number of hidden layers and the number of hidden layer neurons, the number of neurons in the competing network layer, the learning rate α, the discount factor γ, the number of bootstrapping steps n_step, the maximum number of iterations M, the time interval between single iterations T, the parameter update interval k of the online Q-network, the parameter synchronization interval C of the target Q-network, the experience pool size D, and the batch size.

[0041] 4) Set the current iteration number to l. If l ≤ M, then randomly select the environment state variable s at a certain time t. t Start training the current generation; otherwise, skip to step 11.

[0042] 5) The upper-level intelligent agent first interacts with the environment regarding the state variable s.t Interaction, obtaining volatility action a based on an ε-greedy greedy strategy. upper,t and corresponding reward r upper,t Then, the upper-level intelligent agent will perform the wave-like action a. upper,t State variable s t and the start-up time of each generator at the current moment Closing Time Information is passed to the lower-level solver; the steady-state action 'a' is obtained by minimizing the generator operating cost through the lower-level solver. lower,t and corresponding reward r lower,t The results are then uploaded to the upper-level intelligent agent.

[0043] 6)a upper,t and a lower,t The response actions a of the two-layer, multi-step competitive, dual-depth Q-network intelligent agent together constitute a t This enables a two-layer, multi-step competitive, dual-depth Q-network agent to change from its current state s. t Transition to the next state s t+1 r upper,t and r lower,t The reward function r of the two-layer, multi-step competitive dual-depth Q-network agent together constitutes the reward function r. t After obtaining the rewards for n_step time intervals after the current time t, the data (s) t ,a t G t ,s t+1 The flag is stored in the experience pool for use in updating the agent's parameters, where G... t The n-step reward value after the agent adopts multi-step bootstrapping technology: where flag represents the program flag bit. Setting it to 0 will allow the program to run normally in iteration, while setting it to 1 will terminate the current iteration and start a new loop iteration.

[0044] 7) If the conditions for updating the online Q-network parameters are met, randomly select batch_size empirical data from the experience pool and iterate the network parameters of the upper-layer agent; otherwise, proceed to step 8):

[0045] 8) If the target Q-network parameter synchronization condition is met, then the neural network parameters θ of the online Q-network will be... w The neural network parameters θ are synchronously copied to the target Q-network. w- (Otherwise, skip to step 9).

[0046] 9) If flag == 1 or t = T, the two-layer multi-step competitive dual-depth Q network agent ends this iteration, then execute l = l + 1 and jump to step 4); otherwise, jump to step 10. T represents the number of iterations required for each iteration.

[0047] 10) Set st =s t+1 If t = t + 1, jump to step 5) to continue this training;

[0048] 11) The training of the two-layer, multi-step competitive, dual-depth Q-network agent is completed, and the neural network model parameters θ are saved. w .

[0049] In the above technical solution, step S4, the process of deploying the trained two-layer multi-step competitive dual-depth Q-agent model, includes:

[0050] 1) Load the neural network model parameters θ w ;

[0051] 2) Input the state variable s at the current time t. t The intelligent agent will obtain the all-electric ship generator output P based on the principle of maximizing value. i,DG (t), Battery output P bat (t) and speed v(t);

[0052] 3) If t = T, jump to step 4); otherwise, set s. t =s t+1 And t = t + 1, jump to step 2);

[0053] 4) At the end of the scheduling period, output the agent action sequence {P} for this scheduling period. i,DG (t),P bat (t),v(t),t∈T}.

[0054] The beneficial effects of this invention are:

[0055] 1) Compared with traditional methods for solving uncertainties, this invention is directly driven by data, avoiding the impact of prediction errors of uncertain variables on the optimization algorithm, and can realize end-to-end control from measurement data to the generator, battery and speed of the ship's power system; at the same time, compared with traditional methods, this invention also has better dynamic real-time performance, global optimization performance and robust performance.

[0056] 2) Compared with the classic DQN method, the MSD3QN algorithm used in the dual-layer deep reinforcement learning algorithm of this invention integrates competitive network architecture, dual Q learning and multi-step bootstrapping technology on the basis of the classic DQN algorithm, which greatly improves the training efficiency and convergence performance of the agent.

[0057] 3) Compared with existing deep reinforcement learning methods, this invention divides the agent's action space into fluctuating actions and steady-state actions according to the response purpose. These actions are responded to by the upper-layer MSD3QN agent and the lower-layer optimization solver, respectively, which reduces the complexity of the agent's action space and improves the training efficiency of the agent. Attached Figure Description

[0058] Figure 1 This is a flowchart of the joint optimization method for power generation and navigation scheduling of all-electric ships based on deep reinforcement learning, as described in this invention.

[0059] Figure 2 Schematic diagram of the power system architecture of an all-electric ship

[0060] Figure 3 A schematic diagram of the planned speed and range of the all-electric ship.

[0061] Figure 4 The training flowchart of the improved two-layer MSD3QN algorithm of this invention is shown below.

[0062] Figure 5 This is a flowchart illustrating the joint optimization of power generation and navigation scheduling for an all-electric ship using the improved two-layer MSD3QN algorithm. Detailed Implementation

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but these descriptions do not constitute a limitation on the present invention.

[0064] To alleviate the crises of resource scarcity and climate change, electrification has become an inevitable trend in the shipping industry. The joint optimization strategy for power generation and navigation scheduling of ship power systems is crucial for the flexibility, stability, and economy of all-electric ship operation. Given the uncertainties in the navigation environment and electricity consumption behavior of all-electric ships, this embodiment proposes a joint optimization method for power generation and navigation scheduling of all-electric ships based on deep reinforcement learning, using an improved double-layer MSD3QN algorithm (multi-step dueling double deep Q network). Figure 1 As shown, the main steps are as follows: 1) Establish a speed model of the all-electric ship, an operation model of the ship's power system consisting of a diesel generator and a lithium battery, and a battery capacity decay model, considering the uncertainty of the navigation environment; 2) Establish a joint optimization model of the ship's power system power generation and navigation scheduling with uncertainty as the optimization objective of minimizing generator operating costs and battery capacity decay; 3) Model the joint optimization model using a Markov decision process, use the historical dataset of state variables to train the network model parameters of the improved two-layer deep reinforcement learning algorithm, and save the trained neural network model parameters; 4) Load the trained two-layer MSD3QN agent model, input the state variables at the current moment in the scheduling cycle to obtain the action sequence of the generator, battery output power and all-electric ship speed of the ship's power system at the current moment.

[0065] Specifically, this embodiment presents a joint optimization method for power generation and navigation scheduling of an all-electric ship based on deep reinforcement learning. It includes the following steps:

[0066] S1, as Figure 2 As shown, the all-electric ship's power system consists of a power generation system, an energy storage system, a power transmission and distribution system, a propulsion system, and an energy management system. The generator sets and energy storage system supply power to the service load and propulsion load through the power transmission and distribution system. The energy management system obtains the operating status of each subsystem and sends scheduling commands through an information network to achieve stable system operation. This embodiment establishes a speed model for the all-electric ship, an operating model for the ship's power system composed of diesel generators and lithium batteries, and a battery capacity decay model, considering the uncertainties of the navigation environment, based on the equipment included in the all-electric ship's power system.

[0067] Speed ​​model:

[0068] Generator output model:

[0069] Battery SOC model: SOC(t) = Q bat (t) / Q bat

[0070]

[0071]

[0072] Battery capacity degradation model: ΔQ loss (t)=ΔSOH(t)·100%

[0073]

[0074] Ah total (c,T bat )=[20 / (B(c)·exp(-E a (c) / RT bat ))] 1 / z

[0075] Where P prop (t) represents the propulsion power, and c s,1 and c s,2 P represents the hull resistance coefficient and constants related to the hull type. i,DG (t) represents the generator output power, δ i,DG (t) Generator operating status, η i,DG (t) Output power load factor, Generator maximum power; SOC(t) is the battery state of charge, Q bat (t) represents the battery capacity, Q batFor the rated capacity of the battery, I bat (t) represents the battery current, P bat (t) represents the battery power, with a positive value indicating discharge and a negative value indicating charging; ΔQ loss (t) represents the battery's degraded capacity, ΔSOH(t) represents the change in the battery's state of health, Ah total (c,T bat () represents the battery ampere-hour capacity.

[0076] S2. Establish a joint optimization model for ship power system power generation and navigation scheduling with uncertainties, aiming to minimize generator operating costs and battery degradation capacity. Generator operating costs include fuel cost FC(t) and start-up cost SC(t), while battery degradation capacity is represented by the equivalent battery capacity degradation cost ρΔQ. loss The system operation constraints included in the joint optimization model mainly include: overall system power balance constraints and spinning reserve constraints, generator operation constraints, battery operation constraints, and all-electric ship navigation constraints.

[0077] Fuel cost model:

[0078] Startup cost model:

[0079]

[0080] Battery capacity degradation equivalent cost: ΔCost loss =ρΔQ loss (t)

[0081] Power balance constraints:

[0082] Rotational spare constraint:

[0083] Generator operating constraints: (Upper and Lower Limits)

[0084] (Climbing constraint)

[0085] (Minimum running time constraint)

[0086] Battery operating constraints: (Upper and Lower Limits)

[0087] (SOC operational constraints)

[0088] All-electric ship navigation constraints: v min (t)≤v(t)≤v max (t)(Speed ​​upper and lower limits constraints)

[0089] Dis n (1-δ v )≤Dis(t)≤(1+δ v )Dis n (Flight range constraints)

[0090] Where a i b i c i For fuel efficiency, this embodiment... Figure 2 The fuel coefficients and start-up costs of each piece of equipment are shown in Table 1; ρ is the equivalent attenuation cost conversion coefficient. For daily usage loads with uncertainties, as shown in Table 2 in this embodiment; R DG (t) and R bat (t) represents the rotating reserve power of the generator and battery, μ R This is the rotational reserve factor; For the slope rate, The generator start-up time, The generator shut-off time The minimum running time is shown in Table 3 in this embodiment; v(t) is the speed of the all-electric ship, Dis(t) is the actual distance traveled, and Dis... n (t) represents the planned voyage distance, such as Figure 3 As shown; δ v This is due to flight path deviation.

[0091] Table 1

[0092]

[0093] Table 2

[0094]

[0095] Table 3

[0096]

[0097] S3, modeling the joint optimization model established in S2 using a Markov decision process, trains the network model parameters of the improved two-layer MSD3QN algorithm using a historical dataset of state variables, including:

[0098] S31 defines the state space, action space, and reward function of the improved MSD3QN algorithm reinforcement learning agent model based on the state variables, optimization variables, and optimization objective of the joint optimization model in S2. The optimization variables are the independent variables solved in step S2, i.e., the action variables or action space quantities after Markov decision process modeling, specifically referring to the generator's output power, generator start / stop state, battery output power, and ship speed. The state variables of the optimization model become the agent's state variables after Markov decision process modeling. Both the state variables of the optimization model and the agent's state variables are essentially the state information of the all-electric ship. The name change aims to distinguish the states before and after Markov modeling.

[0099] The MSD3QN algorithm proposed in this invention improves upon the traditional DQN algorithm by integrating a competitive duel network architecture, double Q-learning, and multi-step bootstrapping techniques. It improves the agent's exploration efficiency by introducing a competitive network to reduce exploration of low-value actions, reduces overestimation of the value function to enhance the stability of agent training, and improves the convergence performance by increasing the accuracy of the value function estimate. The specific components of the agent model in this embodiment are as follows:

[0100] 1) State space S:

[0101]

[0102] Where s(t)∈S; It is the daily load power with uncertainties. The additional drag coefficient for navigation considering environmental factors, SOC(t) is the battery's state of charge, and T bat (t) is the battery operating temperature, ΔQ loss (t) represents the battery degradation capacity; v min (t) and v max (t) represents the upper and lower limits of the speed, Dis n Dis(t) represents the planned voyage, Dis(t) represents the actual voyage, and t represents the time of travel.

[0103] 2) The motion space A can be obtained from the parameters of each piece of equipment in the ship's electrical system in Table 3:

[0104]

[0105] Among them, a t ∈A;ΔP i,DG (t), P bat v(t) and v(t) represent the discrete actions corresponding to the change in power of the i-th generator, battery power, and speed at time t in a two-layer, multi-step competitive dual-depth Q-network agent, respectively. This is the minimum output power of the battery. v is the battery's maximum output power. min The minimum speed of an all-electric ship, v max N is the maximum speed of the all-electric ship. i,DG N bat N v These represent the number of discrete actions for the power of the i-th generator, the power of the battery, and the speed of the ship.

[0106] The action of generator 1 can be determined by power balance constraints. The action space dimensions of the agent are as follows:

[0107]

[0108] The reward function r of a two-layer, multi-step competitive, dual-depth Q-network agent is as follows:

[0109] r t = -β·[FC(t)+SC(t)+ρΔQ loss (t)+penalty_all(t)]

[0110] Where FC(t) is the fuel cost, SC(t) is the startup cost, and ρΔQ loss (t) represents the cost of equivalent battery capacity decay, β is the scaling factor of the reward function, and penalty_all(t) is the penalty value for violating the running constraints in step S1.

[0111] S32, the actions contained in the agent's action space are divided into wave-like actions A according to their response. f and steady-state action A s Two types of agents are used, employing an improved MSD3QN algorithm at the upper layer and the commercial solver Gurobi at the lower layer to simplify the agent's action space design. The wave-like action A... f This is for changes in battery output and speed, primarily used to respond to uncertainties arising from changes in the navigation environment and the randomness of user behavior; the actions change relatively frequently; steady-state action A s The power output of the generator is mainly used to provide steady-state power for the ship's electrical system, and its operation changes little during the same sailing phase.

[0112] Specifically, after classifying the agent's action space, the state space S and reward function r of the upper-layer MSD3QN agent remain unchanged, while its discrete action space A remains unchanged. upper The design is as follows, with a motion space dimension of 341:

[0113]

[0114] The optimization actions of the lower-level optimization solver can be solved using the established unit combination model, as detailed below, where δi,DG(t) represents the operating state of generator i:

[0115] A lower,t =A s ={P i,DG (t),δ i,DG (t)}, i=1,2,3,4

[0116] Where, δ i,DG (t) represents the operating state of the i-th generator at time t, P i,DG (t) represents the power of the i-th generator at time t; A f A represents the action space of wave-like actions; s The action space that represents steady-state or stable actions.

[0117] S33: Obtain state variables from the historical dataset of the all-electric ship, train the improved two-layer MSD3QN algorithm until the maximum number of iterations is reached, and then fix the network model parameters of the agent, such as... Figure 4 As shown, the specific steps of the improved two-layer MSD3QN algorithm based on agent action classification are as follows:

[0118] 1) Retrieve daily load history data of the all-electric ship from the database. Battery initial operating temperature data T bat,0 and navigation log data (including planned speed variation range v) min (t) and v max (t), Planned Flight Dis n (t) and other voyage information, as well as environmental information such as sea breeze and waves), will be used to convert the impact of the navigation environment on ship propulsion resistance into an equivalent environmental additional resistance coefficient. The historical data information read is used to construct a training set for the intelligent agent;

[0119] 2) Based on the physical parameters of each device in the all-electric ship's power system, determine the action dimension dimA of the upper-level intelligent agent. f =341;

[0120] 3) Set up two neural networks with the same architecture, namely the online Q network and the target Q network, and initialize the agent neural network hyperparameters and important reinforcement learning parameters, including the number of input and output layers, the number of hidden layers and the number of hidden layer neurons, the number of neurons in the competing network layer, the learning rate α, the discount factor γ, the number of bootstrapping steps n_step, the maximum number of iterations M, the number of time intervals per iteration T, the parameter update interval k of the online Q network, the parameter synchronization interval C of the target Q network, the experience pool size D, and the batch size of the batch data.

[0121] 4) Set the current iteration number to l. If l ≤ M, then randomly select the environment state variable s at a certain time t. t Start training the current generation; otherwise, skip to step 11.

[0122] 5) The upper-level MSD3QN agent first interacts with the environment to exchange state variables s t Interaction, obtaining volatility response action a based on an ε-greedy strategy. upper,t and its corresponding action reward r upper,t Then, the upper-level MSD3QN agent will perform a wave-like response action a. upper,t , environment state variable s t and the start-up time of each generator at the current moment Closing Time The information is passed to the lower-level Gurobi solver; the steady-state action a is obtained by minimizing the generator operating cost through the optimization objective of the lower-level solver. lower,t and its reward r lower,t The results are then uploaded to the upper-layer MSD3QN agent.

[0123] 6)a upper,t and a lower,t The response actions a together constitute the two-layer MSD3QN agent t Make the agent change from its current state s t Transition to the next state st+1, r upper,t and r lower,t The reward function r that together constitutes the two-layer MSD3QN agent t After obtaining the rewards for n_step time intervals after the current time t, a set of data (s) is generated. t ,a t G t ,s t+1 The flag is stored in the experience pool for updating the agent's network parameters, where G... t The n_step reward value after the agent adopts multi-step bootstrapping technique:

[0124]

[0125] 7) If the online Q-network parameter update condition is met, i.e. (t%k==0), then randomly select batch_size empirical data from the experience pool and iterate the network parameters of the upper-layer agent using the gradient update method described in the following formula; otherwise, jump to step 8:

[0126]

[0127]

[0128] 8) If the target Q-network parameter synchronization condition is met, i.e. (t%C==0), then the neural network parameters θ of the online Q-network will be... w The neural network parameters θ are synchronously copied to the target Q-network. w- (Otherwise, skip to step 9).

[0129] 9) If flag == 1 or t = T, the agent ends this iteration, then execute l = l + 1 and jump to step 4); otherwise, jump to step 10), where T represents the number of loops required for each iteration of the program, i.e., the number of time intervals T between single iterations.

[0130] 10) Set s t =s t+1 If t = t + 1, jump to step 5) to continue this training;

[0131] 11) After the agent training is completed, save the neural network model parameters θ. w .

[0132] S4, such as Figure 5 As shown, the trained two-layer MSD3QN agent model is deployed. By inputting the state variables at the current moment within the scheduling cycle, the adjustments to the generator and battery output power of the ship's power system and the speed of the all-electric ship at the current moment can be obtained. The state variables are then transferred to the next moment based on the response actions of the two-layer MSD3QN agent. The specific steps are as follows:

[0133] 1) Load the neural network model parameters θ w ;

[0134] 2) Input the environment state variable s at the current time t. t The intelligent agent will obtain the all-electric ship generator output P based on the principle of maximizing value. i,DG (t), Battery output P bat (t) and speed v(t);

[0135] 3) If t = T, jump to step 4); otherwise, set s. t =s t+1 And t = t + 1, jump to step 2);

[0136] 4) At the end of the scheduling period, output the agent action sequence {P} for this scheduling period. i,DG (t),P bat (t),v(t),t∈T}.

[0137] S5: Determine if the current time is the maximum time of the scheduling cycle. If the condition is met, it means that the current scheduling cycle has ended, and the action sequence of the agent is output. Otherwise, return to S4 to continue execution.

[0138] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0139] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A joint optimization method for power generation and navigation scheduling of an all-electric ship based on deep reinforcement learning, characterized in that: Includes the following steps: S1. Considering the uncertainty of the navigation environment, establish the speed model, generator output model, battery SOC model and battery capacity decay model of the all-electric ship. S2. Based on the constraints of the speed model, generator output model, battery SOC model and battery capacity decay model, a joint optimization model for ship power system power generation and navigation scheduling with uncertainty is established with the optimization objective of minimizing generator operating cost and battery capacity decay. S3 will model the Markov decision process using the joint optimization model and train the parameters of the two-layer multi-step competitive dual-depth Q agent model using the historical dataset of the state variables of the all-electric ship. S4. Deploy the trained two-layer multi-step competitive dual-depth Q agent model, input the state variables of the current moment in the scheduling cycle to obtain the agent's actions at the current moment, including the adjustment of the generator and battery output power of the ship's power system and the speed of the all-electric ship; the state variables of the two-layer multi-step competitive dual-depth Q network agent are then transferred to the next moment according to the response actions of the two-layer multi-step competitive dual-depth Q network agent. S5, determine whether the current time is the maximum time of the scheduling cycle. If the condition is met, it means that the current scheduling cycle has ended. Output the action sequence of the two-layer multi-step competitive dual-depth Q network agent. Otherwise, return to S4 to continue execution. Step S3 includes the following steps: S31, Define the state space, action space and reward function of the two-layer multi-step competitive dual-depth Q network agent based on the state variables, optimization variables and optimization objectives obtained from the joint optimization model in step S2; S32, the two-layer multi-step competitive dual-depth Q-network agent includes an upper-layer multi-step competitive dual-depth Q-network agent and a lower-layer solver; the actions contained in the agent's action space are divided into two categories according to the response: fluctuating actions and steady-state actions. Fluctuating actions are responded to by the upper-layer multi-step competitive dual-depth Q-network agent; steady-state actions are responded to by the lower-layer solver. S33: Obtain state variables from the historical dataset of state variables of the all-electric ship, train the model of the two-layer multi-step competitive dual-depth Q network agent until the maximum number of iterations is reached, and fix the model parameters of the two-layer multi-step competitive dual-depth Q network agent. In step S32, the upper-layer multi-step competitive dual-deep Q-network intelligent agent A upper The design is as follows: Lower-level optimization solver A lower，t The optimization action is solved by establishing a unit combination model, and its design is as follows: in, δ i,DG ( t Let P be the operating state of the i-th generator at time t. i,DG (t) represents the power of the i-th generator at time t; A f A represents the action space of wave-like actions; s The action space that represents steady-state or stable actions.

2. The method according to claim 1, characterized in that: In step S2, the generator operating cost includes fuel cost and start-up cost, and the battery capacity degradation is represented by the equivalent battery capacity degradation cost. The system operating constraints included in the joint optimization model include: overall system power balance constraints and spinning reserve constraints; generator operating constraints, including power output constraints, ramping constraints, and minimum start-stop time constraints; battery operating constraints, including power output constraints, SOC constraints, and temperature constraints; and all-electric ship navigation constraints, including speed and range constraints.

3. The method according to claim 1, characterized in that: In step S2, the uncertainties included in the joint optimization model include the uncertainties brought about by the navigation environment to the propulsion load of the all-electric ship and the uncertainties brought about by the power consumption behavior of the ship's users to the daily service load.

4. The method according to claim 2, characterized in that: In step S31, the state space S of the two-layer multi-step competitive dual-depth Q-network agent is as follows: Where s(t)∈S; It is the daily load power with uncertainties. The additional drag coefficient for navigation when environmental factors are taken into account. It refers to the battery's state of charge. It is the battery operating temperature. This refers to the battery's capacity degradation. and These are the upper and lower limits of the cruising speed. In order to plan the voyage, For the actual voyage, For sailing time; Action space of a two-layer, multi-step competitive, dual-depth Q-network agent A As shown in the following formula: in, a t ∈A; , , These represent the discrete actions corresponding to the change in power of the i-th generator, battery power, and speed at time t in a two-layer, multi-step competitive, dual-depth Q-network agent. Let i be the maximum ramp power of the i-th generator. This is the minimum output power of the battery. This is the battery's maximum output power. The minimum speed for a fully electric ship. This is the maximum speed of a fully electric ship. , , These represent the discrete number of actions for the i-th generator power, battery power, and speed, respectively. The dimensions of the action space are as follows: Reward function of a two-layer, multi-step competitive dual-depth Q-network agent r As shown in the following formula: in, For fuel costs, For start-up costs, For the equivalent battery capacity degradation cost, The scaling factor of the reward function, The penalty value is for violating the running constraints in step S1.

5. The method according to claim 2, characterized in that: In step S32, the fluctuating action is the change in battery output and speed, which is used to respond to the uncertainty caused by changes in the navigation environment and the randomness of user behavior; the steady-state action is the output of the generator, which is used to bear the steady-state power of the ship's electrical system.

6. The method for joint optimization of power generation and navigation scheduling of all-electric ships based on deep reinforcement learning according to claim 1, characterized in that, The training process of the S33 two-layer multi-step competitive dual-depth Q-network agent is as follows: 1) Read historical data information from the database; the historical data information includes: daily load history data of the all-electric ship. Battery initial operating temperature data Using navigation log data, the impact of the navigation environment on ship propulsion resistance is equivalently converted into an environmentally added drag coefficient. The historical data information read is used to construct a training set for the intelligent agent. 2) Determine the action dimension dim of the upper-level intelligent agent based on the physical parameters of each device in the all-electric ship's power system. A f = N bat × N v ; 3) Set up two neural networks with the same architecture: an online Q-network and a target Q-network; initialize the agent neural network hyperparameters and important reinforcement learning parameters, including the number of input and output layers, the number of hidden layers and the number of hidden layer neurons, the number of neurons in the competing network layers, and the learning rate. α Discount Factor γ Bootstrap steps n_step Maximum number of iterations M Number of time intervals per iteration T Online Q network parameter update interval k Target Q network parameter synchronization interval C Experience pool size D Batch processing data size batch_size ; 4) Set the current iteration number to l If satisfied l ≤ M Then a certain moment is randomly selected. t Environment state variables s t Begin training the current generation; otherwise, skip to step 11. 5) The upper-level intelligent agent first exchanges state variables with the environment. s t Interaction, based on ε-greedy Greedy strategies result in volatile actions. a upper,t and corresponding rewards r upper,t Then, the upper-level intelligent agent will perform wave-like actions. a upper,t State variables s t and the start-up time of each generator at the current moment Closing time Information is passed to the lower-level solver; the steady-state action is obtained by minimizing the generator operating cost through the lower-level solver. a lower,t and corresponding rewards r lower,t The results are then uploaded to the upper-level intelligent agent. 6) a upper,t and a lower,t The response actions of the two-layer, multi-step competitive dual-depth Q-network intelligent agent together constitute the response actions of the agent. a t Enables a two-layer, multi-step competitive, dual-depth Q-network agent to change from its current state. s t Transition to the next state s t+1 , r upper,t and r lower,t The reward function that together constitutes the two-layer, multi-step competitive dual-depth Q-network agent r t In obtaining the current time t after n_step After the rewards for each time period, the data ( s t , a t , G t , s t+1 , flag ) are stored in the experience pool for use in updating the agent's parameters, where G t After adopting multi-step bootstrapping technology for intelligent agents n Step reward value: where flag represents the program flag bit. Setting it to 0 will allow the program to run normally in iteration, while setting it to 1 will terminate the current iteration and start a new loop iteration. 7) If the conditions for updating online Q network parameters are met, then randomly select parameters from the experience pool. batch_size Use empirical data to iterate the network parameters of the upper-layer agent; otherwise, proceed to step 8). 8) If the target Q-network parameter synchronization condition is met, then the neural network parameters of the online Q-network will be... θ w Neural network parameters synchronously copied to the target Q-network θ w- Otherwise, skip to step 9: 9) If flag ==1 or t = T The two-layer, multi-step competitive, dual-depth Q-network agent completes its current iteration and then executes... l=l+1 If the condition is met, proceed to step 4; otherwise, proceed to step 10. T represents the number of iterations required for each iteration. 10) Settings s t = s t+1 and t = t +1, skip to step 5) to continue this training; 11) Training of the two-layer, multi-step competitive, dual-depth Q-network agent is complete; save the neural network model parameters. θ w .

7. The method for joint optimization of power generation and navigation scheduling of all-electric ships based on deep reinforcement learning according to claim 6, characterized in that, Step S4, the process of deploying the trained two-layer, multi-step competitive dual-depth Q-agent model, includes: 1) Loading neural network model parameters θ w ; 2) Enter the current time t State variables s t The intelligent agent will obtain the power output of the all-electric ship generator according to the principle of maximizing value. P i,DG ( t ), battery output P bat ( t ) and speed v ( t ); 3) If t = T (Jump to step 4), otherwise set s t = s t+1 and t = t +1, jump to step 2); 4) At the end of the scheduling period, output the agent action sequence for this scheduling period. P i,DG ( t ), P bat ( t ), v ( t ), t ∈ T }

Images