Hydrogen electricity and photovoltaic coupling ship energy management method based on Double DQN

Abstract

The invention discloses a hydrogen electricity and photovoltaic coupling ship energy management method based on Double DQN, and the method comprises the steps: introducing a shipborne photovoltaic power generation system, and building a ship energy management system; the energy regulation and control process of the hybrid power ship is represented through a Markov decision process MDP, and a ship energy management system is used as an intelligent agent in the MDP and continuously interacts with the system of the whole ship and the marine environment where the ship is located. The ship energy management system determines the behavior decision at the current moment according to the state fed back by the current environment and the award obtained by the behavior decision at the previous moment; and based on the constructed Double DQN network, training the intelligent agent through a Double DQN algorithm to obtain an energy optimal output proportion. According to the method, development of intelligent and automatic navigation of the ship is promoted, the ship can be adaptively adjusted under complex sea conditions and climate conditions through application of the Double DQN algorithm, and the shipping industry is promoted to develop towards a more intelligent and green direction.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of electrical engineering and marine engineering, and relates to ship energy management, specifically to a ship energy management method based on the coupling of hydrogen electricity and photovoltaics using Double DQN. Background Technology

[0002] With increasing global environmental pollution and greenhouse gas emissions, the shipping industry, as one of the world's major transportation sectors, faces increasingly severe carbon emission pressure. Traditional ship propulsion systems largely rely on fossil fuels, resulting in large emissions of carbon dioxide and other harmful gases. In response to global environmental policies and to reduce the shipping industry's carbon footprint, hydrogen energy, as a green and clean energy source, is gradually becoming an important research direction for future ship propulsion systems. Hydrogen fuel cell-powered ships offer advantages such as high efficiency and zero emissions, providing stable power output and significantly reducing environmental pollution.

[0003] However, the energy supply of hydrogen fuel cells is limited by the storage and supply of hydrogen, and their output power is easily affected by load fluctuations and the external environment, leading to energy supply instability. Furthermore, lithium batteries commonly used in ships face challenges in terms of charge / discharge efficiency and system lifespan. To improve the overall efficiency of energy management systems, photovoltaic systems are increasingly being introduced into ship energy management, utilizing solar energy to provide auxiliary power. However, photovoltaic output is highly volatile and subject to significant uncertainties due to factors such as weather and sunlight.

[0004] Therefore, how to achieve efficient and stable energy management under the combined effect of multiple energy forms such as hydrogen energy, lithium batteries and photovoltaics has become a technical problem that urgently needs to be solved. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the existing technology, a ship energy management method based on the coupling of hydrogen electricity and photovoltaics using Double DQN is provided.

[0006] Technical Solution: To achieve the above objectives, this invention provides a ship energy management method based on Double DQN-coupled hydrogen-electricity and photovoltaic systems, comprising the following steps:

[0007] S1: Introduce a shipborne photovoltaic power generation system and establish a ship energy management system;

[0008] S2: The energy regulation process of hybrid ships is characterized by Markov Decision Process (MDP). The ship energy management system is regarded as an intelligent agent in the MDP, which continuously interacts with the entire ship system and the marine environment in which the ship is located. The ship energy management system will determine the current action decision based on the current environmental feedback and the reward obtained from the previous action decision.

[0009] S3: Based on the constructed Double DQN network, the agent is trained using the Double DQN algorithm to obtain the optimal energy output ratio.

[0010] Furthermore, the ship energy management system in step S1 includes five parts: hydrogen fuel cell, lithium battery energy storage, shipborne photovoltaic power generation system, propulsion load, and total ship load.

[0011] Furthermore, the specific expression of the ship energy management system in step S1 is as follows:

[0012] Hydrogen fuel cells:

[0013] Power generation capacity:

[0014]

[0015] in, For hydrogen fuel cells Power generation during the period In order to be in Hydrogen consumption over a period of time; For the power generation efficiency of hydrogen fuel cells, The energy density of hydrogen gas;

[0016] Hydrogen fuel cell power output range:

[0017]

[0018] in, For hydrogen fuel cells Operating status during a given time period; and These represent the maximum and minimum output of the hydrogen fuel cell, respectively. This represents the maximum hydrogen storage capacity of a hydrogen fuel cell.

[0019] Lithium battery energy storage

[0020] Charge-discharge model:

[0021]

[0022]

[0023] in, for The energy storage capacity of lithium batteries at any given time; The self-discharge rate of lithium battery energy storage; and They are respectively Real-time actual charge / discharge power and theoretical charge / discharge power; Improve the charging efficiency of lithium battery energy storage; The energy storage and discharge efficiency of lithium batteries;

[0024] Lithium-ion battery state of charge range:

[0025]

[0026]

[0027] in, For lithium battery energy storage capacity, and These are the upper and lower limits of the state of charge of a lithium battery. For lithium batteries The state of charge at any given moment.

[0028] Shipborne photovoltaic power generation system:

[0029]

[0030]

[0031]

[0032] in, for The power output of the shipborne photovoltaic power generation system at all times. To improve the power generation efficiency of shipborne photovoltaic power generation systems, The area of ​​the photovoltaic panel; for Horizontal global irradiance at any time for Horizontal direct irradiance at any given time for Horizontal diffuse radiance at any given time; , These are the angle between the photovoltaic panel and the sunlight, and the angle of inclination with the horizontal plane, respectively. Where is the diffuse reflection constant. The reflectance index, Zenith angle;

[0033] Propulsion load:

[0034]

[0035] in, for Ship propulsion load during the period for Ship speed during the period; , The propulsion coefficient;

[0036] Total ship load:

[0037]

[0038] in, For the ship during navigation Total load during the period; for The workload of daily life services during certain periods.

[0039] Furthermore, in step S2, the Markov Decision Process (MDP) consists of quintuples. Uniquely determined; among them, The state space contains all possible states in the environment; The action space contains all executable actions of the agent. This is the state transition function. Indicates the agent's state Execute action Then transferred to state The probability of; The reward function needs to be manually set, corresponding to the optimization goal. The AI ​​can understand the reward value obtained from the environment based on the reward function and optimize its own strategy accordingly. This is a discount factor, and its value ranges from [value range missing]. Between these values, a larger value indicates a greater focus on long-term cumulative rewards, while a smaller value indicates a greater focus on short-term current rewards.

[0040] Furthermore, the establishment and operation of the intelligent agent in step S2 includes:

[0041] A1: As the optimization object of reinforcement learning, the agent's policy can be modeled as the probability of taking a certain action in a certain state:

[0042]

[0043] in, In order to be in Take action in a state The probability of;

[0044] A2: In an MDP, from The weighted sum of all rewards up to the end state is called the reward, and the reward is calculated as follows:

[0045]

[0046] in for The reward of time for Momentary rewards;

[0047] A3: Establish the state value function:

[0048]

[0049] in, This indicates that in MDP, policy-based... The state-value function is defined as the function derived from the state. Departure following strategy Expected returns that can be obtained;

[0050] Establish the action value function:

[0051] In MDP, since actions exist, an additional action-value function needs to be defined:

[0052]

[0053] in, This indicates that the strategy is followed in MDP. At that time, regarding the current state Execute action Expected return;

[0054] The relationship between state value function and action value function: when using a strategy In the middle, state The value is equal to the value based on the strategy in that state. The result of multiplying the probability of taking each action by its corresponding value and then summing the results:

[0055]

[0056] Use strategy At that time, state Take action below The value equals the immediate reward plus the product of the decayed state transition probabilities of all possible next states and their corresponding values:

[0057]

[0058] A4: Establish the Bellman optimal equation:

[0059] The Bellman expectation equation is derived by examining the relationship between the state value function and the action value function:

[0060]

[0061] Furthermore, the goal of reinforcement learning in step A4 is typically to find a policy that maximizes the expected reward for the agent starting from the initial state. A finite Markov decision process has at least one policy. Satisfy: For any state s and policy All have then strategy This is called the optimal strategy;

[0062] The state-value function under the optimal policy is called the optimal state-value function. :

[0063]

[0064] The action-value function under the optimal strategy is called the optimal action-value function. :

[0065]

[0066] The relationship between the optimal state value function and the optimal action value function is as follows:

[0067]

[0068] The Bellman optimal equation is obtained:

[0069]

[0070] Furthermore, the process of training the agent using the Double DQN algorithm in step S3 includes:

[0071] B1: Initialize the parameters of the experience replay pool, training network, and target network in the DQN network so that the parameters of the two networks are the same;

[0072] B2: In the DQN network, the agent continuously interacts with the marine environment in which the system exists, and receives a quadruple at each interaction. ,in For the present The state of the system at any given time, including The power of hydrogen fuel cells, the charging and discharging power of lithium batteries, the power generation power of photovoltaic power generation system, ship speed, the state of charge of lithium batteries, and the living service load at any time; For the agent to acquire the state Afterwards, according to A greedy strategy selects a hydrogen fuel cell load ratio factor. For the agent to execute Afterwards, the system environment will provide a reward to the agent; Represented as The state of the system at any given moment;

[0073] B3: Store the quadruplets obtained from the interaction between the Double DQN agent and the marine environment in which the system is located in the experience replay pool. The storage principle of the experience replay pool is as follows: if the number of tuples in the experience replay pool does not exceed the capacity limit of the experience replay pool, the quadruplets are directly stored in the experience replay pool; if the number of tuples in the experience replay pool exceeds the capacity limit of the experience replay pool, the earliest stored quadruplet is deleted and a new quadruplet is stored.

[0074] B4: When the number of samples in the experience replay pool reaches a specified value, the experience replay pool will randomly sample in small batches. The sample data is used as the training set for Double DQN. The data is input into the training network and the target network for training, and the loss function is calculated.

[0075] B5: Update the parameters of the Double DQN training network using gradient descent based on the loss function value. ;

[0076] B6: When training DQN network parameters When the number of updates reaches a multiple of C, the network parameters will be trained. Copy to target network parameters ;

[0077] B7: Repeat steps B4-B6 until the loss function value converges, ultimately achieving the optimal energy output ratio.

[0078] Furthermore, the quadruple in step B2 middle:

[0079] It is expressed as follows:

[0080]

[0081] This indicates that the intelligent agent obtains the current ship status. Afterwards, according to The greedy algorithm selects an optimal allocation decision among the ship's energy resources, namely:

[0082]

[0083] in, Greedy strategy means 1- Probability utilizes the action that maximizes expected reward, to An action is randomly selected based on probability; the hydrogen fuel cell load ratio factor is used as the decision variable, defined as follows:

[0084]

[0085] in, This represents the maximum output power of the hydrogen fuel cell system; when When the value equals 1, the hydrogen fuel cell system operates at maximum power. When the value is 0, the hydrogen fuel cell system operates under no-load conditions; Represented as:

[0086]

[0087] Indicates that the agent performs an action. The environment then provides the agent with a reward value.

[0088] Furthermore, the reward value in step B2 The reward is calculated based on a reward function designed to take into account the hydrogen consumption characteristics of the hydrogen fuel cell system, while ensuring that the state of charge of the lithium battery energy storage system is maintained within a safe range. Inside, the details are as follows:

[0089]

[0090]

[0091]

[0092]

[0093] in, for The cost of hydrogen fuel at all times For the price of hydrogen; for Maintenance costs of lithium battery energy storage systems at all times; for time Penalty function, for Penalty coefficient.

[0094] Furthermore, the calculation of the loss function in step B4 includes:

[0095] (31)

[0096] in As a discount factor, , The closer a value is to 0, the more attention is paid to short-term rewards; conversely, the closer a value is to long-term rewards. For policy-based In state The agent performs actions The expected reward of the network output after training with Double DQN. To train network parameters; For policy-based In state The agent performs actions The maximum expected return of the target network output after DQN These are the target network parameters.

[0097] Based on the above, the ship energy tube solution provided by this invention has the following characteristics:

[0098] 1. Fully explore the working characteristics of hydrogen fuel cells and lithium batteries, combine the uncertainty of photovoltaic power output, and construct a ship hybrid power system model based on the multi-condition load requirements of ships.

[0099] 2. Utilize the Double DQN algorithm for deep reinforcement learning to optimize the energy output of hydrogen fuel cells and lithium batteries in real time based on current load demand and energy storage status, and according to photovoltaic output characteristics.

[0100] 3. Based on the ship's historical operating conditions and current environmental conditions, the system learns and adjusts itself using the Double DQN algorithm to select the optimal energy supply strategy for the ship.

[0101] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0102] 1. This invention overcomes the limitations of traditional energy management systems in coordinating hydrogen and lithium energy storage by introducing a photovoltaic power generation system. It also utilizes the Double DQN algorithm to optimize energy efficiency, ensuring optimal synergy between different energy sources. Furthermore, by modeling the uncertainties of the photovoltaic system, it enhances the ship's energy management capabilities in complex environments, ensuring maximum energy utilization.

[0103] 2. The system performs well in dealing with uncertainties. The introduction of the Double DQN algorithm solves the overestimation problem of the DQN algorithm, enhances the ship system's ability to cope with dynamic environmental changes, and improves stability and reliability.

[0104] 3. Through precise energy regulation, the ship significantly increases its sailing distance while maintaining its core performance and speed. At the same time, the introduction of photovoltaic power generation system reduces dependence on traditional energy sources and improves the economy and greenness of the voyage.

[0105] 4. This invention promotes the development of intelligent and automated navigation of ships. Through the application of the Double DQN algorithm, ships can make adaptive adjustments under complex sea conditions and weather conditions, driving the shipping industry towards a more intelligent and green direction. Attached Figure Description

[0106] Figure 1 This is a schematic flowchart of the method of the present invention;

[0107] Figure 2 A diagram illustrating the training process for deploying EMS based on the Double DQN algorithm;

[0108] Figure 3 This is a graph showing the power variation of a hydrogen fuel cell with and without photovoltaic power. Detailed Implementation

[0109] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0110] Example 1:

[0111] like Figure 1 As shown, this embodiment provides a ship energy management method based on Double DQN-coupled hydrogen electricity and photovoltaics, including the following steps:

[0112] S1: Introduce a shipborne photovoltaic power generation system and establish a ship energy management system;

[0113] The specific expression of a ship energy management system is as follows:

[0114] Hydrogen fuel cells:

[0115] Power generation capacity:

[0116]

[0117] in, For hydrogen fuel cells Power generation during the period In order to be in Hydrogen consumption over a period of time; For the power generation efficiency of hydrogen fuel cells, The energy density of hydrogen gas;

[0118] Hydrogen fuel cell power output range:

[0119]

[0120] in, For hydrogen fuel cells Operating status during a given time period; and These represent the maximum and minimum output of the hydrogen fuel cell, respectively. This represents the maximum hydrogen storage capacity of a hydrogen fuel cell.

[0121] Lithium battery energy storage

[0122] Charge-discharge model:

[0123]

[0124]

[0125] in, for The energy storage capacity of lithium batteries at any given time; The self-discharge rate of lithium battery energy storage; and They are respectively Real-time actual charge / discharge power and theoretical charge / discharge power; Improve the charging efficiency of lithium battery energy storage; The energy storage and discharge efficiency of lithium batteries;

[0126] Lithium-ion battery state of charge range:

[0127]

[0128]

[0129] in, For lithium battery energy storage capacity, and These are the upper and lower limits of the state of charge of a lithium battery. For lithium batteries The state of charge at any given moment.

[0130] Shipborne photovoltaic power generation system:

[0131]

[0132]

[0133]

[0134] in, for The power output of the shipborne photovoltaic power generation system at all times. To improve the power generation efficiency of shipborne photovoltaic power generation systems, The area of ​​the photovoltaic panel; for Horizontal global irradiance at any time for Horizontal direct irradiance at any given time for Horizontal diffuse radiance at any given time; , These are the angle between the photovoltaic panel and the sunlight, and the angle of inclination with the horizontal plane, respectively. Where is the diffuse reflection constant. The reflectance index, Zenith angle;

[0135] Propulsion load:

[0136]

[0137] in, for Ship propulsion load during the period for Ship speed during the period; , The propulsion coefficient;

[0138] Total ship load:

[0139]

[0140] in, For the ship during navigation Total load during the period; for The workload of daily life services during certain periods.

[0141] S2: The energy regulation process of hybrid ships is characterized by Markov Decision Process (MDP). The ship energy management system is regarded as an intelligent agent in the MDP, which continuously interacts with the entire ship system and the marine environment in which the ship is located. The ship energy management system will determine the current action decision based on the current environmental feedback and the reward obtained from the previous action decision.

[0142] The core of the energy management system proposed in this invention is the development of a highly adaptive and economically efficient Energy Management System (EMS). The integrated electric power system of a hybrid-powered ship mainly consists of a hydrogen fuel cell, a lithium battery energy storage system, and a photovoltaic power generation system. The EMS adjusts the power ratio provided by the hydrogen fuel cell and lithium battery energy storage system in real time based on the ship's operating status, changes in the external environment, and the output characteristics of the photovoltaic power generation system. The energy regulation process of a hybrid-powered ship can be characterized by a Markov decision process (MDP), where the current state of the system depends only on the state at the previous moment. In an MDP, an agent executes actions, continuously interacting with the environment and improving its strategy based on environmental feedback.

[0143] Markov Decision Processes (MDPs) consist of quintuples. Uniquely determined; among them, The state space contains all possible states in the environment; The action space contains all executable actions of the agent. This is the state transition function. Indicates the agent's state Execute action Then transferred to state The probability of; The reward function needs to be manually set, corresponding to the optimization goal. The AI ​​can understand the reward value obtained from the environment based on the reward function and optimize its own strategy accordingly. This is a discount factor, and its value ranges from [value range missing]. Between these values, a larger value indicates a greater focus on long-term cumulative rewards, while a smaller value indicates a greater focus on short-term current rewards.

[0144] The establishment and operation of intelligent agents include:

[0145] A1: As the optimization object of reinforcement learning, the agent's policy can be modeled as the probability of taking a certain action in a certain state:

[0146]

[0147] in, In order to be in Take action in a state The probability of;

[0148] A2: In an MDP, from The weighted sum of all rewards up to the end state is called the reward, and the reward is calculated as follows:

[0149]

[0150] in for The reward of time for Momentary rewards;

[0151] A3: Establish the state value function:

[0152]

[0153] in, This indicates that in MDP, policy-based... The state-value function is defined as the function derived from the state. Departure following strategy Expected returns that can be obtained;

[0154] Establish the action value function:

[0155] In MDP, since actions exist, an additional action-value function needs to be defined:

[0156]

[0157] in, This indicates that the strategy is followed in MDP. At that time, regarding the current state Execute action Expected return;

[0158] The relationship between state value function and action value function: when using a strategy In the middle, state The value is equal to the value based on the strategy in that state. The result of multiplying the probability of taking each action by its corresponding value and then summing the results:

[0159]

[0160] Use strategy At that time, state Take action below The value equals the immediate reward plus the product of the decayed state transition probabilities of all possible next states and their corresponding values:

[0161]

[0162] A4: Establish the Bellman optimal equation:

[0163] The Bellman expectation equation is derived by examining the relationship between the state value function and the action value function:

[0164]

[0165] The goal of reinforcement learning is usually to find a policy that maximizes the expected reward for the agent starting from the initial state. A finite Markov decision process has at least one policy. Satisfy: For any state s and policy All have then strategy This is called the optimal strategy;

[0166] The state-value function under the optimal policy is called the optimal state-value function. :

[0167]

[0168] The action-value function under the optimal strategy is called the optimal action-value function. :

[0169]

[0170] The relationship between the optimal state value function and the optimal action value function is as follows:

[0171]

[0172] The Bellman optimal equation is obtained:

[0173]

[0174] S3: Based on the constructed Double DQN network, the agent is trained using the Double DQN algorithm to obtain the optimal energy output ratio.

[0175] The process of training an agent using the Double DQN algorithm includes:

[0176] B1: Initialize the parameters of the experience replay pool, training network and target network in the DQN network so that the parameters of the two networks are the same. Copy the same network parameters to initialize the target network and initialize the experience replay pool.

[0177] B2: In the DQN network, the agent continuously interacts with the marine environment in which the system exists, and receives a quadruple at each interaction. ,in For the present The state of the system at any given time, including The power of hydrogen fuel cells, the charging and discharging power of lithium batteries, the power generation power of photovoltaic power generation system, ship speed, the state of charge of lithium batteries, and the living service load at any time; For the agent to acquire the state Afterwards, according to A greedy strategy selects a hydrogen fuel cell load ratio factor. For the agent to execute Afterwards, the system environment will provide a reward to the agent; Represented as The state of the system at any given moment;

[0178] Quadruple middle:

[0179] It is expressed as follows:

[0180]

[0181] This indicates that the intelligent agent obtains the current ship status. Afterwards, according to The greedy algorithm selects an optimal allocation decision among the ship's energy resources, namely:

[0182]

[0183] in, Greedy strategy means 1- Probability utilizes the action that maximizes expected reward, to An action is randomly selected with probability; in this embodiment, the hydrogen fuel cell load ratio factor is used as the decision variable, which is defined as:

[0184]

[0185] in, This represents the maximum output power of the hydrogen fuel cell system; when When the value equals 1, the hydrogen fuel cell system operates at maximum power. When the value is 0, the hydrogen fuel cell system operates under no-load conditions; Represented as:

[0186]

[0187] Indicates that the agent performs an action. The environment then provides the agent with a reward value. The reward is calculated based on a reward function designed to take into account the hydrogen consumption characteristics of the hydrogen fuel cell system, while ensuring that the state of charge of the lithium battery energy storage system is maintained within a safe range. Inside, the details are as follows:

[0188]

[0189]

[0190]

[0191]

[0192] in, for The cost of hydrogen fuel at all times For the price of hydrogen; for Maintenance costs of lithium battery energy storage systems at all times; for time Penalty function, for Penalty coefficient.

[0193] B3: Store the quadruplets obtained from the interaction between the Double DQN agent and the marine environment in which the system is located in the experience replay pool. The storage principle of the experience replay pool is as follows: if the number of tuples in the experience replay pool does not exceed the capacity limit of the experience replay pool, the quadruplets are directly stored in the experience replay pool; if the number of tuples in the experience replay pool exceeds the capacity limit of the experience replay pool, the earliest stored quadruplet is deleted and a new quadruplet is stored.

[0194] B4: When the number of samples in the experience replay pool reaches a specified value, the experience replay pool will randomly sample in small batches. The sample data is used as the training set for Double DQN. The data is input into the training network and the target network for training, and the loss function is calculated.

[0195] The calculation of the loss function includes:

[0196] (31)

[0197] in As a discount factor, , The closer a value is to 0, the more attention is paid to short-term rewards; conversely, the closer a value is to long-term rewards. For policy-based In state The agent performs actions The expected reward of the network output after training with Double DQN. To train network parameters; For policy-based In state The agent performs actions The maximum expected return of the target network output after DQN These are the target network parameters.

[0198] B5: Update the parameters of the Double DQN training network using gradient descent based on the loss function value. ;

[0199] B6: When training DQN network parameters When the number of updates reaches a multiple of C, the network parameters will be trained. Copy to target network parameters ;

[0200] B7: Repeat steps B4-B6 until the loss function value converges, ultimately achieving the optimal energy output ratio.

[0201] Example 2:

[0202] To verify the feasibility and effectiveness of the method of the present invention, this embodiment uses hydrogen fuel cells, lithium battery energy storage systems, and photovoltaic power generation systems in hybrid ships as research objects for case studies and simulation analysis, as detailed below:

[0203] The basic parameters of hybrid-powered ships are as follows: time interval The rated power of the hydrogen fuel cell is 1 hour. The power output is 500kW, the energy storage system capacity is 2000kWh, the photovoltaic power generation efficiency is 0.18, and the photovoltaic panel area is 300 square meters. State of charge safe range The charge-discharge conversion efficiency is [0.2, 0.8]. and Both are 0.95, the self-discharge rate of lithium battery energy storage. The maximum ship speed is 20 knots, and the maximum living service load is 200kW. The intelligent decision-making level of the EMS is gradually improved through experience playback of historical data and dynamic interactive decision-making of ship intelligent agents.

[0204] Figure 2 The training process for deploying EMS based on the Double DQN algorithm is presented. A total of 1500 Markov decision sequences were explored and learned. It can be seen that the cumulative reward has basically stabilized and converged by the 400th round, but there is a drop in reward at the 1250th round. This may be because the model relies too heavily on... A greedy strategy can lead to a short-term drop in rewards. This phenomenon is generally a temporary reaction to an imbalance between exploration and exploitation, and the rewards will eventually recover in subsequent training rounds.

[0205] In this embodiment, the average output power of the hydrogen fuel cell and the operating cost per kilometer under both photovoltaic and photovoltaic operating modes are shown in Table 1.

[0206] Table 1 Comparison of operating results with and without photovoltaics

[0207]

[0208] As shown in Table 1, both the optimized average load demand and the average output power of the hydrogen fuel cell are lower than those before optimization, indicating that the method of this invention fully considers the conditions of photovoltaic power generation system integration, achieving a harmonious balance between maritime economics and environmental sustainability. Compared with no photovoltaic integration, after integrating photovoltaics, the average output power of the hydrogen fuel cell decreased from 444.91kW to 389.35kW, a reduction of 12.49%. This is because after integrating the photovoltaic power generation system, the system can provide some electricity, thereby reducing dependence on the hydrogen fuel cell. The operating cost per kilometer decreased from 0.93 million yuan to 0.75 million yuan, a reduction of 19.35%. This is because with the photovoltaic power generation system, the load on the hydrogen fuel cell is reduced, and the system's demand for hydrogen is also reduced, thus reducing hydrogen fuel consumption.

[0209] Figure 3 The curves show the power variation of hydrogen fuel cells with and without photovoltaic (PV) access. From 0-8 hours, when sunlight conditions are poor, the curves for both PV and non-PV access curves show roughly the same trend. From 8-18 hours, when sunlight conditions are better, the curve with PV access is below the curve without PV access, indicating that the PV power generation system can generate more electricity to support ship operation. From 18-24 hours, when sunlight conditions are poor, the curve without PV access is still above the curve with PV access, especially from 22-24 hours. This is because with PV access, the energy storage system can better perform peak shaving and valley filling, especially when directly using lithium battery energy storage systems for power supply near ports. Therefore, the photovoltaic-access hydrogen-powered ship energy management method based on the Double DQN algorithm provided in this invention effectively improves the economic efficiency and environmental friendliness of ship energy.

Claims

1. A Double DQN-based ship energy management method for coupling hydrogen electricity and photovoltaics, characterized in that, It comprises the following steps: S1: introducing a ship-mounted photovoltaic power generation system and establishing a ship energy management system; S2: representing the energy regulation process of a hybrid ship by a Markov decision process (MDP), taking the ship energy management system as an agent in the MDP, continuously interacting with the entire ship system and the marine environment in which the ship is located, and determining the behavior decision at the current time according to the state feedback of the current environment and the reward obtained by the behavior decision at the last time; S3: training the agent by a Double DQN algorithm based on the constructed Double DQN network, and obtaining the optimal energy output ratio.

2. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 1, characterized in that, The ship energy management system in step S1 comprises a hydrogen fuel cell, a lithium battery energy storage, a ship-mounted photovoltaic power generation system, a propulsion load and a total ship load.

3. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 2, characterized in that, The ship energy management system in step S1 is specifically expressed as follows: Hydrogen fuel cell: Power generation: ； wherein, hydrogen fuel cell period power generation, is the amount of hydrogen consumed during the time period; is the power generation efficiency of the hydrogen fuel cell, is the energy density of hydrogen gas; Hydrogen fuel cell power output range: ； wherein, is the operating state of the hydrogen fuel cell at the time period; and are the maximum and minimum power output of the hydrogen fuel cell, respectively; is the maximum hydrogen storage capacity of the hydrogen fuel cell; Lithium battery energy storage Charging and discharging model: ； ； wherein, is the lithium battery energy storage capacity at the moment; is the lithium battery energy storage self-discharge rate; and are respectively the actual charging and discharging power and the theoretical charging and discharging power at the moment; is the lithium battery energy storage charging efficiency; is the lithium battery energy storage discharging efficiency; Lithium battery state of charge interval: ； ； wherein, is the lithium battery energy storage capacity, and are the upper and lower lithium battery state of charge limits, respectively, is the lithium battery state of charge at the time instant t. is the lithium battery state of charge at the time instant t. Ship-mounted photovoltaic power generation system: ； ； ； wherein, is the instantaneous power generated by the shipboard photovoltaic power generation system, is the efficiency of the shipboard photovoltaic power generation system, is the area of the photovoltaic panel; is the instantaneous horizontal global irradiance, is the instantaneous horizontal direct irradiance, is the instantaneous horizontal diffuse irradiance; , is the angle between the photovoltaic panel and the sun rays and the angle of inclination from the horizontal, respectively; is the diffuse reflection constant, is the reflection index, is the zenith angle; Propulsion load: ； wherein, is the ship propulsion load for the period, is the ship speed for the period; , is the propulsion coefficient; Total ship load: ； wherein, for the ship during navigation period total load; for period life service load.

4. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 3, characterized in that, The Markov decision process MDP in the step S2 is a five-tuple unique; wherein, is a state space containing all possible states in the environment; is an action space containing all executable actions of the agent; is a state transition function, denotes the probability that the agent transitions from a state to a state after performing an action ; is a reward function; is a discount factor, taking values in the range .

5. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 4, characterized in that, The establishment and operation of the agent in step S2 comprise: A1: As an optimization object of reinforcement learning, the strategy of the agent is modeled as the probability of taking a certain action in a certain state: A3: Establish a state value function: ； wherein, to take action in state the probability that A2: In one MDP, from The weighted sum of all rewards from time instant to the terminal state is called the return, which is computed as follows: ； wherein is a reward at the moment, is a reward at the moment; Establish an action value function: ； wherein, represents the state-value function in MDP based on policy is defined as the expected return that can be obtained from state following policy . A4: Establish the Bellman optimal equation: ； wherein, represents the expected reward obtained by performing action at the current state when the MDP follows policy . Relationship between state value function and action value function: In using a policy the value of a state is equal to the result of multiplying the probabilities of taking all actions in that state based on the policy and summing the corresponding values: ； Using the policy At time The value of an action is equal to the immediate reward plus the discounted sum of the state transition probabilities of all possible next states multiplied by the corresponding values: ； The relationship between the optimal state value function and the optimal action value function is as follows: The Bellman expectation equation is derived from the relationship between the state value function and the action value function: .

6. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 5, characterized in that, The limited Markov decision process in step A4 has at least one policy satisfies: for any state s and policy , both have , the policy is called the optimal policy; The state value function under the optimal policy is called the optimal state value function : ； The action value function under the optimal strategy is called the optimal action value function : ； Get the Bellman optimal equation: ； The process of training the agent by the Double DQN algorithm in step S3 comprises: 。 7. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 6, characterized in that, B1: Initialize the experience replay pool, training network and target network parameters in the DQN network, so that the two network parameters are the same; B3: Store the four-tuple obtained by the interaction of the Double DQN agent and the marine environment in which the system is located in the experience replay pool; the storage principle of the experience replay pool is: if the number of tuples in the experience replay pool does not exceed the upper limit of the capacity of the experience replay pool, directly store the four-tuple in the experience replay pool; if the number of tuples in the experience replay pool exceeds the upper limit of the capacity of the experience replay pool, delete the earliest stored four-tuple and store the new four-tuple; B2: In the DQN network, the agent continuously interacts with the marine environment in which the system exists, and receives a quadruple at each interaction. ,in For the present The state of the system at any given time, including The power of hydrogen fuel cells, the charging and discharging power of lithium batteries, the power generation power of photovoltaic power generation system, ship speed, the state of charge of lithium batteries, and the living service load at any time; For the agent to acquire the state Afterwards, according to A greedy strategy selects a hydrogen fuel cell load ratio factor. For the agent to execute Afterwards, the system environment will provide a reward to the agent; Represented as The state of the system at any given moment; B7: Repeat steps B4-B6 until the loss function value converges, and finally obtain the optimal energy output ratio. B4: When the samples in the experience replay pool reach a specified value, the experience replay pool is randomly sampled in small batches The sample data is input into the training network and the target network for training, and the loss function is calculated. B5: updating Double DQN training network parameters according to loss function value using gradient descent method ; B6: When the DQN training network parameters are updated a multiple of C times, the training network parameters are copied to the target network parameters ; The calculation of the loss function in step B4 comprises:

8. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 7, characterized in that, The quad of step B2 In: is represented as follows: ； representing the agent obtaining the current ship state then, according to a greedy algorithm selects an optimal allocation decision between the ship's energy sources, namely: ； wherein, The greedy policy represents, 1- The probability exploits the action with the highest expected reward, to The probability randomly selects an action; the hydrogen fuel cell load ratio factor is used as the decision variable, which is defined as: ； wherein, Pmax is the maximum output power of the hydrogen fuel cell system; when Pmax is the maximum output power of the hydrogen fuel cell system; when Pmax is the maximum output power of the hydrogen fuel cell system; when is represented as: ； representing the agent performing an action an environment then feeds back to the agent a reward value.

9. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 8, characterized in that, The reward value in step B2 The reward is calculated according to a reward function, the design of which takes into account the hydrogen consumption characteristics of the hydrogen fuel cell system, while ensuring that the state of charge of the lithium battery energy storage system is maintained in a safe interval The reward is calculated according to a reward function, the design of which takes into account the hydrogen consumption characteristics of the hydrogen fuel cell system, while ensuring that the state of charge of the lithium battery energy storage system is maintained in a safe interval The reward is calculated according to a reward function, the design of which takes into account the hydrogen consumption characteristics of the hydrogen fuel cell system, while ensuring that ； ； ； ； wherein, is instantaneous hydrogen fuel cost, is the hydrogen price; is instantaneous lithium battery energy storage system maintenance cost; is instantaneous penalty function, is penalty coefficient.

10. The Double DQN-based hydrogen power and photovoltaic coupled ship energy management method according to claim 9, characterized in that, ​ （31）； in As a discount factor, ; For policy-based In state The agent performs actions The expected reward of the network output after training with Double DQN. To train network parameters; For policy-based In state The agent performs actions The maximum expected return of the target network output after DQN These are the target network parameters.

Images