Ship energy real-time optimization method and system under new energy uncertainty

Abstract

The invention provides a ship energy real-time optimization method and system under new energy uncertainty, and relates to the technical field of artificial intelligence and ships, and the method comprises the steps: S1, building a new energy ship power network system model; s2, forming a Markov representation ship energy regulation and control process according to the established power network system model; and S3, according to a ship energy regulation and control process, optimizing ship energy scheduling under new energy uncertainty based on a DQN algorithm. According to the invention, the utilization mode of the ship photovoltaic energy under the condition of uncertain photovoltaic output can be optimized, and the energy management problem of the new energy ship under the condition of uncertain photovoltaic output is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and ship technology, and in particular to a method and system for real-time optimization of ship energy under uncertainty of new energy. Background Art

[0002] With the dual carbon goals, the country's energy structure is transforming towards a green and clean direction. In the shipping industry, intelligent all-electric ships are gradually becoming an important way to save energy, reduce emissions and improve ship operation efficiency. While bringing opportunities, it also brings huge challenges to the shipping industry. Due to the changeable environmental conditions of marine operations and frequent load fluctuations, the ship's power system has a certain impact. Therefore, the energy optimization management of all-electric ships will become a key technology to promote the safe, stable and efficient operation of all-electric ships.

[0003] Energy optimization management of electric ships will become a key technology to promote safe, stable and efficient operation of electric ships. A good energy management system can improve the service life of the energy storage system of all-electric ships, promote the efficient use of energy, and accelerate the deep integration of transportation networks and energy networks. Therefore, the operation of electric ships at present needs to consider how to manage and distribute the energy of a ship under the uncertainty of new energy power generation and frequent load fluctuations, so as to achieve green, efficient, safe and stable operation of ships.

[0004] Reinforcement learning methods, which have the advantage of being model-free, can find the optimal strategy in an uncertain environment. They are gradually being widely used in optimizing and formulating strategies for microgrids, and have played an important role in optimizing the configuration of terrestrial energy. However, few people have studied the combination of reinforcement learning methods with marine transportation. Most of the current research is to optimize the output of ship power generation systems through traditional methods to reduce ship operating costs. However, none of the above methods involve the real-time coordinated optimization of the energy of power generation and energy storage systems during ship navigation.

[0005] With the continuous reform of my country's inland river shipping industry, the number of new energy ships is increasing. How to better utilize the advantages of new energy power generation under the condition of frequent load fluctuations and uncertainty of new energy power generation, reduce the fuel consumption of diesel engines and maintain the charge state of energy storage at the same time is one of the issues that need to be studied urgently.

[0006] The existing literature "Xiao Nengqi, Xu Xiang, Zhou Ruiping. Energy management control strategy of ship diesel-electric hybrid power system [J]. Journal of Harbin Engineering University, 2020, 41(01): 153-160.", in order to make the ship's propulsion system operate under the most reasonable working conditions during navigation, a ship's four-engine dual-propeller hybrid propulsion system is taken as the research object, and the operating conditions and corresponding energy flows of the system's four single-machine modes, six PTI modes and two parallel modes are studied; the characteristic parameters of the ship's navigation conditions are extracted and the standard navigation conditions are established, and the navigation conditions to which the samples belong are identified by using the established navigation condition fuzzy pattern recognition model, and a rule-based energy management control strategy based on condition identification is proposed to uniformly manage and distribute the energy of the ship's diesel-electric hybrid propulsion system. During the ship's navigation, the above method is used to identify the navigation conditions in real time, so that the ship can sail under the optimal conditions by switching the clutch state command to reduce fuel consumption and emissions.

[0007] Then the technology in this document is an energy management method based on working conditions. Based on the knowledge and rules of domain experts, clear rules can be established and perform well in familiar scenarios. However, it is difficult to design complete rules in complex, nonlinear or unknown environments, resulting in performance degradation and difficulty in adapting to dynamic and changing offshore environments.

[0008] Existing literature: "Fang Shiton, Wang Hongdong, Zhang Shenxi, et al. Optimal ship energy storage system scheduling considering battery life loss [J]. Proceedings of the CSEE, 2020, 40(23): 7566-7578." With the advancement of electric propulsion technology, all-electric ships have become the main trend of future ship design. In order to ensure the power balance of all-electric ships, batteries, as an energy buffer device, are gradually used in ship design and operation. However, compared with the load of land microgrids, the ship load changes more drastically, which may lead to frequent deep charging and discharging of the ship energy storage system, resulting in loss of energy storage system life. In order to extend the service life of the ship energy storage system, this paper proposes an economic scheduling model for all-electric ship operation considering battery life loss. The model takes ship operating costs as the objective function and greenhouse gas emissions, ship power generation system, voyage management, etc. as constraints. After constraint relaxation, the proposed model is transformed into a two-layer optimization problem and solved using the column and constraint generation algorithm.

[0009] However, the research on energy management of all-electric ships is to optimize from the single energy storage aspect, and has not yet involved the real-time coordinated optimization of the energy of power generation and energy storage systems. Summary of the invention

[0010] In view of the defects in the prior art, the present invention provides a method and system for real-time optimization of ship energy under new energy uncertainty.

[0011] According to a method and system for real-time optimization of ship energy under uncertainty of new energy provided by the present invention, the scheme is as follows:

[0012] In a first aspect, a method for real-time optimization of ship energy under new energy uncertainty is provided, the method comprising:

[0013] Step S1: Establish a new energy ship power network system model;

[0014] Step S2: According to the established power network system model, a ship energy control process represented by Markov is formed;

[0015] Step S3: According to the ship energy control process, the ship energy scheduling under the uncertainty of new energy is optimized based on the DQN algorithm.

[0016] Preferably, the step S1 comprises:

[0017] Step S1.1: Establishing an uncertain model of shipborne photovoltaic power generation;

[0018] The output of the shipboard photovoltaic power generation system is corrected by using the data along the ship's route and combining the correction method, as follows:

[0019]

[0020] Among them, P t PV represents the output power of the onboard photovoltaic power generation system at time t; η PV Represents the conversion efficiency of solar photovoltaic panels; S PV is the area of ​​photovoltaic panels (m 2 );I t Represents the sunlight intensity received at time t (W / m 2 ); and They represent the direct sunlight, scattered sunlight and ground reflection at time t respectively; θ is the angle between the photovoltaic panel and the light; φ is the angle between the photovoltaic panel and the horizontal plane; ε is the reflectivity; η PV_ref is the maximum conversion efficiency of solar photovoltaic panels; η MPPT is the maximum power tracking control conversion efficiency; T is the working temperature of the solar photovoltaic panel; T ref is the temperature of the solar photovoltaic panel when it is working normally; β is the temperature coefficient;

[0021] Step S1.2: Establishing a ship propulsion load model;

[0022]

[0023] Among them, P tPro represents the propulsion load of the ship at time t; V t represents the speed of the electric ship at time t; c1 and c2 are the ship's own parameters related to the hull resistance, which are determined by the ship's own design characteristics;

[0024] Step S1.3: Establishing a shipboard energy storage charging and discharging power model;

[0025]

[0026] Among them, e t and e t-1 Respectively represent the charge state of the shipboard energy storage system at time t and the previous time; η c and η d They represent the charging and discharging efficiency of the energy storage system; S represents the capacity of the energy storage system; P t ESS represents the charging and discharging power of the energy storage system at time t, and ΔT represents the time interval;

[0027] Step S1.4: Establishing a ship energy system network model;

[0028] P t G +P t PV +P t ESS =P t Ser +P t Pro

[0029] Where P t G , P t PV and P t ESS Respectively represent the output power of the generator, photovoltaic system and energy storage system at time t; P t Ser P represents the ship life service load at time t; t Pro represents the ship propulsion load at time t;

[0030] The constraints of the new energy ship power system are as follows:

[0031] 1) Diesel generator active power limit:

[0032]

[0033] Among them, P t G represents the output power of the ship's diesel engine at time t; and Respectively represent the minimum and maximum output power of the diesel generator system;

[0034] 2) Energy storage system power limit:

[0035]

[0036] Where P t ESS Represents the charging and discharging power of the energy storage system at time t; and They represent the lower and upper limits of the energy storage system power respectively.

[0037] Preferably, step S2 comprises:

[0038] Step S2.1: Design state space;

[0039] S t = {v t ,P t Ser ,P t PV ,e t ,t}

[0040] Among them, v t represents the ship's speed at time t; P t Ser represents the life service load at time t; P t PV is the output power of the photovoltaic panel at time t; e t is the state of charge of the energy storage system at time t;

[0041] Step S2.2: Design action space;

[0042] The diesel generator load ratio factor is used as the decision variable, denoted as ratio t , which is defined as:

[0043] ratio t =P t G / P GN

[0044] A t ={ratio t}

[0045] Among them, A t represents the state space; P t G is the output power of the diesel generator at time t; P GN is the rated power of the diesel generator;

[0046] Step S2.3: Design reward function;

[0047] The reward function is segmented according to the state of charge of the energy storage system; the decision system makes a decision action A t , the charge state of the energy storage system is determined by e t becomes e t+1 , if e t+1 If the upper and lower limits of the state of charge are exceeded, e t and e t+1 If the trend is opposite to the expectation, a penalty needs to be imposed through the reward function, as shown below:

[0048] ①e t+1 <0:

[0049] r t =-9

[0050] ②0≤e t+1 <e Low :

[0051]

[0052] ③e High ≤e t+1 <1:

[0053]

[0054] ④e Low ≤e t+1 <e High :

[0055] r t =0.4e t+1 -0.6ratio t

[0056] ⑤e t+1 >1:

[0057] r t =-3

[0058] Among them, e t 、e t+1 is the charge state of the energy storage system at time t and the next time; e Low and e High are the lower and upper limits of the safe range of the state of charge respectively; |Δe max | represents the absolute value of the maximum change in the state of charge of the energy storage system within a period of time; ratio t is the load proportional factor of the generator; r t is the reward obtained by the agent at time t.

[0059] Preferably, the step S3 comprises:

[0060] Step S3.1: Estimate the value function based on the Q-learning algorithm;

[0061] Based on the idea of ​​temporal difference learning, the optimal action value function is estimated through the Q-learning algorithm. The temporal difference update formula is as follows:

[0062]

[0063] Among them, Q(S t ,A t ) is the action value function at time t, α is the learning rate, R t is the single-step reward obtained by the agent, γ is the discount factor, Q(S t+1 ,a) is the action value function of taking action a in the next state at time t+1;

[0064] Step S3.2: Establishing a real-time energy management strategy based on the DQN algorithm;

[0065] The deep Q network algorithm is used to train the ship agent. Step S3.2 uses the temporal difference method to incrementally update the action value function Q(s,a); so that Q(s,a) and r+γmax a'∈A Q(s',a') is close, and the DQN algorithm update is to make Q ω (s,a) approximates r+γmax a'∈A Q ω (s',a'), where ω * are the parameters of the Q network;

[0066] For the four-tuple dataset {(s i ,a i ,r i ,s′ i )} i＝1,…,N , the neural network learns the optimal value of the parameters by minimizing the loss function, so the Q network used to fit the action value function is designed as follows:

[0067]

[0068] Among them, N is the number of four-tuple data sets, Q ω (s i ,a i ) is the Q value of all actions a in each state s, r i For state s i The reward obtained under ω (s′ i,a′) is the Q value of executing action a′ in state s′.

[0069] Preferably, the DQN algorithm introduces two sets of neural networks, namely the current network Q ω (s,a) and target network Among them, the current network is used to calculate Q in the loss function ω (s i ,a i ), the target network is used to calculate

[0070] The target network is regarded as a copy of the current network, and its network parameters are synchronized with the current network at regular intervals.

[0071] Preferably, the DQN algorithm introduces an experience replay mechanism;

[0072] The four-tuple data obtained from each interactive sampling is stored in the experience replay pool, and random sampling is performed from the experience replay pool when updating the Q network.

[0073] In a second aspect, a real-time optimization system for ship energy under new energy uncertainty is provided, the system comprising:

[0074] Module M1: Establishing a new energy ship power network system model;

[0075] Module M2: Based on the established power network system model, a ship energy control process characterized by Markov is formed;

[0076] Module M3: According to the ship energy control process, the DQN algorithm is used to optimize the ship energy scheduling under the uncertainty of new energy.

[0077] Preferably, the module M1 comprises:

[0078] Module M1.1: Establishing uncertainty model of shipborne photovoltaic power generation;

[0079] The output of the shipboard photovoltaic power generation system is corrected by using the data along the ship's route and combining the correction method, as follows:

[0080]

[0081] Among them, P t PV represents the output power of the onboard photovoltaic power generation system at time t; η PV Represents the conversion efficiency of solar photovoltaic panels; S PV is the area of ​​photovoltaic panels (m 2 );I t Represents the sunlight intensity received at time t (W / m 2 ); and They represent the direct sunlight, scattered sunlight and ground reflection at time t respectively; θ is the angle between the photovoltaic panel and the light; φ is the angle between the photovoltaic panel and the horizontal plane; ε is the reflectivity; η PV_ref is the maximum conversion efficiency of solar photovoltaic panels; η MPPT is the maximum power tracking control conversion efficiency; T is the working temperature of the solar photovoltaic panel; T ref is the temperature of the solar photovoltaic panel when it is working normally; β is the temperature coefficient;

[0082] Module M1.2: Establish ship propulsion load model;

[0083]

[0084] Among them, P t Pro represents the propulsion load of the ship at time t; V t represents the speed of the electric ship at time t; c1 and c2 are the ship's own parameters related to the hull resistance, which are determined by the ship's own design characteristics;

[0085] Module M1.3: Establishing the charging and discharging power model of onboard energy storage;

[0086]

[0087] Among them, e t and e t-1 Respectively represent the charge state of the shipboard energy storage system at time t and the previous time; η c and η d They represent the charging and discharging efficiency of the energy storage system; S represents the capacity of the energy storage system; P t ESS represents the charging and discharging power of the energy storage system at time t, and ΔT represents the time interval;

[0088] Module M1.4: Establishing a network model of ship energy system;

[0089] P t G +P t PV +P t ESS =P t Ser +P t Pro

[0090] Where P t G , P t PV and P tESS Respectively represent the output power of the generator, photovoltaic system and energy storage system at time t; P t Ser P represents the ship life service load at time t; t Pro represents the ship propulsion load at time t;

[0091] The constraints of the new energy ship power system are as follows:

[0092] 1) Diesel generator active power limit:

[0093]

[0094] Among them, P t G represents the output power of the ship's diesel engine at time t; and Respectively represent the minimum and maximum output power of the diesel generator system;

[0095] 2) Energy storage system power limit:

[0096]

[0097] Where P t ESS Represents the charging and discharging power of the energy storage system at time t; and Respectively represent the lower and upper limits of the energy storage system power;

[0098] Preferably, the module M2 comprises:

[0099] Module M2.1: Design state space;

[0100] S t = {v t ,P t Ser ,P t PV ,e t ,t}

[0101] Among them, v t represents the ship's speed at time t; P t Ser represents the life service load at time t; P t PV is the output power of the photovoltaic panel at time t; e t is the state of charge of the energy storage system at time t;

[0102] Module M2.2: Designing action spaces;

[0103] The diesel generator load ratio factor is used as the decision variable, denoted as ratio t , which is defined as:

[0104] ratio t =P t G / P GN

[0105] A t ={ratio t}

[0106] Among them, A t represents the state space; P t G is the output power of the diesel generator at time t; P GN is the rated power of the diesel generator;

[0107] Module M2.3: Design reward function;

[0108] The reward function is segmented according to the state of charge of the energy storage system; the decision system makes a decision action A t , the charge state of the energy storage system is determined by e t becomes e t+1 , if e t+1 If the upper and lower limits of the state of charge are exceeded, e t and e t+1 If the trend is opposite to the expectation, a penalty needs to be imposed through the reward function, as shown below:

[0109] ①e t+1 <0:

[0110] r t =-9

[0111] ②0≤e t+1 <e Low :

[0112]

[0113] ③e High ≤e t+1 <1:

[0114]

[0115] ④e Low ≤e t+1 <e High :

[0116] r t =0.4e t+1 -0.6ratio t

[0117] ⑤e t+1 >1:

[0118] r t =-3

[0119] Among them, e t 、e t+1 is the charge state of the energy storage system at time t and the next time; e Low and e High are the lower and upper limits of the safe range of the state of charge respectively; |Δe max | represents the absolute value of the maximum change in the state of charge of the energy storage system within a period of time; ratio t is the load proportional factor of the generator; r t is the reward obtained by the agent at time t.

[0120] Preferably, the module M3 comprises:

[0121] Module M3.1: Estimating the value function based on the Q-learning algorithm;

[0122] Based on the idea of ​​temporal difference learning, the optimal action value function is estimated through the Q-learning algorithm. The temporal difference update formula is as follows:

[0123]

[0124] Among them, Q(S t ,A t ) is the action value function at time t, α is the learning rate, R t is the single-step reward obtained by the agent, γ is the discount factor, Q(S t+1 ,a) is the action value function of taking action a in the next state at time t+1;

[0125] Module M3.2: Establish real-time energy management strategy based on DQN algorithm;

[0126] The deep Q network algorithm is used to train the ship agent. Module M3.2 uses the temporal difference method to incrementally update the action value function Q(s,a); so that Q(s,a) and r+γmax a'∈A Q(s',a') is close, and the DQN algorithm update is to make Q ω (s,a) approximates r+γmax a'∈A Q ω (s',a'), where ω * are the parameters of the Q network;

[0127] For the four-tuple dataset {(s i ,a i,r i ,s′ i )} i＝1,…,N , the neural network learns the optimal value of the parameters by minimizing the loss function, so the Q network used to fit the action value function is designed as follows:

[0128]

[0129] Among them, N is the number of four-tuple data sets, Q ω (s i ,a i ) is the Q value of all actions a in each state s, r i For state s i The reward obtained under ω (s′ i ,a′) is the Q value of executing action a′ in state s′.

[0130] Preferably, the DQN algorithm introduces two sets of neural networks, namely the current network Q ω (s,a) and target network Among them, the current network is used to calculate Q in the loss function ω (s i ,a i ), the target network is used to calculate

[0131] The target network is regarded as a copy of the current network, and its network parameters are synchronized with the current network at regular intervals;

[0132] The DQN algorithm introduces an experience replay mechanism;

[0133] The four-tuple data obtained from each interactive sampling is stored in the experience replay pool, and random sampling is performed from the experience replay pool when updating the Q network.

[0134] Compared with the prior art, the present invention has the following beneficial effects:

[0135] 1. The present invention adopts the deep reinforcement learning (DQN) algorithm to intelligently control the ship energy system, quickly adapt to the frequent load fluctuations, adjust the production and consumption of electricity in real time, and ensure the energy stability of the ship under different working conditions;

[0136] 2. The present invention selects the ship speed as a state variable, so that the ship energy management intelligent decision-making system can track the changes in ship speed in real time, achieve dynamic and real-time decision-making, improve energy utilization efficiency, and ensure the comfort and benefits of the ship during navigation;

[0137] 3. Under uncertain lighting conditions, the present invention realizes intelligent management of photovoltaic power generation through DQN, and minimizes reliance on traditional fuel power generation according to changes in light levels, thereby reducing energy consumption of ships and achieving environmentally friendly and economical energy utilization;

[0138] 4. The present invention adopts a reinforcement learning method, which can learn in an unknown environment, adapt to different ship working conditions and maritime situations, and has the ability of autonomous learning and optimization to improve its performance;

[0139] 5. The present invention focuses on real-time coordinated optimization of energy storage and generators in the face of frequent load fluctuations and unpredictable photovoltaic power generation. While improving fuel economy, it can also maintain the charge state of the energy storage system in a safe and stable range, thereby avoiding frequent deep charging and discharging of the ship energy storage system and extending the life of the energy storage system.

[0140] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0141] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0142] Figure 1 This is the structure diagram of the power network for new energy ships;

[0143] Figure 2 It is a schematic diagram of the overall process of the present invention;

[0144] Figure 3 This is a flowchart of the algorithm based on reinforcement learning;

[0145] Figure 4 is the load change during navigation (unit: kW);

[0146] Figure 5 The output of each energy source of the shipboard new energy system based on reinforcement learning (unit: kW);

[0147] Figure 6 The state of charge change curve and charging and discharging conditions of the energy storage system (unit on the left: kW);

[0148] Figure 7 Comparison of generator output under uncertainty of new energy (unit: kW). DETAILED DESCRIPTION

[0149] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0150] The embodiment of the present invention provides a method for real-time optimization of ship energy under uncertainty of new energy. The structure of the new energy ship power system is as follows: Figure 1 As shown, it includes shipboard photovoltaic power generation system, diesel engine power generation system, energy storage system, propulsion load and living load.

[0151] The real-time optimization method for ship energy under new energy uncertainty provided by the present invention refers to Figure 2 As shown in the figure, the deep reinforcement learning method is used to combine the characteristics of the ship during navigation, the characteristics of new energy output and the intelligent algorithm, and the new energy ship power system is modeled and studied. The utilization mode of ship photovoltaic energy under the uncertainty of photovoltaic output is optimized, and the energy management problem of new energy ships under the uncertainty of photovoltaic output is solved. Specifically, it includes the following contents:

[0152] Step S1: Establish a new energy ship power network system model.

[0153] The step S1 comprises:

[0154] Step S1.1: Establishing an uncertain model of shipborne photovoltaic power generation;

[0155] The solar irradiance received by the ship varies with the ship's location, date, local time, and time zone when the ship is traveling. Therefore, it is necessary to use these data along the ship's route and combine them with a reasonable correction method to correct the output of the ship's photovoltaic power generation system, as follows:

[0156]

[0157] Among them, P t PV represents the output power of the onboard photovoltaic power generation system at time t; η PV Represents the conversion efficiency of solar photovoltaic panels; S PV is the area of ​​photovoltaic panels (m 2 );I t Represents the solar illuminance received at time t (W / m 2 ); and They represent the direct sunlight, scattered sunlight and ground reflection at time t respectively; θ is the angle between the photovoltaic panel and the light; φ is the angle between the photovoltaic panel and the horizontal plane; ε is the reflectivity; ηPV_ref is the maximum conversion efficiency of solar photovoltaic panels; η MPPT For maximum power tracking control conversion efficiency, the present invention takes 1; T is the temperature of the solar photovoltaic panel when it is working; T ref It is the temperature of the solar photovoltaic panel when it is working normally, generally 25℃; β is the temperature coefficient.

[0158] Step S1.2: Establishing a ship propulsion load model;

[0159]

[0160] Among them, P t Pro represents the propulsion load of the ship at time t; V t represents the speed of the electric ship at time t; c1 and c2 are the ship's own parameters related to the hull resistance and are determined by the ship's own design characteristics.

[0161] Step S1.3: Establishing a shipboard energy storage charging and discharging power model;

[0162]

[0163] Among them, e t and e t-1 Respectively represent the charge state of the shipboard energy storage system at time t and the previous time; η c and η d They represent the charging and discharging efficiency of the energy storage system; S represents the capacity of the energy storage system; P t ESS represents the charging and discharging power of the energy storage system at time t, and ΔT represents the time interval.

[0164] Step S1.4: Establishing a ship energy system network model;

[0165] P t G +P t PV +P t ESS =P t Ser +P t Pro

[0166] Where P t G , P t PV and P t ESS Respectively represent the output power of the generator, photovoltaic system and energy storage system at time t; P t Ser P represents the ship life service load at time t;t Pro represents the ship propulsion load at time t;

[0167] For a new energy ship power system, the constraints to be observed are as follows:

[0168] 1) Diesel generator active power limit:

[0169]

[0170] Among them, P t G represents the output power of the ship's diesel engine at time t; and Respectively represent the minimum and maximum output power of the diesel generator system;

[0171] 2) Energy storage system power limit:

[0172]

[0173] Where P t ESS Represents the charging and discharging power of the energy storage system at time t; and They represent the lower and upper limits of the energy storage system power respectively.

[0174] Step S2: Based on the established power grid system model, a ship energy regulation process represented by Markov is formed.

[0175] Ship energy regulation process characterized by Markov model:

[0176] The key to real-time energy control of new energy ships is to tailor a real-time energy management intelligent decision-making system with adaptability and good economy for ships. It is necessary to be able to coordinate and control the power flow of the ship's energy system in each period according to the ship's operating status, so as to minimize operating costs and improve economic and environmental benefits. The real-time energy control problem of all-electric ships can be regarded as a sequential decision-making problem in a finite time domain, which can be characterized by a Markov decision process. Its basic property is that the current state of the system is only related to the state and action of the previous moment.

[0177] In the Markov decision process, the agent is responsible for executing actions and interacting with its environment. The process of environmental state change is affected by both the random process of spontaneous changes in the environment and the action behavior of the agent.

[0178] The Markov decision process can be represented by a five-tuple (S, A, T, r, γ), where S represents the state space, A represents the action space, T represents the state transition function, T(s′|s, a) represents the probability of reaching state s′ after executing action a in state s, r is the reward function, which depends on state s and action a, and γ is the discount factor, which ranges from [0, 1). When γ is close to 1, the agent pays more attention to the long-term cumulative reward, and when it is close to 0, it pays more attention to the short-term reward. In a Markov decision process, starting from the state at time t until the terminal state, the sum of the decay of all rewards is called the return, as shown in the formula:

[0179]

[0180] Among them, G t is the sum of all reward decays, R t is the single-step reward obtained by the agent, γ is the discount factor, and k is the number of transitions from the start to the end of the state.

[0181] The probability distribution of the agent's action selection in the action space under a given state is called a strategy, usually represented by π. Suppose the agent takes action a in state s, and the subsequent actions still follow the strategy π. The expected return obtained at this time is defined as the action value function, denoted by Q π (s,a) means that the mathematical expression is:

[0182]

[0183] in, represents the expected return under strategy π, S t and A t They represent the state space and action space at time t, s and a are the ship state and the action performed by the agent at time t, respectively. is the sum of all reward decays from time t to the terminal state. The action value function under the optimal strategy is called the optimal action value function, denoted by Q * (s,a), which is related to the action-value function Q π The relationship between (s,a) is:

[0184]

[0185] Step S2.1: Design state space;

[0186] The selection of state variables needs to be able to characterize the state of the all-electric ship during navigation. During the navigation of the ship, the ship speed changes dynamically, and the change in ship speed further determines the demand for propulsion power. The ship's real-time energy management intelligent decision-making system needs to be able to track this dynamic change. For this reason, ship speed and acceleration are selected as state variables. In addition, since the demand for ship life service load is essentially random and highly uncertain, it is difficult to achieve accurate prediction. The ship's real-time energy management intelligent decision-making system can only accurately know the demand for life service load in the current period. For this reason, the power demand of life service load is selected as the state variable. In this paper, the energy source of the all-electric ship is considered to be composed of a diesel generator set and an energy storage system, where the energy storage system can both generate power and absorb the power of the diesel generator set. In order to maintain the state of charge of the energy storage system in a safe range, the state of charge is also selected as a state variable. In summary, the state space of this paper is defined as follows:

[0187] S t = {v t ,P t Ser ,P t PV ,e t ,t}

[0188] Among them, v t represents the ship's speed at time t; P t Ser represents the life service load at time t; P t PV is the output power of the photovoltaic panel at time t; e t is the state of charge of the energy storage system at time t.

[0189] Step S2.2: Design action space;

[0190] The action space represents the action behaviors that can be controlled by the intelligent decision-making system. In order to decide the power allocation between the energy sources of the electric ship, this paper proposes to use the diesel generator load ratio factor as the decision variable, denoted as ratio t , which is defined as:

[0191] ratio t =P t G / P GN

[0192] A t ={ratio t}

[0193] Among them, A t represents the state space; P t Gis the output power of the diesel generator at time t; P GN is the rated power of the diesel generator.

[0194] Step S2.3: Design reward function;

[0195] The reward function determines the performance indicators that the algorithm focuses on and is the yardstick for guiding the intelligent agent to achieve the optimal decision. In order to improve fuel efficiency, reduce operating costs, and reduce carbon emissions. The ship real-time energy management intelligent decision-making system needs to reduce the output of the diesel generator set as much as possible and keep the charge state of the energy storage system in a safe range. The present invention sets the reward function in sections according to the charge state of the energy storage system; the intelligent decision-making system makes a decision action A t , the charge state of the energy storage system is determined by e t becomes e t+1 , if e t+1 If the upper and lower limits of the state of charge are exceeded, e t and e t+1 If the trend is opposite to the expectation, a penalty needs to be imposed through the reward function, as shown below:

[0196] ①e t+1 <0:

[0197] r t =-9

[0198] ②0≤e t+1 <e Low :

[0199]

[0200] ③e High ≤e t+1 <1:

[0201]

[0202] ④e Low ≤e t+1 <e High :

[0203] r t =0.4e t+1 -0.6ratio t

[0204] ⑤e t+1 >1:

[0205] r t =-3

[0206] Among them, e t 、e t+1is the charge state of the energy storage system at time t and the next time; e Low and e High are the lower and upper limits of the safe range of the state of charge respectively; |Δe max | represents the absolute value of the maximum change in the state of charge of the energy storage system within a period of time; ratio t is the load proportional factor of the generator; r t is the reward obtained by the agent at time t.

[0207] Step S3: According to the ship energy control process, the ship energy scheduling under the uncertainty of new energy is optimized based on the DQN algorithm.

[0208] The step S3 specifically includes:

[0209] Step S3.1: Estimate the value function based on the Q-learning algorithm;

[0210] The core of deploying a ship real-time energy management intelligent decision-making system is to continuously improve and optimize the real-time energy management strategy. If the optimal action-value function can be solved, the optimal strategy can be easily determined. However, it is usually not feasible to directly solve the optimal action-value function in real situations. The classic reinforcement learning method Q-learning algorithm can estimate the optimal action-value function based on the idea of ​​temporal difference learning. Its temporal difference update formula is as follows:

[0211]

[0212] Among them, Q(S t ,A t ) is the action value function at time t, α is the learning rate, R t is the single-step reward obtained by the agent, γ is the discount factor, Q(S t+1 ,a) is the action value function of taking action a in the next state at time t+1;

[0213] Step S3.2: Establishing a real-time energy management strategy based on the DQN algorithm;

[0214] When the state and action of the environment are discrete, the Q-learning algorithm can maintain a table to store the Q value of each state-action pair, so as to easily update the Q value. However, the state quantities of the ship system and the marine environment in this article are mostly continuous, and the Q value update method in tabular form is not applicable. If we want to still obtain the action value function in a continuous environment, the idea of ​​function fitting provides us with a good solution. The deep reinforcement learning algorithm combines the advantages of deep learning and reinforcement learning. Through the powerful expression ability of neural networks, it can well fit the action value function and directly establish a mapping relationship from continuous state space to action. It can be applied to the energy management and optimization scheduling of all-electric ships. The present invention adopts the deep Q network (Deep Q Network, DQN) algorithm to train the ship intelligent body, that is, to train the real-time energy management intelligent decision-making system of electric ships. Step S3.2 incrementally updates the action value function Q(s,a) through the time difference method; the goal is to make Q(s,a) and r+γmax a'∈A Q(s',a') is as close as possible, and the goal of the DQN algorithm update is to make Q ω (s,a) approximates r+γmax a'∈A Q ω (s',a'), where ω * are the parameters of the Q network.

[0215] For the four-tuple dataset {(s i ,a i ,r i ,s′ i )} i＝1,…,N , the neural network learns the optimal value of the parameters by minimizing the loss function, so the Q network used to fit the action value function is designed as follows:

[0216]

[0217] Among them, N is the number of four-tuple data sets, Q ω (s i ,a i ) is the Q value of all actions a in each state s, r i For state s i The reward obtained under ω (s′ i ,a′) is the Q value of executing action a′ in state s′.

[0218] However, when the Q network representing the action value function updates its parameters, r i +γmax a′∈A Q ω (s′ i,a′) will also change due to the change of parameters, which may easily lead to instability in training. To solve this problem, the DQN algorithm introduces two sets of neural networks, namely the current network Q ω (s,a) and target network Among them, the current network is used to calculate Q in the loss function ω (s i ,a i ), the target network is used to calculate The target network can be regarded as a copy of the current network, and its network parameters are synchronized with the current network at regular intervals. Since the four-tuple data obtained by the Markov decision process itself does not meet the assumption of independent and identical distribution, in order to break the correlation between samples and avoid fitting the Q network to the most recently sampled data, the DQN algorithm also introduces an experience replay mechanism. The four-tuple data obtained from each interactive sampling is stored in the experience replay pool, and random sampling is performed from the experience replay pool when updating the Q network. In summary, the basic process of the DQN algorithm is as follows: Figure 3 shown.

[0219] The present invention also provides a real-time optimization system for ship energy under new energy uncertainty, which can be implemented by executing the process steps of the real-time optimization method for ship energy under new energy uncertainty, that is, those skilled in the art can understand the real-time optimization method for ship energy under new energy uncertainty as a preferred implementation of the real-time optimization system for ship energy under new energy uncertainty. The system specifically includes the following contents:

[0220] Module M1: Establish a new energy ship power network system model.

[0221] The module M1 includes:

[0222] Module M1.1: Establishing uncertainty model of shipborne photovoltaic power generation;

[0223] The solar irradiance received by the ship varies with the ship's location, date, local time, and time zone when the ship is traveling. Therefore, it is necessary to use these data along the ship's route and combine them with a reasonable correction method to correct the output of the ship's photovoltaic power generation system, as follows:

[0224]

[0225] Among them, P t PV represents the output power of the onboard photovoltaic power generation system at time t; η PV Represents the conversion efficiency of solar photovoltaic panels; S PV is the area of ​​photovoltaic panels (m 2 );I tRepresents the sunlight intensity received at time t (W / m 2 ); and They represent the direct sunlight, scattered sunlight and ground reflection at time t respectively; θ is the angle between the photovoltaic panel and the light; φ is the angle between the photovoltaic panel and the horizontal plane; ε is the reflectivity; η PV_ref is the maximum conversion efficiency of solar photovoltaic panels; η MPPT For maximum power tracking control conversion efficiency, the present invention takes 1; T is the temperature of the solar photovoltaic panel when it is working; T ref It is the temperature of the solar photovoltaic panel when it is working normally, generally 25℃; β is the temperature coefficient.

[0226] Module M1.2: Establish ship propulsion load model;

[0227]

[0228] Among them, P t Pro represents the propulsion load of the ship at time t; V t represents the speed of the electric ship at time t; c1 and c2 are the ship's own parameters related to the hull resistance and are determined by the ship's own design characteristics.

[0229] Module M1.3: Establishing the charging and discharging power model of onboard energy storage;

[0230]

[0231] Among them, e t and e t-1 Respectively represent the charge state of the shipboard energy storage system at time t and the previous time; η c and η d They represent the charging and discharging efficiency of the energy storage system; S represents the capacity of the energy storage system; P t ESS represents the charging and discharging power of the energy storage system at time t, and ΔT represents the time interval.

[0232] Module M1.4: Establishing a network model of ship energy system;

[0233] P t G +P t PV +P t ESS =P t Ser +P t Pro

[0234] Where P tG , P t PV and P t ESS Respectively represent the output power of the generator, photovoltaic system and energy storage system at time t; P t Ser P represents the ship life service load at time t; t Pro represents the ship propulsion load at time t;

[0235] For a new energy ship power system, the constraints to be observed are as follows:

[0236] 1) Diesel generator active power limit:

[0237]

[0238] Among them, P t G represents the output power of the ship's diesel engine at time t; and Respectively represent the minimum and maximum output power of the diesel generator system;

[0239] 2) Energy storage system power limit:

[0240]

[0241] Where P t ESS Represents the charging and discharging power of the energy storage system at time t; and They represent the lower and upper limits of the energy storage system power respectively.

[0242] Module M2: Based on the established power grid system model, a Markov-characterized ship energy regulation process is formed.

[0243] Ship energy regulation process characterized by Markov model:

[0244] The key to real-time energy control of new energy ships is to tailor a real-time energy management intelligent decision-making system with adaptability and good economy for ships. It is necessary to be able to coordinate and control the power flow of the ship's energy system in each period according to the ship's operating status, so as to minimize operating costs and improve economic and environmental benefits. The real-time energy control problem of all-electric ships can be regarded as a sequential decision-making problem in a finite time domain, which can be characterized by a Markov decision process. Its basic property is that the current state of the system is only related to the state and action of the previous moment.

[0245] In the Markov decision process, the agent is responsible for executing actions and interacting with its environment. The process of environmental state change is affected by both the random process of spontaneous changes in the environment and the action behavior of the agent.

[0246] The Markov decision process can be represented by a five-tuple (S, A, T, r, γ), where S represents the state space, A represents the action space, T represents the state transition function, T(s′|s, a) represents the probability of reaching state s′ after executing action a in state s, r is the reward function, which depends on state s and action a, and γ is the discount factor, which ranges from [0, 1). When γ is close to 1, the agent pays more attention to the long-term cumulative reward, and when it is close to 0, it pays more attention to the short-term reward. In a Markov decision process, starting from the state at time t until the terminal state, the sum of the decay of all rewards is called the return, as shown in the formula:

[0247]

[0248] Among them, G t is the sum of all reward decays, R t is the single-step reward obtained by the agent, γ is the discount factor, and k is the number of transitions from the start to the end of the state.

[0249] The probability distribution of the agent's action selection in the action space under a given state is called a strategy, usually represented by π. Suppose the agent takes action a in state s, and the subsequent actions still follow the strategy π. The expected return obtained at this time is defined as the action value function, denoted by Q π (s,a) means that the mathematical expression is:

[0250]

[0251] in, represents the expected return under strategy π, S t and A t They represent the state space and action space at time t, s and a are the ship state and the action performed by the agent at time t, respectively. is the sum of all reward decays from time t to the terminal state. The action value function under the optimal strategy is called the optimal action value function, denoted by Q * (s,a), which is related to the action-value function Q π The relationship between (s,a) is:

[0252]

[0253] Module M2.1: Design state space;

[0254] The selection of state variables needs to be able to characterize the state of the all-electric ship during navigation. During the navigation of the ship, the ship speed changes dynamically, and the change in ship speed further determines the demand for propulsion power. The ship's real-time energy management intelligent decision-making system needs to be able to track this dynamic change. For this reason, ship speed and acceleration are selected as state variables. In addition, since the demand for ship life service load is essentially random and highly uncertain, it is difficult to achieve accurate prediction. The ship's real-time energy management intelligent decision-making system can only accurately know the demand for life service load in the current period. For this reason, the power demand of life service load is selected as the state variable. In this paper, the energy source of the all-electric ship is considered to be composed of a diesel generator set and an energy storage system, where the energy storage system can both generate power and absorb the power of the diesel generator set. In order to maintain the state of charge of the energy storage system in a safe range, the state of charge is also selected as a state variable. In summary, the state space of this paper is defined as follows:

[0255] S t = {v t ,P t Ser ,P t PV ,e t ,t}

[0256] Among them, v t represents the ship's speed at time t; P t Ser represents the life service load at time t; P t PV is the output power of the photovoltaic panel at time t; e t is the state of charge of the energy storage system at time t.

[0257] Module M2.2: Designing action spaces;

[0258] The action space represents the action behaviors that can be controlled by the intelligent decision-making system. In order to decide the power allocation between the energy sources of the electric ship, this paper proposes to use the diesel generator load ratio factor as the decision variable, denoted as ratio t , which is defined as:

[0259] ratio t =P t G / P GN

[0260] A t ={ratio t}

[0261] Among them, A t represents the state space; P t Gis the output power of the diesel generator at time t; P GN is the rated power of the diesel generator.

[0262] Module M2.3: Design reward function;

[0263] The reward function determines the performance indicators that the algorithm focuses on and is the yardstick for guiding the intelligent agent to achieve the optimal decision. In order to improve fuel efficiency, reduce operating costs, and reduce carbon emissions. The ship real-time energy management intelligent decision-making system needs to reduce the output of the diesel generator set as much as possible and keep the charge state of the energy storage system in a safe range. The present invention sets the reward function in sections according to the charge state of the energy storage system; the intelligent decision-making system makes a decision action A t , the charge state of the energy storage system is determined by e t becomes e t+1 , if e t+1 If the upper and lower limits of the state of charge are exceeded, e t and e t+1 If the trend is opposite to the expectation, a penalty needs to be imposed through the reward function, as shown below:

[0264] ①e t+1 <0:

[0265] r t =-9

[0266] ②0≤e t+1 <e Low :

[0267]

[0268] ③e High ≤e t+1 <1:

[0269]

[0270] ④e Low ≤e t+1 <e High :

[0271] r t =0.4e t+1 -0.6ratio t

[0272] ⑤e t+1 >1:

[0273] r t =-3

[0274] Among them, e t 、e t+1is the charge state of the energy storage system at time t and the next time; e Low and e High are the lower and upper limits of the safe range of the state of charge respectively; |Δe max | represents the absolute value of the maximum change in the state of charge of the energy storage system within a period of time; ratio t is the load proportional factor of the generator; r t is the reward obtained by the agent at time t.

[0275] Module M3: According to the ship energy control process, the DQN algorithm is used to optimize the ship energy scheduling under the uncertainty of new energy.

[0276] The module M3 specifically includes:

[0277] Module M3.1: Estimating the value function based on the Q-learning algorithm;

[0278] The core of deploying a ship real-time energy management intelligent decision-making system is to continuously improve and optimize the real-time energy management strategy. If the optimal action-value function can be solved, the optimal strategy can be easily determined. However, it is usually not feasible to directly solve the optimal action-value function in real situations. The classic reinforcement learning method Q-learning algorithm can estimate the optimal action-value function based on the idea of ​​temporal difference learning. Its temporal difference update formula is as follows:

[0279]

[0280] Among them, Q(S t ,A t ) is the action value function at time t, α is the learning rate, R t is the single-step reward obtained by the agent, γ is the discount factor, Q(S t+1 ,a) is the action value function of taking action a in the next state at time t+1;

[0281] Module M3.2: Establish real-time energy management strategy based on DQN algorithm;

[0282] When the state and action of the environment are discrete, the Q-learning algorithm can maintain a table to store the Q value of each state-action pair, so as to easily update the Q value. However, the state quantities of the ship system and the marine environment in this article are mostly continuous, and the Q value update method in tabular form is not applicable. If we want to still obtain the action value function in a continuous environment, the idea of ​​function fitting provides us with a good solution. The deep reinforcement learning algorithm combines the advantages of deep learning and reinforcement learning. Through the powerful expression ability of neural networks, it can well fit the action value function and directly establish a mapping relationship from continuous state space to action. It can be applied to the energy management and optimization scheduling of all-electric ships. The present invention adopts the deep Q network (Deep Q Network, DQN) algorithm to train the ship intelligent body, that is, to train the real-time energy management intelligent decision-making system of electric ships. Module M3.2 incrementally updates the action value function Q(s,a) through the time difference method; the goal is to make Q(s,a) and r+γmax a'∈A Q(s',a') is as close as possible, and the goal of the DQN algorithm update is to make Q ω (s,a) approximates r+γmax a'∈A Q ω (s',a'), where ω * are the parameters of the Q network.

[0283] For the four-tuple dataset {(s i ,a i ,r i ,s′ i )} i＝1,…,N , the neural network learns the optimal value of the parameters by minimizing the loss function, so the Q network used to fit the action value function is designed as follows:

[0284]

[0285] Among them, N is the number of four-tuple data sets, Q ω (s i ,a i ) is the Q value of all actions a in each state s, r i For state s i The reward obtained under ω (s′ i ,a′) is the Q value of executing action a′ in state s′.

[0286] However, when the Q network representing the action value function updates its parameters, r i +γmax a′∈A Q ω (s′ i,a′) will also change due to the change of parameters, which may easily lead to instability in training. To solve this problem, the DQN algorithm introduces two sets of neural networks, namely the current network Q ω (s,a) and target network Among them, the current network is used to calculate Q in the loss function ω (s i ,a i ), the target network is used to calculate The target network can be regarded as a copy of the current network, and its network parameters are synchronized with the current network at regular intervals. Since the four-tuple data obtained by the Markov decision process itself does not meet the assumption of independent and identical distribution, in order to break the correlation between samples and avoid fitting the Q network to the most recently sampled data, the DQN algorithm also introduces an experience replay mechanism. The four-tuple data obtained from each interactive sampling is stored in the experience replay pool, and random sampling is performed from the experience replay pool when updating the Q network. In summary, the basic process of the DQN algorithm is as follows: Figure 3 shown.

[0287] Next, the present invention is described in more detail by using examples.

[0288] This scheme uses actual ship data as an example, constructs the DQN algorithm according to the parameters in Table 1, and trains the ship agent under various lighting conditions through 5000 Markov decision sequences. The trained ship real-time energy management intelligent decision system is tested on the test scene. The test results are as follows: Figure 4 , Figure 5 , Figure 6 and Figure 7 shown.

[0289] Table 1 Q network and hyperparameter design

[0290] parameter Numeric parameter Numeric Learning Rate 0.002 Neuron activation function ReLU Discount Factor 0.98 Number of hidden layers 1 Experience replay pool capacity 10000 Number of neurons in each hidden layer 128 Number of mini-batches 64 Target network update cycle 10

[0291] It can be seen from the result graph that in the online decision-making process, the trained ship real-time energy management intelligent decision-making system can always maintain the energy storage system charge state in the safe range of 0.25 to 0.85 under different lighting conditions, and from 12:00 to 14:00 when the light intensity is strong, the energy storage system and the photovoltaic system can cooperate with each other to provide sufficient power to support the load, and from 21:00 to 0:00 when the light intensity is weak, the generator can cooperate with the energy storage to achieve load support. In addition, the existence of the photovoltaic system reduces the power output of the generator by 2000kW during the entire voyage, which reduces fuel by about 530L, greatly reducing the energy consumption of the generator and improving the economy. The specific comparison results are shown in Table 2.

[0292] Table 2 Comparison of ship system operation with and without photovoltaics

[0293]

[0294] The embodiment of the present invention provides a method and system for real-time optimization of ship energy under the uncertainty of new energy. In order to overcome the uncertainty of photovoltaic output and efficiently utilize the beneficial effects brought by new energy, according to different lighting conditions and under the condition of frequent load fluctuations, the flow of energy between energy storage, generator and photovoltaic system is reasonably optimized, and the energy consumption of generator is greatly reduced under the premise of ensuring that the state of charge of the energy storage system is stable in a safe range, so as to realize economical and reliable operation of the ship. This solution has the following characteristics:

[0295] A mobile energy network structure model for ships during navigation is established, which takes into account the uncertainty of the output of the shipboard photovoltaic system, which is conducive to reducing the energy consumption during the navigation and improving the economy of ship operation.

[0296] The energy management process during the ship's navigation is based on the Markov decision process, so that the energy regulation of the ship at the next moment is only related to the energy distribution status of the ship at this moment. The network structure of the future ship energy system does not depend on the energy flow in the previous power network.

[0297] Based on the reinforcement learning method, a real-time energy control algorithm framework was established during the navigation of ships under the uncertainty of new energy output. Through intelligent agent learning, strategies to adapt to different photovoltaic output fluctuations were learned, thereby achieving more effective ship energy regulation.

[0298] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0299] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A real-time optimization method for ship energy under new energy uncertainty, It is characterized in that include: Step S1: Establish a new energy ship power network system model; Step S2: According to the established power network system model, a ship energy control process represented by Markov is formed; Step S3: According to the ship energy control process, the ship energy scheduling under the uncertainty of new energy is optimized based on the DQN algorithm.

2. The method for real-time optimization of ship energy under new energy uncertainty according to claim 1, It is characterized in that The step S1 comprises: Step S1.1: Establishing an uncertain model of shipborne photovoltaic power generation; The output of the shipboard photovoltaic power generation system is corrected by using the data along the ship's route and combining the correction method, as follows: Among them, P t PV represents the output power of the onboard photovoltaic power generation system at time t; η PV Represents the conversion efficiency of solar photovoltaic panels; S PV is the area of ​​photovoltaic panels (m 2 );I t Represents the sunlight intensity received at time t (W / m 2 ); and They represent the direct sunlight, scattered sunlight and ground reflection at time t respectively; θ is the angle between the photovoltaic panel and the light; φ is the angle between the photovoltaic panel and the horizontal plane; ε is the reflectivity; η PV_ref is the maximum conversion efficiency of solar photovoltaic panels; η MPPT is the maximum power tracking control conversion efficiency; T is the working temperature of the solar photovoltaic panel; T ref is the temperature of the solar photovoltaic panel when it is working normally; β is the temperature coefficient; Step S1.2: Establishing a ship propulsion load model; Among them, P t Pro represents the propulsion load of the ship at time t; V t represents the speed of the electric ship at time t; c 1 and c 2 It is a ship parameter related to hull resistance and is determined by the ship's own design characteristics; Step S1.3: Establishing a shipboard energy storage charging and discharging power model; Among them, e t and e t-1 Respectively represent the charge state of the shipboard energy storage system at time t and the previous time; η c and η d They represent the charging and discharging efficiency of the energy storage system; S represents the capacity of the energy storage system; P t ESS represents the charging and discharging power of the energy storage system at time t, and ΔT represents the time interval; Step S1.4: Establishing a ship energy system network model; P t G +P t PV +P t ESS =P t Ser +P t Pro Where P t G , P t PV and P t ESS Respectively represent the output power of the generator, photovoltaic system and energy storage system at time t; P t Ser P represents the ship life service load at time t; t Pro represents the ship propulsion load at time t; The constraints of the new energy ship power system are as follows: 1) Diesel generator active power limit: Among them, P t G represents the output power of the ship's diesel engine at time t; and Respectively represent the minimum and maximum output power of the diesel generator system; 2) Energy storage system power limit: Where P t ESS Represents the charging and discharging power of the energy storage system at time t; and They represent the lower and upper limits of the energy storage system power respectively.

3. The method for real-time optimization of ship energy under new energy uncertainty according to claim 1, It is characterized in that The step S2 comprises: Step S2.1: Design state space; S t ={v t ,P t Ser ,P t PV ,e t ,t} Among them, v t represents the ship's speed at time t; P t Ser represents the life service load at time t; P t PV is the output power of the photovoltaic panel at time t; e t is the state of charge of the energy storage system at time t; Step S2.2: Design action space; The diesel generator load ratio factor is used as the decision variable, denoted as ratio t , which is defined as: reason t =P t G / P GN A t ={ratio t } Among them, A t represents the state space; P t G is the output power of the diesel generator at time t; P GN is the rated power of the diesel generator; Step S2.3: Design reward function; The reward function is segmented according to the state of charge of the energy storage system; the decision system makes a decision action A t , the charge state of the energy storage system is determined by e t becomes e t+1 , if e t+1 If the upper and lower limits of the state of charge are exceeded, e t and e t+1 If the trend is opposite to the expectation, a penalty needs to be imposed through the reward function, as shown below: ①e t+1 <0： r t ＝-9 ②0≤e t+1 <e Low : ③e High ≤e t+1 <1: ④e Low ≤e t+1 <e High : r t =0.4e t+1 -0.6ratio t ⑤e t+1 >1： r t ＝-3 Among them, e t 、e t+1 is the charge state of the energy storage system at time t and the next time; e Low and e High are the lower and upper limits of the safe range of the state of charge respectively; |Δe max | represents the absolute value of the maximum change in the state of charge of the energy storage system within a period of time; ratio t is the load proportional factor of the generator; r t is the reward obtained by the agent at time t.

4. The method for real-time optimization of ship energy under new energy uncertainty according to claim 1, It is characterized in that The step S3 comprises: Step S3.1: Estimate the value function based on the Q-learning algorithm; Based on the idea of ​​temporal difference learning, the optimal action value function is estimated through the Q-learning algorithm. The temporal difference update formula is as follows: Among them, Q(S t ,A t ) is the action value function at time t, α is the learning rate, R t is the single-step reward obtained by the agent, γ is the discount factor, Q(S t+1 ,a) is the action value function of taking action a in the next state at time t+1; Step S3.2: Establishing a real-time energy management strategy based on the DQN algorithm; The deep Q network algorithm is used to train the ship agent. Step S3.2 uses the temporal difference method to incrementally update the action value function Q(s,a); so that Q(s,a) and r+γmax a'∈A Q(s',a') is close, and the DQN algorithm update is to make Q ω (s,a) approximates r+γmax a'∈A Q ω (s',a'), where ω * are the parameters of the Q network; For the four-tuple dataset {(s i ,a i ,r i ,s i ′)} i＝1,…,N , the neural network learns the optimal value of the parameters by minimizing the loss function, so the Q network used to fit the action value function is designed as follows: Among them, N is the number of four-tuple data sets, Q ω (s i ,a i ) is the Q value of all actions a in each state s, r i For state s i The reward obtained under ω (s i ′,a′) is the Q value of executing action a′ in state s′.

5. The method for real-time optimization of ship energy under new energy uncertainty according to claim 1, It is characterized in that The DQN algorithm introduces two sets of neural networks, namely the current network and target network Among them, the current network is used to calculate Q in the loss function ω (s i ,a i ), the target network is used to calculate The target network is regarded as a copy of the current network, and its network parameters are synchronized with the current network at regular intervals.

6. The method for real-time optimization of ship energy under new energy uncertainty according to claim 1, It is characterized in that The DQN algorithm introduces an experience replay mechanism; The four-tuple data obtained from each interactive sampling is stored in the experience replay pool, and random sampling is performed from the experience replay pool when updating the Q network.

7. A real-time optimization system for ship energy under new energy uncertainty, It is characterized in that include: Module M1: Establishing a new energy ship power network system model; Module M2: Based on the established power network system model, a ship energy control process characterized by Markov is formed; Module M3: According to the ship energy control process, the DQN algorithm is used to optimize the ship energy scheduling under the uncertainty of new energy.

8. The ship energy real-time optimization system under new energy uncertainty according to claim 7, It is characterized in that The module M1 comprises: Module M1.1: Establishing uncertainty model of shipborne photovoltaic power generation; The output of the shipboard photovoltaic power generation system is corrected by using the data along the ship's route and combining the correction method, as follows: Among them, P t PV represents the output power of the onboard photovoltaic power generation system at time t; η PV Represents the conversion efficiency of solar photovoltaic panels; S PV is the area of ​​photovoltaic panels (m 2 );I t Represents the sunlight intensity received at time t (W / m 2 ); and They represent the direct sunlight, scattered sunlight and ground reflection at time t respectively; θ is the angle between the photovoltaic panel and the light; φ is the angle between the photovoltaic panel and the horizontal plane; ε is the reflectivity; η PV_ref is the maximum conversion efficiency of solar photovoltaic panels; η MPPT is the maximum power tracking control conversion efficiency; T is the working temperature of the solar photovoltaic panel; T ref is the temperature of the solar photovoltaic panel when it is working normally; β is the temperature coefficient; Module M1.2: Establish ship propulsion load model; Among them, P t Pro represents the propulsion load of the ship at time t; V t represents the speed of the electric ship at time t; c 1 and c 2 It is a ship parameter related to hull resistance and is determined by the ship's own design characteristics; Module M1.3: Establishing the charging and discharging power model of onboard energy storage; Among them, e t and e t-1 Respectively represent the charge state of the shipboard energy storage system at time t and the previous time; η c and η d They represent the charging and discharging efficiency of the energy storage system; S represents the capacity of the energy storage system; P t ESS represents the charging and discharging power of the energy storage system at time t, and ΔT represents the time interval; Module M1.4: Establishing a network model of ship energy system; P t G +P t PV +P t ESS =P t Ser +P t Pro Where P t G , P t PV and P t ESS Respectively represent the output power of the generator, photovoltaic system and energy storage system at time t; P t Ser P represents the ship life service load at time t; t Pro represents the ship propulsion load at time t; The constraints of the new energy ship power system are as follows: 1) Diesel generator active power limit: Among them, P t G represents the output power of the ship's diesel engine at time t; and Respectively represent the minimum and maximum output power of the diesel generator system; 2) Energy storage system power limit: Where P t ESS Represents the charging and discharging power of the energy storage system at time t; and Respectively represent the lower and upper limits of the energy storage system power; The module M2 comprises: Module M2.1: Design state space; S t ={v t ,P t Ser ,P t PV ,e t ,t} Among them, v t represents the ship's speed at time t; P t Ser represents the life service load at time t; P t PV is the output power of the photovoltaic panel at time t; e t is the state of charge of the energy storage system at time t; Module M2.2: Designing action spaces; The diesel generator load ratio factor is used as the decision variable, denoted as ratio t , which is defined as: reason t =P t G / P GN A t ={ratio t } Among them, A t represents the state space; P t G is the output power of the diesel generator at time t; P GN is the rated power of the diesel generator; Module M2.3: Design reward function; The reward function is segmented according to the state of charge of the energy storage system; the decision system makes a decision action A t , the charge state of the energy storage system is determined by e t becomes e t+1 , if e t+1 If the upper and lower limits of the state of charge are exceeded, e t and e t+1 If the trend is opposite to the expectation, a penalty needs to be imposed through the reward function, as shown below: ①e t+1 <0： r t ＝-9 ②0≤e t+1 <e Low : ③e High ≤e t+1 <1: ④e Low ≤e t+1 <e High : r t =0.4e t+1 -0.6ratio t ⑤e t+1 >1： r t ＝-3 Among them, e t 、e t+1 is the charge state of the energy storage system at time t and the next time; e Low and e High are the lower and upper limits of the safe range of the state of charge respectively; |Δe max | represents the absolute value of the maximum change in the state of charge of the energy storage system within a period of time; ratio t is the load proportional factor of the generator; r t is the reward obtained by the agent at time t.

9. The ship energy real-time optimization system under new energy uncertainty according to claim 7, It is characterized in that The module M3 comprises: Module M3.1: Estimating the value function based on the Q-learning algorithm; Based on the idea of ​​temporal difference learning, the optimal action value function is estimated through the Q-learning algorithm. The temporal difference update formula is as follows: Among them, Q(S t ,A t ) is the action value function at time t, α is the learning rate, R t is the single-step reward obtained by the agent, γ is the discount factor, Q(S t+1 ,a) is the action value function of taking action a in the next state at time t+1; Module M3.2: Establish real-time energy management strategy based on DQN algorithm; The deep Q network algorithm is used to train the ship agent. Module M3.2 uses the temporal difference method to incrementally update the action value function Q(s,a); so that Q(s,a) and r+γmax a'∈A Q(s',a') is close, and the DQN algorithm update is to make Q ω (s,a) approximates r+γmax a'∈A Q ω (s',a'), where ω * are the parameters of the Q network; For the four-tuple dataset {(s i ,a i ,r i ,s′ i )} i＝1,…,N , the neural network learns the optimal value of the parameters by minimizing the loss function, so the Q network used to fit the action value function is designed as follows: Among them, N is the number of four-tuple data sets, Q ω (s i ,a i ) is the Q value of all actions a in each state s, r i For state s i The reward obtained under ω (s i ′,a′) is the Q value of executing action a′ in state s′.

10. The ship energy real-time optimization system under new energy uncertainty according to claim 6, It is characterized in that The DQN algorithm introduces two sets of neural networks, namely the current network Q ω (s,a) and target network Among them, the current network is used to calculate Q in the loss function ω (s i ,a i ), the target network is used to calculate The target network is regarded as a copy of the current network, and its network parameters are synchronized with the current network at regular intervals; The DQN algorithm introduces an experience replay mechanism; The four-tuple data obtained from each interactive sampling is stored in the experience replay pool, and random sampling is performed from the experience replay pool when updating the Q network.

Images