Method for constructing and verifying ship energy management strategy based on deep reinforcement learning

Abstract

The application discloses a kind of ship energy management strategy construction and verification method based on deep reinforcement learning, comprising: collecting the running data in certain navigation of ship and pre-processing;According to the running data, the ship energy management strategy model based on deep reinforcement learning is constructed;Ship energy management strategy model is trained, the performance of the model is evaluated and optimized;The actual working condition of ship is simulated through hybrid power ship semi-physical simulation platform;Ship energy management strategy model carries out deep reinforcement learning, and the energy distribution strategy and corresponding control hybrid power ship semi-physical simulation platform action instruction are output;The hybrid power ship semi-physical simulation platform adjusted and controlled according to energy distribution strategy is continuously data-collected, and the data collected is used as feedback data;According to feedback data, the ship energy management strategy model is optimized and adjusted.The application can learn and adapt from actual running data, with better self-adaptive ability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of ship energy management, and particularly relates to a ship energy management strategy construction and verification method based on deep reinforcement learning. BACKGROUND

[0002] Shipping industry is one of the important pillars of world trade and transportation, and also one of the main sources of air pollution and greenhouse gas emissions. According to the statistics of the International Maritime Organization (IMO), in 2018, the carbon dioxide emissions of ships accounted for 2.89% of the total global emissions, and it is expected to increase to 17% by 2050. In order to cope with the increasingly severe environmental problems and energy crisis, the shipping industry is facing great pressure and challenges of energy saving and emission reduction.

[0003] In order to improve the energy efficiency of ships and reduce the emissions of ships, hybrid power systems have become an important direction for the development of future ship power. Ship hybrid power system refers to the combination of two or more power sources (such as internal combustion engine, battery, fuel cell, etc.) in different ways (such as series, parallel and hybrid, etc.), through energy conversion, storage and distribution, etc., to meet the power demand of ships. Compared with the traditional single power source, the ship hybrid power system has the advantages of high output power, energy recovery and storage, improvement of ship safety and reliability, reduction of ship noise and vibration, etc. How to effectively manage the energy flow between multiple energy sources of hybrid power system to meet the power demand under different working conditions, while optimizing the performance indicators (such as fuel consumption, emissions, etc.) of ship power system, involves how to develop a reasonable energy management strategy or decision rule to guide the switching state and power distribution of each energy source.

[0004] The existing hybrid power system energy management strategies mainly include the following three types:

[0005] (1) Rule-based strategy, which relies on the in-depth understanding of ship power system by engineers and rich practical experience. This kind of strategy usually controls the switching state and power distribution of energy sources through pre-set logical threshold values.

[0006] (2) Optimization-based strategy, which can be divided into global optimization and real-time optimization, and uses advanced mathematical algorithms to find the optimal energy distribution scheme. These strategies can overcome the shortcomings of rule-based strategies to a certain extent, provide more accurate and flexible control, and theoretically achieve higher energy efficiency and lower emissions.

[0007] (3) Learning-based strategy, which does not require complete system model and prediction information, and has the ability of online learning and adaptive adjustment, but needs a large amount of data and training time.

[0008] In recent years, deep reinforcement learning (DRL) has shown its powerful learning and optimization capabilities in multiple fields as an emerging artificial intelligence technology. DRL enables intelligent agents to learn optimal strategies through interaction with the environment, and can find optimal solutions through trial and error and self-improvement without fully understanding the environment model. In the field of ship energy management, DRL has the potential to learn complex energy distribution strategies to achieve real-time and adaptive control of ship hybrid power systems, thereby optimizing the energy efficiency of ships and reducing emissions.

[0009] Currently, the existing energy management strategies based on deep reinforcement learning use a data-driven construction method. The data-driven method focuses on learning and mapping using a large amount of data to establish the relationship between input and output, rather than relying on explicit rules or patterns. The advantage is that it can handle complex nonlinear relationships and learn potential rules from historical data. The verification method used by the strategy focuses on model software simulation verification. Although this method has fast verification speed and low experimental cost, it may not fully simulate the complexity of the real world, resulting in limited credibility and flexibility of the simulation results. In addition, the existing energy management strategies based on deep reinforcement learning rarely use a complete strategy construction and verification system. Therefore, how to construct and verify the ship energy management strategy based on deep reinforcement learning has become a problem to be solved. SUMMARY

[0010] The purpose of the present application is to provide a ship energy management strategy construction and verification method based on deep reinforcement learning, which can learn and adapt from actual operation data, has better adaptive ability, is more flexible in handling complex systems and nonlinear problems, and can handle diversified operating conditions and working conditions, and has better generalization ability.

[0011] To solve the above technical problems, the technical scheme of the present application is: a ship energy management strategy construction and verification method based on deep reinforcement learning, applied to a ship energy management strategy construction and verification system based on deep reinforcement learning. The main body of the system is a hybrid power ship semi-physical simulation platform, which at least includes a simulated hybrid power device and a measurement and control system. The simulated hybrid power device is used to simulate ship loads and actual ship working conditions. The measurement and control system is used for data acquisition, and the ship energy management strategy construction and verification method based on deep reinforcement learning is stored on it. The method includes the following steps:

[0012] Collecting operation data in a certain segment of ship navigation and preprocessing, dividing the preprocessed data into a training set and a validation set according to a predetermined proportion, and randomly selecting data from the training set to generate a test set;

[0013] A ship energy management strategy model based on deep reinforcement learning is constructed according to operation data;

[0014] The ship energy management strategy model is trained through data of a training set, and performance of the model is evaluated and optimized through data of a verification set and a test set;

[0015] Actual working conditions of the ship are simulated through a hybrid power ship semi-physical simulation platform, simulation data are collected and used as input of the ship energy management strategy model;

[0016] The ship energy management strategy model performs deep reinforcement learning according to the simulation data, and outputs an energy distribution strategy and corresponding action instructions for controlling the hybrid power ship semi-physical simulation platform;

[0017] The hybrid power ship semi-physical simulation platform controlled according to the energy distribution strategy is continuously data-collected, and the collected data are used as feedback data;

[0018] The ship energy management strategy model is adjusted and optimized according to the feedback data, strategy parameters are dynamically adjusted through deep reinforcement learning, and a dynamically adjusted ship energy management strategy model is generated to adapt to different operation conditions and actual working conditions of the actual ship.

[0019] The operation data at least include ship speed, heading, wave condition, wind speed, wind direction, engine speed, engine output power, battery state and fuel consumption.

[0020] The preprocessing specifically includes data cleaning, feature selection and data conversion of the operation data, missing values in the operation data are filled through a K-nearest neighbor interpolation algorithm, and abnormal values in the operation data are normalized through Z-score.

[0021] The construction method of the ship energy management strategy model is that state space definition and action space definition are performed on a state space of deep reinforcement learning and all possible actions executable by an agent in the ship energy management strategy model, and a reward function for evaluating immediate feedback obtained after the agent performs an action is defined.

[0022] The method for defining the state space specifically includes:

[0023] demand power P demand of the ship in the whole working condition, engine output power P out , battery state of charge SOC, speed V, wind speed U, course over ground COG, wave condition and wind direction F are used as the state space S of deep reinforcement learning, and are expressed as:

[0024] S={P demand ,SOC,P outV, U, COG, W, F

[0025] The method for defining the action space is specifically represented as:

[0026] A = {IEP, DEP, CB, DB}

[0027] Wherein, A is the action space, IEP is increasing engine power output, DEP is reducing engine power output, CB is adjusting battery charging state, and DB is adjusting battery discharging state.

[0028] The rewards set in the reward function at least include a combustion efficiency reward for representing an inverse ratio of fuel consumption, a fluctuation reward for representing an inverse ratio of power fluctuation, an energy storage efficiency reward for representing a positive ratio of charging and discharging efficiency of supercapacitor and battery, and an environmental impact reward for representing an inverse ratio of emission; the reward function is represented as:

[0029] R(S, A) = a * FuelEff_reward + b * Stab_reward + g * ESS_reward + d * Env_reward

[0030] Wherein, R(S, A) is the reward function based on the state space definition and the action space definition, a, b, g and d are weight coefficients, FuelEff_reward is the fuel efficiency reward, Stab_reward is the fluctuation reward, ESS_reward is the energy storage efficiency reward, and Env_reward is the environmental impact reward.

[0031] The selected deep learning reinforcement algorithm is one or more of DQN, DDPG, A3C and PPO algorithms.

[0032] A ship energy management strategy construction and verification system based on deep reinforcement learning is also provided, the main body of the system is a hybrid power ship semi-physical simulation platform, which at least includes a simulated hybrid power device and a measurement and control system, wherein,

[0033] The simulated hybrid power device is used to simulate ship load and actual working conditions of the ship;

[0034] The measurement and control system is used for data acquisition, and comprises at least a rotating speed and torque sensor, a battery charge coefficient collector, a voltage and current sensor, a dSPACE real-time simulator, an information acquisition board and a drive board; wherein the rotating speed and torque sensor is used for real-time monitoring and collecting rotating speed and torque data of the analog ship main engine and shaft motor; the dSPACE real-time simulator is used for receiving signals from the rotating speed and torque sensor, the battery charge coefficient collector and the voltage and current sensor, and performing analog-digital conversion, processing operation and power distribution control, and has stored thereon the ship energy management strategy construction and verification method based on deep reinforcement learning according to any one of the preceding aspects, and generates a dynamically adjusted ship energy management strategy model based on the method to adapt to different operating conditions and actual working conditions of an actual ship.

[0035] The analog hybrid power device comprises at least two motors, a gear reduction box, a clutch, a magnetic powder brake, a super capacitor, a storage battery and a control device; wherein one of the two motors simulates a ship main engine and the other simulates a shaft motor; the control device comprises an AC / DC and DC / DC converter and a DC / AC frequency converter, and is used for changing the power output of the simulated shaft motor by adjusting the input voltage and current thereof; the magnetic powder brake is used for realizing ship load and simulating actual working conditions of the ship.

[0036] Also provided is a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the method according to any one of the preceding aspects.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] The present application constructs a strategy by data driving, and verifies the strategy by a semi-physical simulation platform, and the data driving can learn and adapt from actual operation data, has better self-adaptive ability, is more flexible in processing complex systems and nonlinear problems, and can process diversified operating conditions and working conditions, and has better generalization ability; compared with model-in-loop simulation verification, the semi-physical simulation platform verification can simulate the operating conditions of the ship in a real physical environment, more truly reflects the performance of the energy management strategy in actual application, and can combine real hardware (such as an engine, a generator, a battery, etc.) with the energy management strategy for testing, provides a hardware-in-loop (HIL) test environment, and can more accurately evaluate the energy management strategy and system integration. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The figure is a flowchart of an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0041] The technical scheme of the present application is: a ship energy management strategy construction and verification method based on deep reinforcement learning, which adopts data-driven strategy construction and semi-physical simulation platform verification. Compared with mechanism model, data-driven can learn and adapt from actual operation data, has better self-adaptive ability, is more flexible in dealing with complex systems and nonlinear problems, and can handle diversified operation conditions and working conditions, has better generalization ability. Compared with model-in-loop simulation verification, semi-physical simulation platform verification can simulate the operation condition of the ship in the real physical environment, more truly reflect the performance of the energy management strategy in the actual application. And it can combine real hardware (such as engine, generator, battery, etc.) with energy management strategy for testing, provide hardware-in-loop (HIL) test environment, more accurately evaluate energy management strategy and system integration. The present application is based on diesel-electric hybrid ship, including super capacitor, storage battery and diesel generator set.

[0042] In order to solve the above problems, the present application mainly includes the following steps:

[0043] Data collection: collect the operation data of ships in a specific area, including but not limited to ship speed, heading, wave condition, wind speed, wind direction, engine state (such as speed, power output, etc.), battery state SOC and fuel consumption, etc.

[0044] Data preprocessing: the collected data is preprocessed, such as cleaning, normalization, etc., to improve the quality and availability of the data. The specific steps should include: first, data cleaning, feature selection, data conversion, then, using K nearest neighbor interpolation to fill in the missing values of the real ship data, using Z-score to normalize the abnormal values, obtaining the preprocessed real-time ship data. Finally, the data is divided into training set, validation set and test set, so as to facilitate the training and evaluation of the model.

[0045] Deep reinforcement learning model construction: the preprocessed data is used to train the deep reinforcement learning model. In the construction process, the action and state space of the model are defined, the reward function is designed, the selection of training algorithm, model training, model evaluation and optimization. The specific steps are as follows:

[0046] State space definition: define the demand power P demand, engine output power P out , battery state of charge SOC, speed V, wind speed U, course over ground COG, wave conditions, and wind direction F as the state space for reinforcement learning.

[0047] S = {P demand , SOC, P out , V, U, COG, W, F}

[0048] Action space definition: The action space defines all possible actions that the agent can perform. In this patent, the action space is designed to increase engine power output (Increase_E_P), decrease engine power output (Decrease_E_P), and adjust battery charge and discharge (Charge_Bat, Discharge_Bat).

[0049] A = {IEP, DEP, CB, DB}

[0050] Designing the reward function: The reward function is used to evaluate the immediate feedback obtained by the agent after performing a certain action. In this patent, the reward function considers the following aspects:

[0051] The reward function should encourage the agent to reduce fuel consumption. Therefore, a reward term that is inversely proportional to fuel consumption is used, such as:

[0052]

[0053] Fuel_cons is the fuel consumption.

[0054] The stability of the power system can be measured by the size of the power fluctuation. Therefore, a reward term that is inversely proportional to the power fluctuation is used, such as:

[0055]

[0056] Pwr_fluc is the power fluctuation.

[0057] The reward function encourages the agent to use supercapacitors and batteries efficiently to prolong their service life and improve energy storage efficiency. Therefore, a reward term that is proportional to the charging and discharging efficiency of supercapacitors and batteries is used, such as:

[0058]

[0059] SC_ef is the supercapacitor charging and discharging efficiency, and Bat_ef is the battery charging and discharging efficiency.

[0060] Environmental impact can be measured by emissions. Therefore, a reward term that is inversely proportional to emissions is used, such as:

[0061]

[0062] Em_rate is the emission rate.

[0063] Substitute these functions into the reward function, i.e.

[0064] R(S, A) = a * FuelEff_reward + β * Stab_reward + γ * ESS_reward + δ * Env_reward

[0065] a, β, γ and δ are weight coefficients, the values of the weight parameters are determined by multiple experiments, and the weight coefficients a, β and γ measure the individual reward value according to the importance of the allocation to each reward.

[0066] During navigation, the speed, heading and wave conditions of the ship directly affect the prediction of fuel consumption and power demand, and the deep reinforcement learning model learns and predicts future power demand through speed, heading and wave conditions, thereby adjusting the power distribution strategy, and secondly the model learns the optimal operation strategy under different wave conditions, adjusts to increase engine power output, reduce engine power output and adjust battery charging and discharging state, thereby achieving the effect of saving energy and generating the optimal strategy.

[0067] In addition, the deep reinforcement learning model adjusts and learns the navigation path through wind speed and direction data to generate a strategy that can reduce the impact of wind resistance and wind waves on energy consumption. Similarly, the deep reinforcement learning model can adjust the engine power output according to the demand power P demand and the engine output power P out to maximize efficiency and reduce fuel consumption and generate the optimal strategy. The deep reinforcement learning adjusts the charging and discharging strategy through battery state SOC data, including optimizing the order and rate of use of supercapacitors and batteries to extend battery life and maximize energy storage utilization.

[0068] Selection of training algorithm: according to the specific implementation method of the present application, the reinforcement learning algorithm is selected from one or more of DQN, DDPG, A3C and PPO algorithms.

[0069] Model training: the preprocessed data is divided into training set and validation set according to the ratio of 8:2, the training set data is used to train the model, and the optimal strategy is learned through multiple iterations.

[0070] Model evaluation and tuning: use the validation set data to evaluate the performance of the model, and tune the model according to the results.

[0071] Semi-physical simulation platform verification: the trained deep reinforcement learning-based energy management strategy model is communicated with the semi-physical simulation platform, and the actual operation environment of the ship is simulated to realize the verification of the energy management strategy model on the hardware-in-the-loop semi-physical simulation platform. The method comprises the following steps:

[0072] A hybrid power ship semi-physical simulation platform is built, which integrates the verification function of the deep reinforcement learning-based energy management strategy. The core is that the trained deep reinforcement learning-based energy management strategy model is deployed in a dSPACE real-time simulator as an energy distribution controller of the semi-physical simulation platform. Through the I / O interface, the control signal input interface of the semi-physical simulation platform is accessed, and the device parameter signal is transmitted to the dSPACE by the built-in signal acquisition board card to provide accurate input data for the strategy model. The control signal from the dSPACE is received by the driving board card and transmitted to each variable frequency device of the semi-physical simulation platform, realizing the power distribution process of each device of the hybrid power ship semi-physical simulation platform, realizing the energy management strategy verification process of the ship under actual working conditions, and generating the verification result of the semi-physical simulation platform verification. This hardware-in-the-loop (HIL) real-time simulation verification system not only simulates the physical behavior of real ship devices, but also realizes high flexibility and rapid iteration capability in the strategy verification process. Compared with the traditional full-power test platform and the real ship test, this method has a significant improvement in verification efficiency and strategy adjustment convenience.

[0073] Based on the verification result of the semi-physical simulation platform verification, the economy, emission and effectiveness of the energy management strategy constructed by the real ship data are evaluated. The evaluation indexes include but are not limited to: power system stability index, battery charge and discharge state change, and power distribution state of each power device.

[0074] The hybrid power ship semi-physical simulation platform mainly comprises:

[0075] Two motors (one simulating a ship main engine and one simulating an axle motor), a gear reducer, two clutches, a magnetic powder brake (controlling load torque, simulating actual ship working conditions), a super capacitor, a lithium battery, a circuit converter, a plurality of sensors, and various motor and battery control modules. Circuit converter and a plurality of sensors.

[0076] The simulated hybrid power device mainly includes two motors, a gear reducer, a clutch, a magnetic powder brake, a super capacitor, and a storage battery. AC / DC and DC / DC converters and DC / AC frequency converters are mainly used to realize efficient integration and simulation of hybrid multi-energy sources. The two motors include one simulating a ship main engine and one simulating an axle motor.

[0077] The control device mainly includes AC / DC and DC / DC converters and a DC / AC frequency converter, and the DC / AC frequency converter mainly adjusts the input voltage and current of the shaft motor to change the power output of the shaft motor.

[0078] The load simulation system mainly includes a magnetic powder brake, which is mainly used to realize the ship load and simulate the actual working condition of the ship.

[0079] The measurement and control system includes a speed and torque sensor, a battery SOC (battery charge coefficient) collector, a voltage and current sensor, a dSPACE real-time simulator, an information collection board card and a drive board card. The speed and torque sensor monitors and collects the speed and torque data of the simulated ship main engine and shaft motor in real time. The dSPACE, as the core of the measurement and control system, is responsible for receiving various signals from the sensor, and performs analog-to-digital conversion, processing operation and power distribution control. It converts the calculation results into analog quantities, controls the power distribution of the experimental platform through the output interface, and realizes efficient energy management.

[0080] The motor of the platform is mainly powered by 380V mains (simulating the ship generator set) and lithium batteries and super capacitors in cooperation with grid-connected power supply. The 380V mains adopts three grid-connected power supply, and connects three transformer protection circuits. The information collection board card imports the collected data into the dSPACE through the I / O interface. The drive board card adjusts the power distribution of the mains and the battery and the input voltage and current of the shaft motor.

[0081] The energy management strategy of the hybrid ship mainly includes a deep reinforcement learning algorithm. At present, the deep reinforcement learning energy management strategies mainly verified include DQN, DDPG, A3C and PPO algorithms, etc. The energy management strategy algorithm realizes code compilation through Matlab / Simulink software, and is arranged into a fixed strategy module and downloaded to the inside of the dSPACE. Through the I / O interface, the specified control unit external interface is accessed, and the power distribution process of the semi-physical simulation platform is realized.

[0082] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0083] The application provides a ship energy management strategy construction and verification method based on deep reinforcement learning, adopts data-driven to construct a ship energy management strategy model based on deep reinforcement learning, downloads the constructed and trained energy management strategy model to a dSPACE, connects an I / O interface of a dSPACE real-time simulator with a hybrid power ship semi-physical simulation platform, and verifies the ship energy management strategy model constructed in the first stage by using the hybrid power ship semi-physical simulation platform, so that the dSPACE loaded with the energy management strategy is used as an energy distribution controller to realize power control processes of simulated main engine, shaft motor, 380V commercial power, lithium battery and super capacitor and the like of the semi-physical simulation platform.

[0084] The hybrid power ship semi-physical simulation platform is a highly integrated system, mainly including two motors (one for simulating a ship main engine and the other for simulating a shaft motor), a gear reducer, two clutch groups, a magnetic powder brake (for controlling load torque to simulate actual ship operation conditions), a super capacitor, a lithium battery, a circuit conversion device and various sensors and motor battery control modules. The platform is specially designed to simulate a hybrid power device, which integrates two motors, a gear reducer, a clutch, a magnetic powder brake, a super capacitor and a battery, and realizes efficient integration and simulation of hybrid multi-energy through AC / DC and DC / DC converters and a DC / AC frequency converter. The DC / AC frequency converter is mainly responsible for adjusting the input voltage and current of the shaft motor, thereby changing its power output. The load simulation system mainly relies on a magnetic powder brake to simulate the load and actual operation conditions of the ship. The measurement and control system also has comprehensive functions, including a speed and torque sensor, a battery SOC collector, a voltage and current sensor, a dSPACE real-time simulator and an information acquisition and driving board. The speed and torque sensor monitors and collects the speed and torque data of the two motors in real time, and the dSPACE, as the core of the measurement and control system, is responsible for receiving, processing and converting various sensor signals, performing power distribution control, and realizing power distribution and efficient energy management of the experimental platform through the output interface. The platform mainly realizes grid-connected power supply of 380V commercial power (simulating a ship generator set) and lithium batteries and super capacitors in cooperation, and adopts a three-way grid-connected mode to connect three transformer protection circuits to ensure stable operation of the system.

[0085] A method for constructing and verifying a ship energy management strategy based on deep reinforcement learning. In the construction process, first, real-time data of a certain segment of the ship's voyage is collected and the collected data is preprocessed. Then the preprocessed data is generated into a training set, a validation set and a test set. A ship energy management strategy model based on deep reinforcement learning is built through Matlab / Simulink software. Then the model is trained, evaluated and optimized, and finally the optimal strategy is output and downloaded to the dSPACE. Then the dSPACE equipped with the trained energy management strategy is connected to the hybrid ship semi-physical simulation platform through the I / O interface to verify the energy management strategy, as shown in Figure 1 includes the following detailed steps:

[0086] Step one: collect real-time data of a certain segment of the ship's voyage, including but not limited to ship speed, heading, wave conditions, load conditions, wind speed, wind direction, engine power output, fuel consumption and battery SOC, etc., and preprocess the collected data. Then the preprocessed data is divided into a training set and a validation set according to the ratio of 8:2, and the test set data is randomly selected from the training set.

[0087] Step two: use Matlab / Simulink software to build a ship energy management strategy model based on deep reinforcement learning according to the preprocessed ship operation data collected in the early stage, including but not limited to state space, action space definition and reward function design.

[0088] Step three: use the training set data to train the energy management strategy model, and then use the validation set and test set data to evaluate the model performance and optimize the model.

[0089] Step four: continuously collect new data and learn, compile and download the trained strategy model to the dSPACE real-time simulator as an energy distribution controller.

[0090] Step five: start the hybrid ship semi-physical simulation platform, make sure all devices are in standby state, including starting the mains and lithium battery power supply, adjusting the two motors to the specified speed, and setting the clutch and load simulation system (magnetic powder brake) to the initial state. Then start the dSPACE real-time simulator, load and start the energy management strategy model.

[0091] Step six: simulate various signals in actual ship operation through the built-in sensors in the platform, such as speed torque sensor, battery SOC collector, voltage current sensor, etc. These signals are collected and transmitted to the dSPACE real-time simulator in real time as the input of the energy management strategy model.

[0092] Step seven: The energy management strategy model in the dSPACE real-time simulator performs deep reinforcement learning calculations based on the input real-time signals and outputs the optimal energy distribution strategy. The strategy outputs specific action instructions, such as increasing engine power output, reducing engine power output, adjusting battery charging and discharging state, etc. Through the I / O interface of dSPACE, the action instructions are transmitted to the platform control signal input interface to control the motor, clutch, magnetic powder brake, super capacitor and lithium battery on the platform to perform corresponding operations according to the strategy instructions.

[0093] Step eight: During the test process, the operating status and parameters of each device on the platform are continuously monitored, such as motor speed, torque, battery SOC, power distribution, etc. The monitored state data is fed back to the strategy model in the dSPACE real-time simulator in real time as the basis for the next calculation.

[0094] Step nine: According to the actual performance and data feedback during the test process, the energy management strategy model is optimized and adjusted online, especially for complex working conditions and abnormal situations. Through the self-improvement ability of deep reinforcement learning, the strategy parameters are dynamically adjusted to adapt to different operating conditions and working conditions.

[0095] Step ten: After completing a certain period of testing, stop the torque output of the magnetic powder brake, return the system to the idle state, gradually stop the operation of the motor and the engagement of the clutch, close the mains and lithium battery power supply switch, and ensure safe shutdown of the experimental platform. After the test is completed, the test data is comprehensively analyzed to evaluate the economy, emissions and effectiveness of the energy management strategy. Evaluation indicators include but are not limited to power system stability indicators, battery charging and discharging state changes, power equipment power distribution status, and overall energy consumption and emission levels.

[0096] Compared with the prior art, the beneficial effects of the present application are:

[0097] Through the application of the present application, from the construction level, the data-driven method can learn and adapt from actual operation data without deep understanding of the system mechanism, has better self-adaptation ability, can handle diversified operating conditions and working conditions, so that the constructed strategy has better generalization ability, and does not need to establish an accurate mathematical model, reducing the complexity and cost of model development. From the verification level, the simulation speed, accuracy and reliability are improved, the energy management strategy model and the semi-physical simulation platform are interacted and communicated, the performance of the energy management strategy in actual application can be more realistically reflected, and the energy management strategy is optimized according to the results obtained by testing and verification.

[0098] 1. The energy management strategy based on deep reinforcement learning not only relies on traditional model-driven methods, but also combines data-driven characteristics, can learn and adapt from actual operation data, and has better adaptive ability and generalization ability.

[0099] 2. The energy management strategy based on deep reinforcement learning can respond to changes in the ship's operating state in real time, automatically adjust strategy parameters through continuous learning and optimization processes to adapt to diverse operating conditions and working conditions, and quickly achieve optimal energy distribution decisions, improving system flexibility and response speed.

[0100] 3. By combining the energy management strategy based on deep reinforcement learning with the simulation platform, the application provides a new verification method that can more realistically reflect the performance of the energy management strategy in actual application, improving the accuracy and reliability of the verification.

[0101] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for ship energy management strategy construction and verification based on deep reinforcement learning, characterized in that, The application is applied to a ship energy management strategy construction and verification system based on deep reinforcement learning. The system body is a hybrid power ship semi-physical simulation platform, which at least includes a simulated hybrid power device and a measurement and control system. The simulated hybrid power device is used to simulate the ship load and the actual working condition of the ship. The measurement and control system is used for data acquisition, and a ship energy management strategy construction and verification method based on deep reinforcement learning is stored on the measurement and control system. The method comprises the following steps: Collecting running data in a certain section of ship navigation and preprocessing, dividing the preprocessed data into training set and verification set according to the preset proportion, and randomly selecting data from the training set to generate test set; According to the running data, a ship energy management strategy model based on deep reinforcement learning is constructed; The ship energy management strategy model is trained by the data of the training set, and the performance of the model is evaluated and optimized by the data of the verification set and the test set; The actual working condition of the ship is simulated by the hybrid power ship semi-physical simulation platform, the simulation data is collected and used as the input of the ship energy management strategy model; The ship energy management strategy model performs deep reinforcement learning according to the simulation data, outputs an energy distribution strategy and corresponding action instructions for controlling the hybrid power ship semi-physical simulation platform; The hybrid power ship semi-physical simulation platform adjusted and controlled according to the energy distribution strategy is continuously data collected, and the collected data is used as feedback data; According to the feedback data, the ship energy management strategy model is optimized and adjusted, the strategy parameters are dynamically adjusted through deep reinforcement learning, and a dynamically adjusted ship energy management strategy model is generated to adapt to different running conditions and actual working conditions of the actual ship.

2. The deep reinforcement learning based ship energy management policy construction and verification method according to claim 1, characterized in that, The running data at least includes the speed, heading, wave condition, wind speed, wind direction, engine speed, engine output power, battery state and fuel consumption of the ship.

3. The deep reinforcement learning based ship energy management policy construction and verification method according to claim 1, characterized in that, The preprocessing specifically includes data cleaning, feature selection and data conversion of the running data, filling of missing values in the running data by K nearest neighbor interpolation algorithm, and normalization processing of abnormal values in the running data by Z score.

4. The deep reinforcement learning based ship energy management policy construction and verification method according to claim 1, characterized in that, The construction method of the ship energy management strategy model is: defining the state space and action space of the deep reinforcement learning and all possible actions executable by the agent in the ship energy management strategy model, and defining the reward function for evaluating the immediate feedback obtained after the agent executes a certain action.

5. The deep reinforcement learning based ship energy management policy construction and verification method according to claim 4, characterized in that, The method of state space definition is specifically: The required power P of the vessel in the entire working condition demand , the engine output power P out , the battery state of charge SOC, the speed V, the wind speed U, the course over ground COG, the wave condition and the wind direction F as the state space S of deep reinforcement learning, expressed as: S = {P demand , SOC, P out , V, U, COG, W, F} The method of action space definition is specifically represented as: A={IEP,DEP,CB,DB} Wherein, A is the action space, IEP is to increase the engine power output, DEP is to reduce the engine power output, CB is to adjust the battery charging state, and DB is to adjust the battery discharging state.

6. The deep reinforcement learning based ship energy management policy construction and verification method according to claim 5, characterized in that, The rewards set in the reward function at least include the combustion efficiency reward representing the inverse ratio of fuel consumption, the fluctuation reward representing the inverse ratio of power fluctuation, the energy storage efficiency reward representing the positive ratio of super capacitor and battery charging and discharging efficiency, and the environmental impact reward representing the inverse ratio of emission; the reward function is represented as: R(S, A) = a * FuelEff_reward + b * Stab_reward + g * ESS_reward + d * Env_reward wherein R(S, A) is a reward function defined based on state space and action space, a, b, g and d are weight coefficients, FuelEff_reward is a fuel efficiency reward, Stab_reward is a fluctuation reward, ESS_reward is an energy efficiency storage reward, and Env_reward is an environmental impact reward.

7. The deep reinforcement learning based ship energy management policy construction and verification method according to claim 1, characterized in that, The selected deep learning reinforcement algorithm is one or more of DQN, DDPG, A3C and PPO algorithms.

8. A system for ship energy management policy construction and verification based on deep reinforcement learning, characterized in that, The system body is a hybrid power ship semi-physical simulation platform, which at least comprises a simulated hybrid power device and a measurement and control system, wherein The simulated hybrid power device is used to simulate ship load and actual working conditions of the ship. The measurement and control system is used for data acquisition, and the measurement and control system at least comprises a speed and torque sensor, a battery charge coefficient collector, a voltage and current sensor, a dSPACE real-time simulator, an information acquisition board and a drive board; wherein the speed and torque sensor is used to monitor and collect the speed and torque data of the simulated ship main engine and shaft motor in real time; the dSPACE real-time simulator is used to receive signals from the speed and torque sensor, the battery charge coefficient collector and the voltage and current sensor, and to perform analog-digital conversion, processing operation and power distribution control, and it stores the ship energy management strategy construction and verification method based on deep reinforcement learning according to any one of claims 1-7, and generates a dynamically adjusted ship energy management strategy model based on the method to adapt to different operating conditions and actual working conditions of the actual ship.

9. The deep reinforcement learning based ship energy management policy building and verification system according to claim 8, characterized in that, The simulated hybrid power device at least comprises two motors, a gear reduction box, a clutch, a magnetic powder brake, a super capacitor, a storage battery and a control device; wherein one of the two motors simulates a ship main engine and the other simulates a shaft motor; the control device comprises AC / DC and DC / DC converters and a DC / AC frequency converter, which is used to change the power output of the simulated shaft motor by adjusting the input voltage and current thereof; the magnetic powder brake is used to realize ship load and simulate actual working conditions of the ship.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to realize the steps of the method according to any one of claims 1-7. The computer program is executed by a processor to realize the steps of the method according to any one of claims 1-7.

Images