Ship energy management strategy construction and verification method based on deep reinforcement learning

Abstract

The invention discloses a ship energy management strategy construction and verification method based on deep reinforcement learning. The method comprises the following steps: acquiring operation data of a ship in a certain section of navigation and preprocessing the operation data; constructing a ship energy management strategy model based on deep reinforcement learning according to the operation data; training the ship energy management strategy model, and evaluating and optimizing the performance of the model; the actual working condition of the ship is simulated through the hybrid power ship semi-physical simulation platform; the ship energy management strategy model carries out deep reinforcement learning, and outputs an energy distribution strategy and a corresponding instruction for controlling the motion of the semi-physical simulation platform of the hybrid power ship; performing continuous data acquisition on the hybrid ship semi-physical simulation platform adjusted and controlled according to the energy distribution strategy, and taking the acquired data as feedback data; and optimizing and adjusting the ship energy management strategy model according to the feedback data. According to the method, learning and adaptation can be carried out from actual operation data, and the better adaptive capacity is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of ship energy management, and specifically relates to a ship energy management strategy construction and verification method based on deep reinforcement learning. Background Art

[0002] The shipping industry is one of the important pillars of world trade and transportation, and is also one of the main sources of air pollution and greenhouse gas emissions. According to statistics from the International Maritime Organization (IMO), in 2018, carbon dioxide emissions from ships accounted for 2.89% of the world's total emissions, and this proportion is expected to increase to 17% by 2050. In order to cope with increasingly severe environmental problems and energy crises, the shipping industry is facing tremendous pressure and challenges in energy conservation and emission reduction.

[0003] In order to improve the energy efficiency of ships and reduce their emissions, hybrid power systems have become an important direction for the development of future ship power. A ship hybrid power system refers to a system that combines two or more power sources (such as internal combustion engines, batteries, fuel cells, etc.) in different ways (such as series, parallel and hybrid), and meets the ship's power needs through energy conversion, storage and distribution. Compared with traditional single power sources, ship hybrid power systems have the advantages of high output power, energy recovery and storage, improved ship safety and reliability, and reduced ship noise and vibration. How to effectively manage the energy flow between multiple energy sources in a hybrid power system to meet the power requirements under different working conditions and optimize the performance indicators of the ship power system (such as fuel consumption, emissions, etc.) involves how to formulate reasonable energy management strategies or decision rules to guide the switching state and power distribution of each energy source.

[0004] The existing hybrid power system energy management strategies mainly include the following three categories:

[0005] (1) Rule-based strategies rely on engineers’ in-depth understanding of ship power systems and rich practical experience. This type of strategy usually controls the switching state and power distribution of energy sources through preset logical thresholds.

[0006] (2) Optimization-based strategies, which can be divided into global optimization and real-time optimization, use advanced mathematical algorithms to find the optimal energy allocation solution. These strategies can overcome the shortcomings of rule-based strategies to a certain extent, provide more precise and flexible control, and thus theoretically achieve higher energy efficiency and lower emissions.

[0007] (3) Learning-based strategies: These strategies do not require a complete system model and prediction information, and have the ability of online learning and adaptive adjustment, but they require a large amount of data and training time.

[0008] In recent years, deep reinforcement learning (DRL), as an emerging artificial intelligence technology, has demonstrated its powerful learning and optimization capabilities in many fields. DRL allows the agent to learn the optimal strategy in the interaction with the environment, and can find the optimal solution through trial and error and self-improvement without fully understanding the environmental model. In the field of ship energy management, DRL has the potential to achieve real-time and adaptive control of ship hybrid power systems by learning complex energy distribution strategies, thereby optimizing the energy efficiency of ships and reducing emissions.

[0009] At present, the construction method adopted by the existing energy management strategy based on deep reinforcement learning is data-driven construction. The data-driven method focuses on using a large amount of data for learning and mapping to establish the relationship between input and output, rather than relying on clear rules or patterns. The advantage is that it can handle complex nonlinear relationships and learn potential laws from historical data. The verification method adopted by the strategy focuses on model software simulation verification. Although this method has fast verification speed and low experimental cost, it may not be able to fully simulate the complexity of the real world, resulting in limited credibility and flexibility of the simulation results. In addition, the existing energy management strategies based on deep reinforcement learning rarely adopt a complete set of strategy construction and verification systems. Therefore, how to systematically build and verify the ship energy management strategy based on deep reinforcement learning has become an urgent problem to be solved. Summary of the invention

[0010] The purpose of the present invention is to provide a method for constructing and verifying a ship energy management strategy based on deep reinforcement learning, which can learn and adapt from actual operating data, has better adaptability, is more flexible in dealing with complex systems and nonlinear problems, and can handle a variety of operating conditions and working conditions, and has better generalization ability.

[0011] In order to solve the above technical problems, the technical solution of the present invention is: a ship energy management strategy construction and verification method based on deep reinforcement learning is applied to a ship energy management strategy construction and verification system based on deep reinforcement learning, the main body of the system is a hybrid ship semi-physical simulation platform, which at least includes a simulated hybrid power device and a measurement and control system, wherein the simulated hybrid power device is used to simulate the ship load and the actual working condition of the ship; the measurement and control system is used for data collection, and the ship energy management strategy construction and verification method based on deep reinforcement learning is stored thereon; the method comprises the following steps:

[0012] Collect and preprocess the operating data of a ship during a certain voyage, divide the preprocessed data into a training set and a validation set according to a preset ratio, and randomly select data from the training set to generate a test set;

[0013] Build a ship energy management strategy model based on deep reinforcement learning according to operation data;

[0014] The ship energy management strategy model is trained using the data from the training set, and the performance of the model is evaluated and optimized using the data from the validation set and test set;

[0015] The actual working conditions of the ship are simulated through the hybrid ship semi-physical simulation platform, and the simulation data is collected and used as the input of the ship energy management strategy model;

[0016] Through the ship energy management strategy model, deep reinforcement learning is performed based on simulation data to output energy allocation strategies and corresponding action instructions for controlling the hybrid ship semi-physical simulation platform;

[0017] Continuously collect data from the hybrid ship semi-physical simulation platform that adjusts control according to the energy distribution strategy and use the collected data as feedback data;

[0018] The ship energy management strategy model is optimized and adjusted according to the feedback data, and the strategy parameters are dynamically adjusted through deep reinforcement learning to generate a dynamically adjusted ship energy management strategy model to adapt to the different operating conditions and actual working conditions of the actual ship.

[0019] The operating data includes at least the ship's speed, heading, wave conditions, wind speed, wind direction, engine speed, engine output power, battery status and fuel consumption.

[0020] The specific preprocessing includes: data cleaning, feature selection, and data conversion of the operating data, filling the missing values ​​in the operating data through the K nearest neighbor interpolation algorithm, and normalizing the outliers in the operating data using the Z score.

[0021] The method for constructing the ship energy management strategy model is as follows: define the state space and action space of the deep reinforcement learning state space and all possible actions that the agent can perform in the ship energy management strategy model, and define a reward function for evaluating the immediate feedback obtained after the agent performs a certain action.

[0022] The method of defining the state space is as follows:

[0023] The power requirement P of the ship in the whole working condition demand , engine output power P out , battery state of charge SOC, speed V, wind speed U, course over ground COG, wave conditions and wind direction F are used as the state space S of deep reinforcement learning, expressed as:

[0024] S={P demand ,SOC,P out,V,U,COG,W,F}

[0025] The method of defining the action space is specifically expressed as follows:

[0026] A={IEP,DEP,CB,DB}

[0027] Among them, A is the action space, IEP is to increase the engine power output, DEP, is to reduce the engine power output, CB is to adjust the battery charging state, and DB is to adjust the battery discharging state.

[0028] The rewards set in the reward function include at least a combustion efficiency reward inversely proportional to fuel consumption, a fluctuation reward inversely proportional to power fluctuation, an energy storage efficiency reward proportional to the charging and discharging efficiency of supercapacitors and batteries, and an environmental impact reward inversely proportional to emissions; the reward function is expressed as:

[0029] R(S,A)=α*FuelEff_reward+β*Stab_reward+γ*ESS_reward+δ*Env_reward

[0030] Among them, R(S,A) is the reward function based on the state space definition and the action space definition, α, β, γ and δ are weight coefficients, FuelEff_reward is the fuel efficiency reward, Stab_reward is the volatility reward, ESS_reward is the energy efficiency storage reward, and Env_reward is the environmental impact reward.

[0031] The selected deep learning reinforcement algorithm is one or more of DQN, DDPG, A3C and PPO algorithms.

[0032] It also provides a ship energy management strategy construction and verification system based on deep reinforcement learning. The main body of the system is a hybrid ship semi-physical simulation platform, which at least includes a simulated hybrid power device and a measurement and control system.

[0033] The simulated hybrid power plant is used to simulate the ship load and actual ship working conditions;

[0034] The measurement and control system is used for data collection, and the measurement and control system includes at least a speed torque sensor, a battery charge factor collector, a voltage and current sensor, a dSPACE real-time simulator, an information acquisition board and a drive board; wherein the speed torque sensor is used to monitor and collect the speed and torque data of the simulated ship main engine and shaft belt motor in real time; the dSPACE real-time simulator is used to receive signals from the speed torque sensor, the battery charge factor collector, and the voltage and current sensors, and perform analog-to-digital conversion, processing operations and power distribution control, and stores thereon a ship energy management strategy construction and verification method based on deep reinforcement learning as described in any of the above items, and generates a dynamically adjusted ship energy management strategy model based on this method to adapt to the different operating conditions and actual working conditions of the actual ship.

[0035] The simulated hybrid power device includes at least two motors, a gear reducer, a clutch, a magnetic powder brake, a supercapacitor, a battery and a control device; among them, one of the two motors simulates the ship's main engine and the other simulates the shaft belt motor; the control device includes AC / DC and DC / DC converters and a DC / AC inverter, which are used to change the power output of the simulated shaft belt motor by adjusting its input voltage and current; the magnetic powder brake is used to realize the ship load and simulate the actual working conditions of the ship.

[0036] A computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention constructs strategies through data-driven and adopts semi-physical simulation platform to verify the strategies. Compared with mechanism models, data-driven can learn and adapt from actual operation data, has better self-adaptation ability, is more flexible in dealing with complex systems and nonlinear problems, and can handle diverse operating conditions and working conditions, and has better generalization ability. Compared with model-in-the-loop simulation verification, semi-physical simulation platform verification can simulate the operating conditions of the ship in a real physical environment, and more realistically reflect the performance of energy management strategies in actual applications. And real hardware (such as engines, generators, batteries, etc.) can be combined with energy management strategies for testing, providing a hardware-in-the-loop (HIL) test environment, which can more accurately evaluate energy management strategies and system integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] The technical solution of the present invention is: a method for constructing and verifying a ship energy management strategy based on deep reinforcement learning, which adopts a data-driven construction strategy and a semi-physical simulation platform to verify the strategy. Compared with the mechanism model, the data-driven model can learn and adapt from the actual operation data, has better self-adaptation ability, is more flexible in dealing with complex systems and nonlinear problems, and can handle a variety of operating conditions and working conditions, and has better generalization ability. Compared with the model-in-the-loop simulation verification, the semi-physical simulation platform can simulate the operating conditions of the ship in a real physical environment, and more realistically reflect the performance of the energy management strategy in actual application. And the real hardware (such as engines, generators, batteries, etc.) can be combined with the energy management strategy for testing, providing a hardware-in-the-loop (HIL) test environment, and more accurately evaluating the energy management strategy and system integration. The present invention is based on a diesel-electric hybrid ship, including a supercapacitor, a battery and a diesel generator set.

[0042] In order to solve the above problems, the present invention mainly includes the following steps:

[0043] Data collection: Collect operating data of ships in a specific area, including but not limited to the ship's speed, heading, wave conditions, wind speed, wind direction, engine status (such as speed, power output, etc.), battery status SOC, and fuel consumption.

[0044] Data preprocessing: Clean and normalize the collected data to improve the quality and availability of the data. The specific steps should include: first, carry out data cleaning, feature selection, and data conversion. Then, use K nearest neighbor interpolation to fill the missing values ​​of the real ship data, and use Z-score to normalize the outliers to obtain the preprocessed real-time ship data. Finally, split the data into training set, validation set, and test set to facilitate model training and evaluation.

[0045] Deep reinforcement learning model construction: The preprocessed data is used to train the deep reinforcement learning model. The construction process defines the model's action and state space, designs the reward function, selects the training algorithm, trains the model, and evaluates and tunes the model. The specific steps are as follows:

[0046] State space definition: The power demand P of the ship in the entire working condition demand, engine output power P out , battery state of charge SOC, ship speed V, wind speed U, course over ground COG, wave conditions and wind direction F are used as the state space for reinforcement learning.

[0047] S={P demand ,SOC,P out ,V,U,COG,W,F}

[0048] Action space definition: The action space defines all possible actions that the agent can perform. The action space of this patent is designed to increase engine power output (Increase_E_P), reduce engine power output (Decrease_E_P) and adjust battery charge and discharge status (Charge_Bat, Discharge_Bat).

[0049] A={IEP,DEP,CB,DB}

[0050] Design reward function: The reward function is used to evaluate the immediate feedback obtained after the agent performs an action. The reward function of this patent considers the following aspects:

[0051] The reward function should encourage the agent to reduce fuel consumption. Therefore, a reward term that is inversely proportional to fuel consumption is used, such as:

[0052]

[0053] Fuel_cons is the fuel consumption.

[0054] The stability of the power system can be measured by the magnitude of the power fluctuation. Therefore, a reward term inversely proportional to the power fluctuation is used, such as:

[0055]

[0056] Pwr_fluc is the power fluctuation.

[0057] The reward function encourages the agent to use supercapacitors and batteries efficiently to extend their service life and improve energy storage efficiency. Therefore, a reward term proportional to the charging and discharging efficiency of supercapacitors and batteries is used, such as:

[0058]

[0059] SC_ef is the supercapacitor charge and discharge efficiency, and Bat_ef is the battery charge and discharge efficiency.

[0060] Environmental impact can be measured by emissions. Therefore, use a reward term that is inversely proportional to emissions, such as:

[0061]

[0062] Em_rate is the emission rate.

[0063] Substituting these functions into the reward function, we get

[0064] R(S,A)=α*FuelEff_reward+β*Stab_reward+γ*ESS_reward+δ*Env_reward

[0065] α, β, γ, and δ are weight coefficients. The values ​​of the weight parameters are determined through multiple experiments. The weight coefficients α, β, and γ weigh the individual reward values ​​according to the importance assigned to each reward.

[0066] During navigation, the ship's speed, heading and wave conditions directly affect the prediction of fuel consumption and electricity demand. The deep reinforcement learning model learns and predicts future power demand through speed, heading and wave conditions, thereby adjusting the power allocation strategy. Secondly, the model learns the best operating strategy under different wave conditions, adjusts to increase engine power output, reduce engine power output and adjust battery charging and discharging status, thereby achieving energy saving effect and generating the optimal strategy.

[0067] In addition, the deep reinforcement learning model adjusts and learns the navigation path through wind speed and direction data to generate strategies that can reduce the impact of wind resistance and waves on energy consumption. demand and engine output power P out Adjust engine power output to maximize efficiency and reduce fuel consumption and generate the optimal strategy. Deep reinforcement learning adjusts the charging and discharging strategy based on battery state SOC data, including optimizing the order and rate of use of supercapacitors and batteries to extend battery life and maximize energy storage utilization.

[0068] Selection of training algorithm: According to the specific implementation method of the present invention, the reinforcement learning algorithm is selected from one or more of DQN, DDPG, A3C and PPO algorithms.

[0069] Model training: The preprocessed data is divided into a training set and a validation set in a ratio of 8:2. The model is trained using the training set data, and the optimal strategy is learned through multiple iterations.

[0070] Model evaluation and tuning: Use validation set data to evaluate model performance and tune the model based on the results.

[0071] Semi-physical simulation platform verification: The trained energy management strategy model based on deep reinforcement learning interacts with the semi-physical simulation platform, and the energy management strategy model is verified on the hardware-in-the-loop semi-physical simulation platform by simulating the actual operating environment of the ship. The method includes the following steps:

[0072] A hybrid ship semi-physical simulation platform based on a real ship is built. The platform integrates the verification function of the energy management strategy based on deep reinforcement learning. The core is to deploy the trained energy management strategy model based on deep reinforcement learning into a dSPACE real-time simulator as the energy distribution controller of the semi-physical simulation platform. Through the I / O interface, it is connected to the control signal input interface of the semi-physical simulation platform. At the same time, the signal acquisition board built into the platform transmits the device parameter signal into dSPACE to provide accurate input data for the strategy model. The drive board receives the control signal from dSPACE and transmits it to each converter device of the semi-physical simulation platform, realizing the power distribution process of each device of the hybrid ship semi-physical simulation platform, realizing the energy management strategy verification process of the ship under actual working conditions, and generating the verification results of the semi-physical simulation platform. This hardware-in-the-loop (HIL) real-time simulation verification system not only simulates the physical behavior of real ship equipment, but also realizes high flexibility and rapid iteration capabilities in the strategy verification process. Compared with the traditional full-power test platform and real ship test, this method has significantly improved the verification efficiency and convenience of strategy adjustment.

[0073] Based on the verification results of the semi-physical simulation platform, the economy, emissions and effectiveness of the energy management strategy constructed based on the real ship data are evaluated. The evaluation indicators include but are not limited to: power system stability indicators, battery charging and discharging status changes, and power distribution status of each power equipment.

[0074] The hybrid ship semi-physical simulation platform mainly includes:

[0075] Two motors (one simulating the ship's main engine and one simulating the shaft belt motor), gear reducer, two sets of clutches, magnetic powder brakes (to control load torque and simulate actual ship working conditions), supercapacitors, lithium batteries, circuit converters, several sensors and various motor battery control modules. Circuit converters and several sensors.

[0076] The simulated hybrid power unit mainly consists of two motors, a gear reducer, a clutch, a magnetic powder brake, a supercapacitor, and a battery. The AC / DC and DC / DC converters and the DC / AC inverter mainly use low-power equipment to achieve efficient integration and simulation of hybrid multi-energy. Two of the motors, one simulates the ship's main engine and the other simulates the shaft belt motor.

[0077] The control devices mainly include AC / DC and DC / DC converters and DC / AC inverters. The DC / AC inverter mainly adjusts the input voltage and current of the shaft-belt motor to change the power output of the shaft-belt motor.

[0078] The load simulation system mainly includes a magnetic powder brake, which is mainly used to realize the ship load and simulate the actual working conditions of the ship.

[0079] The measurement and control system includes a speed torque sensor, a battery SOC (battery charge coefficient) collector, a voltage and current sensor, a dSPACE real-time simulator, an information acquisition board, and a drive board. The speed torque sensor monitors and collects the speed and torque data of the simulated ship main engine and shaft belt motor in real time. As the core of the measurement and control system, dSPACE is responsible for receiving various signals from sensors and performing analog-to-digital conversion, processing operations, and power distribution control. It converts the calculation results into analog quantities, controls the power distribution of the experimental platform through the output interface, and realizes efficient energy management.

[0080] The motor of this platform is mainly powered by 380V mains (simulating ship generator sets) and lithium batteries and supercapacitors. The 380V mains adopts three-way grid connection and connects three transformer protection circuits. The information acquisition board imports the collected data into dSPACE through the I / O interface. The drive board adjusts the power distribution of the mains and the battery and the input voltage and current of the shaft motor.

[0081] The hybrid ship energy management strategy mainly includes a deep reinforcement learning algorithm. The main verified deep reinforcement learning energy management strategies include: DQN, DDPG, A3C and PPO algorithms, etc. The energy management strategy algorithm is compiled through Matlab / Simulink software, organized into a fixed strategy module and downloaded to dSPACE. It is connected to the external interface of the specified control unit through the I / O interface to realize the power allocation process of the semi-physical simulation platform.

[0082] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0083] The present invention provides a method for constructing and verifying a ship energy management strategy based on deep reinforcement learning. The method adopts data-driven construction to construct a ship energy management strategy model based on deep reinforcement learning. The constructed and trained energy management strategy model is downloaded to dSPACE. The I / O interface of the dSPACE real-time simulator is connected to a hybrid power ship semi-physical simulation platform. The hybrid power ship semi-physical simulation platform is used to verify the ship energy management strategy model constructed in the first phase. The dSPACE equipped with the energy management strategy is used as an energy distribution controller to realize the power control process of simulating a main engine, a shaft belt motor, a 380V AC power supply, a lithium battery, a supercapacitor and other equipment on the semi-physical simulation platform.

[0084] The hybrid ship semi-physical simulation platform is a highly integrated system, mainly including two motors (one for simulating the ship's main engine and the other for simulating the shaft belt motor), gear reducers, two sets of clutches, magnetic powder brakes (used to control load torque to simulate the actual ship's operating conditions), supercapacitors, lithium batteries, circuit converters, and a variety of sensors and motor battery control modules. This platform is specially designed to simulate a hybrid power device, which integrates two motors, gear reducers, clutches, magnetic powder brakes, supercapacitors and batteries, and realizes efficient integration and simulation of hybrid multi-energy through AC / DC and DC / DC converters and DC / AC inverters. Among them, the DC / AC inverter is mainly responsible for adjusting the input voltage and current of the shaft belt motor, thereby changing its power output. The load simulation system mainly relies on a magnetic powder brake to simulate the ship's load and actual operating conditions. The measurement and control system is also comprehensive, including speed torque sensor, battery SOC collector, voltage and current sensor, dSPACE real-time simulator, and information acquisition and drive board. The speed and torque sensors monitor and collect the speed and torque data of the two motors in real time, and dSPACE, as the core of the measurement and control system, is responsible for receiving, processing and converting various sensor signals, performing power distribution control, and realizing power distribution and efficient energy management of the experimental platform through the output interface. The platform is mainly powered by 380V mains (simulating ship generator sets) in conjunction with lithium batteries and supercapacitors, and three transformer protection circuits are connected in a three-way grid-connected manner to ensure stable operation of the system.

[0085] The ship energy management strategy construction and verification method based on deep reinforcement learning. During the construction process, it is first necessary to collect real-time data of a certain section of the ship's voyage and preprocess the collected data. Then, the preprocessed data is used to generate a training set, a verification set, and a test set. A ship energy management strategy model based on deep reinforcement learning is built using Matlab / Simulink software. Model training, model evaluation, and tuning are then performed, and finally the optimal strategy is output and downloaded to dSPACE. The dSPACE equipped with the trained energy management strategy is then connected to the hybrid ship semi-physical simulation platform through the I / O interface to verify its energy management strategy, such as Figure 1 The following detailed steps are shown:

[0086] Step 1: Collect real-time data of a certain section of the ship's voyage, including but not limited to the ship's speed, heading, wave conditions, load conditions, wind speed, wind direction, engine power output, fuel consumption and battery SOC, and preprocess the collected data. The preprocessed data is then divided into a training set and a validation set in an 8:2 ratio, and the test set data is randomly selected from the training set.

[0087] Step 2: Use Matlab / Simulink software to build a ship energy management strategy model based on deep reinforcement learning based on the ship operation data collected and preprocessed in the early stage, including but not limited to the definition of state space, action space and the design of reward function.

[0088] Step 3: Use the training set data to train the energy management strategy model, and then use the validation set and test set data to evaluate the model performance and perform optimization.

[0089] Step 4: Continue to collect new data and learn, compile and download the trained strategy model to the dSPACE real-time simulator as an energy distribution controller.

[0090] Step 5: Start the hybrid ship semi-physical simulation platform and ensure that all equipment is in standby mode, including starting the mains power and lithium battery power supply, adjusting the two motors to the specified speed, and setting the clutch and load simulation system (magnetic powder brake) to the initial state. Then start the dSPACE real-time simulator, load and start the energy management strategy model.

[0091] Step 6: Various signals in actual ship operation are simulated through the platform's built-in sensors (such as speed torque sensor, battery SOC collector, voltage and current sensor, etc.), such as speed, heading, wave conditions, wind speed, wind direction, engine status, battery SOC, etc. These signals are collected in real time and transmitted to the dSPACE real-time simulator as input to the energy management strategy model.

[0092] Step 7: The energy management strategy model in the dSPACE real-time simulator performs deep reinforcement learning calculations based on the input real-time signals and outputs the optimal energy allocation strategy. The strategy outputs specific action instructions, such as increasing engine power output, reducing engine power output, adjusting battery charge and discharge status, etc. The action instructions are transmitted to the platform control signal input interface through the dSPACE I / O interface, and the motor, clutch, magnetic powder brake, supercapacitor, lithium battery and other equipment on the control platform perform corresponding operations according to the strategy instructions.

[0093] Step 8: During the test, the operating status and parameters of each device on the platform, such as motor speed, torque, battery SOC, power distribution, etc., are continuously monitored, and the monitored status data is fed back to the strategy model in the dSPACE real-time simulator in real time as the basis for the next calculation.

[0094] Step 9: Based on the actual performance and data feedback during the test, the energy management strategy model is optimized and adjusted online, especially for complex working conditions and abnormal situations. Through the self-improvement capability of deep reinforcement learning, the strategy parameters are dynamically adjusted to adapt to different operating conditions and working conditions.

[0095] Step 10: After completing a certain period of testing, stop the torque output of the magnetic powder brake, return the system to a no-load state, gradually stop the operation of the motor and the clutch, turn off the AC power and lithium battery power switches, and ensure the safe shutdown of the experimental platform. At the end of the test, conduct a comprehensive analysis of the test data to evaluate the economy, emissions, and effectiveness of the energy management strategy. Evaluation indicators include but are not limited to power system stability indicators, battery charge and discharge state changes, power distribution status of each power equipment, and overall energy consumption and emission levels.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] Through the application of the present invention, from the construction level, the data-driven method can learn and adapt from the actual operation data, without the need for in-depth understanding of the system mechanism, and has better self-adaptation ability; it can handle a variety of operating conditions and working conditions, so that the constructed strategy has better generalization ability; it does not need to establish an accurate mathematical model, reducing the complexity and cost of model development. From the verification level, the simulation speed, accuracy and credibility are improved, and the interactive communication between the energy management strategy model and the semi-physical simulation platform can more realistically reflect the performance of the energy management strategy in actual application, and optimize the energy management strategy according to the results obtained by test verification.

[0098] 1. The construction and verification of energy management strategies based on deep reinforcement learning not only relies on traditional model-driven methods, but also combines the characteristics of data-driven. It can learn and adapt from actual operation data and has better adaptability and generalization capabilities.

[0099] 2. The energy management strategy based on deep reinforcement learning can respond to changes in the ship's operating status in real time, and automatically adjust the strategy parameters through continuous learning and optimization processes to adapt to diverse operating conditions and working conditions, thereby quickly achieving the optimal energy allocation decision and improving the flexibility and response speed of the system.

[0100] 3. By combining the energy management strategy based on deep reinforcement learning with , the present invention provides a new verification method that can more realistically reflect the performance of the energy management strategy in practical applications and improve the accuracy and reliability of verification.

[0101] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A ship energy management strategy construction and verification method based on deep reinforcement learning, characterized in that: The invention is applied to a ship energy management strategy construction and verification system based on deep reinforcement learning. The main body of the system is a hybrid ship semi-physical simulation platform, which at least includes a simulated hybrid power device and a measurement and control system, wherein the simulated hybrid power device is used to simulate the ship load and the actual working condition of the ship; the measurement and control system is used for data collection, and a ship energy management strategy construction and verification method based on deep reinforcement learning is stored thereon; the method comprises the following steps: Collect and preprocess the operating data of a ship during a certain voyage, divide the preprocessed data into a training set and a validation set according to a preset ratio, and randomly select data from the training set to generate a test set; Build a ship energy management strategy model based on deep reinforcement learning according to operation data; The ship energy management strategy model is trained using the data from the training set, and the performance of the model is evaluated and optimized using the data from the validation set and test set; The actual working conditions of the ship are simulated through the hybrid ship semi-physical simulation platform, and the simulation data is collected and used as the input of the ship energy management strategy model; Through the ship energy management strategy model, deep reinforcement learning is performed based on simulation data to output energy allocation strategies and corresponding action instructions for controlling the hybrid ship semi-physical simulation platform; Continuously collect data from the hybrid ship semi-physical simulation platform that adjusts control according to the energy distribution strategy and use the collected data as feedback data; The ship energy management strategy model is optimized and adjusted according to the feedback data, and the strategy parameters are dynamically adjusted through deep reinforcement learning to generate a dynamically adjusted ship energy management strategy model to adapt to the different operating conditions and actual working conditions of the actual ship.

2. The method for constructing and verifying a ship energy management strategy based on deep reinforcement learning according to claim 1 is characterized in that: The operating data includes at least the ship's speed, heading, wave conditions, wind speed, wind direction, engine speed, engine output power, battery status and fuel consumption.

3. The method for constructing and verifying a ship energy management strategy based on deep reinforcement learning according to claim 1 is characterized in that: The specific preprocessing includes: data cleaning, feature selection, and data conversion of the operating data, filling the missing values ​​in the operating data through the K nearest neighbor interpolation algorithm, and normalizing the outliers in the operating data using the Z score.

4. The method for constructing and verifying a ship energy management strategy based on deep reinforcement learning according to claim 1 is characterized in that: The method for constructing the ship energy management strategy model is as follows: define the state space and action space of the deep reinforcement learning state space and all possible actions that the agent can perform in the ship energy management strategy model, and define a reward function for evaluating the immediate feedback obtained after the agent performs a certain action.

5. The method for constructing and verifying a ship energy management strategy based on deep reinforcement learning according to claim 4 is characterized in that: The method of defining the state space is as follows: The power requirement P of the ship in the whole working condition demand , engine output power P out , battery state of charge SOC, speed V, wind speed U, course over ground COG, wave conditions and wind direction F are used as the state space S of deep reinforcement learning, expressed as: S={P demand ,SOC,P out ,V,U,COG,W,F} The method of defining the action space is specifically expressed as: A={IEP,DEP,CB,DB} Among them, A is the action space, IEP is to increase the engine power output, DEP, is to reduce the engine power output, CB is to adjust the battery charging state, and DB is to adjust the battery discharging state.

6. The method for constructing and verifying a ship energy management strategy based on deep reinforcement learning according to claim 5 is characterized in that: The rewards set in the reward function include at least a combustion efficiency reward inversely proportional to fuel consumption, a fluctuation reward inversely proportional to power fluctuation, an energy storage efficiency reward proportional to the charging and discharging efficiency of supercapacitors and batteries, and an environmental impact reward inversely proportional to emissions; the reward function is expressed as: R(S,A)=α*FuelEff_reward+β*Stab_reward+γ*ESS_reward+δ*Env_reward Among them, R(S,A) is the reward function based on the state space definition and the action space definition, α, β, γ and δ are weight coefficients, FuelEff_reward is the fuel efficiency reward, Stab_reward is the volatility reward, ESS_reward is the energy efficiency storage reward, and Env_reward is the environmental impact reward.

7. The method for constructing and verifying a ship energy management strategy based on deep reinforcement learning according to claim 1 is characterized in that: The selected deep learning reinforcement algorithm is one or more of DQN, DDPG, A3C and PPO algorithms.

8. A ship energy management strategy construction and verification system based on deep reinforcement learning, characterized by: The main body of the system is a hybrid ship semi-physical simulation platform, which at least includes a simulated hybrid power device and a measurement and control system. The simulated hybrid power plant is used to simulate the ship load and actual ship working conditions; The measurement and control system is used for data collection, and the measurement and control system includes at least a speed torque sensor, a battery charge factor collector, a voltage and current sensor, a dSPACE real-time simulator, an information acquisition board and a drive board; wherein the speed torque sensor is used to monitor and collect the speed and torque data of the simulated ship main engine and the shaft belt motor in real time; the dSPACE real-time simulator is used to receive signals from the speed torque sensor, the battery charge factor collector, and the voltage and current sensors, and perform analog-to-digital conversion, processing operations and power distribution control, and stores thereon the ship energy management strategy construction and verification method based on deep reinforcement learning as described in any one of claims 1-7, and generates a dynamically adjusted ship energy management strategy model based on this method to adapt to the different operating conditions and actual working conditions of the actual ship.

9. The ship energy management strategy construction and verification system based on deep reinforcement learning according to claim 8 is characterized in that: The simulated hybrid power device includes at least two motors, a gear reducer, a clutch, a magnetic powder brake, a supercapacitor, a battery and a control device; among them, one of the two motors simulates the ship's main engine and the other simulates the shaft belt motor; the control device includes AC / DC and DC / DC converters and a DC / AC inverter, which are used to change the power output of the simulated shaft belt motor by adjusting its input voltage and current; the magnetic powder brake is used to realize the ship load and simulate the actual working conditions of the ship.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images