Automatic adjustment method for main engine control parameter of vessel, automatic adjustment program, and automatic adjustment system

Abstract

To provide an automatic adjustment method, an automatic adjustment program, and an automatic adjustment system for a vessel carrying out automatic adjustment of a control parameter for a main engine based on a derived optimum control parameter by carrying out reinforcement learning in an environment reflecting circumstances unique to a vessel.SOLUTION: A propulsion model composed of a hull of a vessel, a propeller, and a main engine are set as an environment for reinforced learning, wave conditions effecting the propulsion model is acquired, an initial value of a control parameter set by a control agent and the wave conditions are applied to the propulsion model, a simulation of the propulsion model for the plurality of wave conditions are carried out in a plurality of evaluation steps in one episode, a reward value based on a reward function for a result of the simulation is calculated, the control parameter is repeatedly changed by updating a measure based on a value function of the reward value and a main engine state of the main engine, and the main engine is controlled by deriving an optimum control parameter based on an optimum measure where the reward value becomes maximum.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an automatic adjustment method, an automatic adjustment program, and an automatic adjustment system for main engine control parameters of a ship, which automatically adjusts the main engine control parameters according to a purpose based on the results of machine learning. [Background technology]

[0002] As the demand for decarbonization increases for ships, the demand for automatic ship operation and safe operation is also increasing. In relation to these issues, for example, the issue of reducing GHG (greenhouse gas) emissions requires main engine control that minimizes fuel consumption rate according to the encountered sea conditions, the issue of maneuverability in rough seas requires main engine control that ensures safety during operation with reduced main engine output, and the issue of autonomous ship operation requires main engine control that takes into account safety and efficiency without relying on human labor. In this way, the importance of main engine control, which is the core of the propulsion system, is increasing. Here, Patent Document 1 discloses a control device for an internal combustion engine that switches between a first operation process that operates an operating part of an internal combustion engine with an operating amount calculated by a first operation process using previously adapted data, and a second operation process that operates an operating part of an internal combustion engine with an operating amount calculated by a second operation process using related specified data that is adapted by reinforcement learning while the vehicle is traveling, and also calculates some of the operating amounts by the first operation process, depending on whether the vehicle is traveling with an automatic accelerator or a manual accelerator, and further that continues to calculate some of the operating amounts by the first operation process even while the second operation process is being executed. Patent document 2 discloses a control policy learning and vehicle control method based on reinforcement learning without active exploration, which includes the steps of applying a passive actor-critic reinforcement learning method to passively collected data related to the operation of a vehicle in order to learn a control policy configured to control the vehicle for the purpose of performing the vehicle operation at the lowest expected cumulative cost, and controlling the vehicle according to the control policy to perform the vehicle operation. Patent Document 3 discloses a learning device having multiple individual learning units, each of which is equipped with a planner that generates information that specifies the behavior of an acting subject corresponding to the individual learning unit, and a reward derivation unit that derives a reward by evaluating evaluated information including feedback information obtained from a simulator by inputting information based on the information that specifies the behavior of the acting subject into the simulator, the planner performs reinforcement learning based on the reward derived by the reward derivation unit, and at least two of the multiple individual learning units have different actions that maximize the reward derived by the reward derivation unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-116740 A [Patent Document 2] JP 2019-31268 A [Patent Document 3] JP 2020-35221 A Summary of the Invention [Problem to be solved by the invention]

[0004] None of Patent Documents 1 to 3 aims to contribute to improving ship safety or reducing GHG emissions by performing reinforcement learning in an environment that reflects circumstances specific to a ship. Therefore, an object of the present invention is to provide an automatic adjustment method, automatic adjustment program, and automatic adjustment system for the main engine control parameters of a ship, which contributes to improving the safety of the ship and reducing GHG emissions by performing reinforcement learning in an environment that reflects the specific circumstances of the ship and automatically adjusting the main engine control parameters based on the derived optimal control parameters. [Means for solving the problem]

[0005] A method for automatically adjusting control parameters of a ship's main engine corresponding to the method described in claim 1 is a method for automatically adjusting the control parameters of the ship's main engine by reinforcement learning using a computer, which is characterized in that a propulsion model consisting of the ship's hull, propeller, and main engine is set as an environment for reinforcement learning, wave conditions acting on the propulsion model are obtained, the initial values ​​of the control parameters set by the control agent and the wave conditions are applied to the propulsion model, a simulation of the propulsion model for each of a plurality of wave conditions is performed in a plurality of evaluation steps in one episode, a reward value based on a reward function for the simulation results is calculated, and the policy is updated based on a value function based on the reward value and the main engine state, and the control parameters are changed by repeating the process, and optimal control parameters are derived based on the optimal policy that maximizes the reward value to control the main engine. According to the present invention as described in claim 1, reinforcement learning is performed in an environment that reflects the circumstances specific to the ship, and the control parameters of the main engine are automatically adjusted based on optimal control parameters derived taking into account a reward function specific to the ship, thereby contributing to improving the safety of the ship and reducing GHG emissions.

[0006] The present invention described in claim 2 is characterized by having a wave condition acquisition step for acquiring set wave conditions, a parameter initial value acquisition step for acquiring initial values ​​of control parameters, a simulation step for simulating a propulsion model, a reward value calculation step for calculating a reward value for the simulation result for each evaluation step, a value function calculation step for calculating a value function based on the reward value and the main engine state of the propulsion model, a policy update step for updating a policy based on the calculation result of the value function and changing the control parameters, a repeat step for repeating the simulation step, reward value calculation step, value function calculation step, and policy update step based on the changed control parameters, and an optimal policy derivation step for deriving an optimal policy that maximizes the reward value. According to the present invention as defined in claim 2, the control agent can derive an optimal policy by using reinforcement learning in an environment that reflects circumstances specific to the ship, taking into account a reward function specific to the ship.

[0007] The present invention as described in claim 3 is characterized by having an operation data acquisition step for acquiring operation data of a ship, a data estimation step for estimating data for the propulsion model using a propulsion model and the operation data, a buffering step for saving the operation data and the propulsion model data together at predetermined intervals, a data analysis step for performing data analysis based on the saved operation data and the propulsion model data, a main engine condition evaluation step for evaluating the main engine condition at predetermined intervals based on the results of the data analysis, an input value generation step for generating input values ​​to be input into a control agent based on the evaluation results of the main engine condition, an optimal control parameter derivation step for deriving optimal control parameters of the main engine based on the optimal measure and input values ​​derived in the optimal measure derivation step, and a control reflection step for repeatedly reflecting the optimal control parameters in the control of the main engine at predetermined intervals. According to the present invention as defined in claim 3, the control parameters of the main engine can be automatically adjusted based on the results of reinforcement learning in order to meet the objectives of improving the safety of the ship, reducing GHG emissions, and the like.

[0008] The present invention as recited in claim 4 is characterized in that the wave conditions are the frequency, wave direction, and wave height of waves that the ship encounters. According to the present invention as set forth in claim 4, the wave conditions used in the reinforcement learning can be set with high accuracy, taking into consideration the effects on the ship and the main engine.

[0009] The present invention as recited in claim 5 is characterized in that the reward function is determined based on at least one of maintaining ship speed, suppressing fluctuations in fuel input amount, suppressing fluctuations in main engine load, and optimizing the main engine fuel consumption rate. According to the present invention as set forth in claim 5, a reward function specific to a ship can be designed from the perspective of maintaining ship speed, suppressing fluctuations in fuel input amount, suppressing fluctuations in main engine load, or optimizing the main engine fuel consumption rate, and a reward value based on the reward function can be calculated.

[0010] The present invention as recited in claim 6 is characterized in that the reward function is designed taking into consideration the main engine torque, the main engine rotation speed, and the propeller inflow speed. According to the present invention as set forth in claim 6, it is possible to design a reward function taking into consideration the main engine torque, the main engine rotation speed, and the propeller inflow speed, and to calculate a reward value.

[0011] The present invention as set forth in claim 7 is characterized in that one of trust region policy optimization (TRPO) and proximal policy optimization (PPO) is used as a policy for changing a control parameter. According to the present invention as set forth in claim 7, it is possible to ensure the stability of learning by imposing constraints on the update of the policy function.

[0012] The present invention as recited in claim 8 is characterized in that, when deriving an optimal policy in the optimal policy derivation step, a threshold value of a reward value or a limited number of episodes is used. According to the present invention as set forth in claim 8, when the reward value reaches a threshold value or the number of episodes reaches a limit value, it is determined that learning is completed, and an optimal policy can be appropriately derived.

[0013] The present invention as described in claim 9 is characterized in that at least one of the calculation results using a simulation model of the ship, actual measurement data of the actual ship, and calculation results using a digital twin of the ship's main engine based on the measurement data of the actual ship is used as operation data. According to the present invention as set forth in claim 9, the data of the propulsion model can be estimated using the operation data, and the data can be analyzed to improve the accuracy of evaluating the main engine condition.

[0014] The present invention as set forth in claim 10 is characterized in that, when deriving the optimal control parameters, the response of the fuel pump and the response of the main engine rotation speed as the main engine state, and the change status of the control parameters are taken into consideration. According to the present invention as defined in claim 10, the optimal control parameters for the main engine can be appropriately derived.

[0015] The present invention as recited in claim 11 is characterized in that the control parameter of the main engine is any one of a PID gain of an electronic governor of the main engine, a main engine command speed, and a fuel injection timing. According to the present invention as set forth in claim 11, the control parameters of the main engine can be automatically adjusted with high accuracy in accordance with the purpose.

[0016] The present invention as described in claim 12 is characterized in that actual measurement data of an actual ship is obtained as operation data, the actual measurement data is applied to a propulsion model, the acting wave conditions are estimated using the propulsion model, and the main engine state is estimated using the propulsion model to derive optimal control parameters for the main engine. According to the present invention as set forth in claim 12, the wave conditions are estimated by applying actual measurement data to the propulsion model, and the main engine state is estimated, so that the optimal control parameters can be derived more appropriately.

[0017] An automatic adjustment program for main engine control parameters of a ship corresponding to the program described in claim 13 is a program for automatically adjusting the control parameters of the main engine of a ship by reinforcement learning, and is characterized in that it causes a computer to execute a wave condition acquisition step, a simulation step, a reward value calculation step, a value function calculation step, a policy update step, a repetition step, and an optimal policy derivation step in a method for automatically adjusting the main engine control parameters of a ship. According to the present invention as set forth in claim 13, the wave condition acquisition step and simulation step, etc., performed when the control agent performs learning can be accurately executed by a program, and an optimal policy can be derived.

[0018] The automatic adjustment program for main engine control parameters of a ship corresponding to the program described in claim 14 is a program for automatically adjusting the control parameters of the main engine of a ship by reinforcement learning, and is characterized in that it causes a computer to execute an operation data acquisition step, a data estimation step, a buffering step, a data analysis step, a main engine condition evaluation step, an input value generation step, an optimal control parameter derivation step, and a control reflection step in a method for automatically adjusting the main engine control parameters of a ship. According to the present invention as set forth in claim 14, the operation data acquisition step and data estimation step, etc., which are performed when the control parameters of the main engine are automatically adjusted according to the purpose based on the results of learning by the control agent, can be accurately executed by a program.

[0019] An automatic adjustment system for main engine control parameters of a ship corresponding to the system described in claim 15 is a system for automatically adjusting the control parameters of the main engine of a ship by reinforcement learning, and is characterized in that it comprises a computer, an input means and an output means for the computer, and causes the computer to execute a wave condition acquisition step, a simulation step, a reward value calculation step, a value function calculation step, a policy update step, a repetition step, and an optimal policy derivation step in the method for automatically adjusting the main engine control parameters of a ship, and outputs the optimal policy. According to the present invention as set forth in claim 15, the control agent can output an optimal policy by using reinforcement learning in an environment that reflects circumstances specific to the ship, taking into account a reward function specific to the ship.

[0020] The automatic adjustment system for main engine control parameters of a ship corresponding to the system described in claim 16 is a system for automatically adjusting the control parameters of the main engine of a ship by reinforcement learning, and is equipped with a computer, a ship operation data acquisition means for acquiring input information for the computer, and a control means for controlling the main engine, and is characterized in that the computer executes the operation data acquisition step, data estimation step, buffering step, data analysis step, main engine condition evaluation step, input value generation step, optimal control parameter derivation step, and control reflection step in the method for automatically adjusting the main engine control parameters of a ship, and applies the derived optimal control parameters to the control means for each evaluation step time. According to the present invention as set forth in claim 16, the control parameters of the main engine can be automatically adjusted based on the results of reinforcement learning in order to meet the objectives of improving the safety of the ship, reducing GHG emissions, and the like. Effect of the Invention

[0021] According to the method for automatically adjusting the main engine control parameters of a ship of the present invention, reinforcement learning is performed in an environment that reflects the circumstances specific to the ship, and the main engine control parameters are automatically adjusted based on optimal control parameters derived by taking into account a reward function specific to the ship, thereby contributing to improving the safety of the ship and reducing GHG emissions.

[0022] In addition, if the method has a wave condition acquisition step for acquiring set wave conditions, a parameter initial value acquisition step for acquiring initial values ​​of control parameters, a simulation step for simulating the propulsion model, a reward value calculation step for calculating a reward value for the simulation results for each evaluation step, a value function calculation step for calculating a value function based on the reward value and the main engine state of the propulsion model, a policy update step for updating the policy based on the calculation result of the value function and changing the control parameters, a repetition step for repeating the simulation step, reward value calculation step, value function calculation step, and policy update step based on the changed control parameters, and an optimal policy derivation step for deriving an optimal policy that maximizes the reward value, the control agent can derive an optimal policy by taking into account a reward function specific to the ship through reinforcement learning in an environment that reflects the circumstances specific to the ship.

[0023] In addition, if the system has an operation data acquisition step for acquiring ship operation data, a data estimation step for estimating data for the propulsion model using the propulsion model and operation data, a buffering step for saving the operation data and propulsion model data together at predetermined intervals, a data analysis step for performing data analysis based on the saved operation data and propulsion model data, a main engine condition evaluation step for evaluating the main engine condition at predetermined intervals based on the results of the data analysis, an input value generation step for generating input values ​​to be input into the control agent based on the evaluation results of the main engine condition, an optimal control parameter derivation step for deriving optimal control parameters for the main engine based on the optimal measure and input values ​​derived in the optimal measure derivation step, and a control reflection step for repeatedly reflecting the optimal control parameters in the control of the main engine at predetermined intervals, the main engine control parameters can be automatically adjusted based on the results of reinforcement learning in order to suit purposes such as improving ship safety and reducing GHG emissions.

[0024] Furthermore, when the wave conditions are the frequency, wave direction, and wave height of the waves that the ship encounters, the wave conditions used in reinforcement learning can be set with high accuracy, taking into account their effects on the ship and main engine.

[0025] In addition, when the reward function is determined based on at least one of maintaining vessel speed, suppressing fluctuations in fuel input amount, suppressing main engine load fluctuations, and optimizing the main engine fuel consumption rate, a reward function specific to the ship can be designed from the perspective of maintaining vessel speed, suppressing fluctuations in fuel input amount, suppressing main engine load fluctuations, or optimizing the main engine fuel consumption rate, and a reward value based on the reward function can be calculated.

[0026] Furthermore, when designing the reward function, in the case where the main engine torque, the main engine speed, and the propeller inflow speed are taken into consideration, a reward function taking into consideration the main engine torque, the main engine speed, and the propeller inflow speed can be designed and the reward value can be calculated.

[0027] In addition, when using either trust region policy optimization (TRPO) or proximal policy optimization (PPO) as a policy for changing control parameters, it is possible to ensure the stability of learning by imposing constraints on the update of the policy function.

[0028] In addition, when deriving the optimal policy in the optimal policy derivation step, if a threshold value of the reward value or a limited number of episodes is used, learning is determined to be completed when the reward value reaches the threshold value or the number of episodes reaches the limited value, and the optimal policy can be appropriately derived.

[0029] In addition, when at least one of the calculation results using a simulation model of the ship, actual measurement data of the actual ship, and calculation results using a digital twin of the ship's main engine based on the measurement data of the actual ship is used as the operation data, the operation data can be used to estimate data for the propulsion model, and data analysis can be performed to improve the accuracy of evaluating the main engine condition.

[0030] Furthermore, when deriving the optimal control parameters, if the response of the fuel pump and the response of the main engine rotation speed as the main engine state, and the change status of the control parameters are taken into consideration, the optimal control parameters of the main engine can be appropriately derived.

[0031] Furthermore, when the control parameter of the main engine is any one of the PID gain of the main engine's electronic governor, the main engine command speed, and the fuel injection timing, the control parameter of the main engine can be automatically adjusted with high accuracy to suit the purpose.

[0032] In addition, when actual measurement data of the actual ship is obtained as operational data, the actual measurement data is applied to a propulsion model, the wave conditions acting through the propulsion model are estimated, and the main engine state is estimated using the propulsion model to derive the optimal control parameters of the main engine, the actual measurement data is applied to the propulsion model to estimate the wave conditions and the main engine state is estimated, so that the optimal control parameters can be more appropriately derived.

[0033] Furthermore, according to the automatic adjustment program for the main engine control parameters of a ship of the present invention, the wave condition acquisition step and simulation step, etc., performed by the control agent during learning can be accurately executed by the program, and the optimal policy can be derived.

[0034] Furthermore, according to the automatic adjustment program for the main engine control parameters of a ship of the present invention, the program can accurately execute steps such as operation data acquisition steps and data estimation steps when automatically adjusting the main engine control parameters to suit the purpose based on the results learned by the control agent.

[0035] In addition, according to the automatic adjustment system for main engine control parameters of a ship of the present invention, the control agent can output an optimal policy by taking into account a reward function specific to the ship through reinforcement learning in an environment that reflects the circumstances specific to the ship.

[0036] Furthermore, according to the automatic adjustment system for the main engine control parameters of a ship of the present invention, the main engine control parameters can be automatically adjusted based on the results of reinforcement learning to suit purposes such as improving the safety of the ship and reducing GHG emissions. [Brief description of the drawings]

[0037] [Figure 1] FIG. 1 is a flow diagram of the process up to deriving an optimal policy (reinforcement learning stage) in the automatic adjustment method for main engine control parameters of a ship according to an embodiment of the present invention. [Diagram 2] Flow diagram of the main engine control (execution stage) based on the results of reinforcement learning in the automatic adjustment method for the main engine control parameters of the same ship. [Diagram 3] A diagram showing the equipment used for reinforcement learning in the automatic adjustment system for the ship's main engine control parameters. [Figure 4] A diagram showing the equipment used to execute main engine control in the automatic adjustment system for main engine control parameters of the ship. [Diagram 5] Overview of the reinforcement learning environment [Figure 6] A flowchart of the contents of Figure 5. [Figure 7] A diagram showing an example of progress in the reinforcement learning process. [Figure 8] Figure showing the effect of the governor gain settings on the response of output items [Figure 9]A diagram showing the transformation function f(x) with different scales [Figure 10] The figure shows the results of the output component response after the equalization transformation. [Figure 11] Comparison of simulation results between this main engine control and conventional main engine control [Figure 12] Image of the automatic adjustment system for the ship's main engine control parameters applied to an actual ship [Figure 13] A schematic flow diagram of the automatic adjustment algorithm by the automatic adjustment system shown in FIG. [Figure 14] FIG. 1 is a diagram showing another embodiment of the automatic adjustment system for main engine control parameters of a ship; [Figure 15] FIG. 13 is a diagram showing yet another embodiment of the automatic adjustment system for main engine control parameters of a ship; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] An automatic adjustment method, an automatic adjustment program, and an automatic adjustment system for main engine control parameters of a ship according to an embodiment of the present invention will be described. Figure 1 is a flow diagram of the process up to deriving an optimal policy (reinforcement learning stage) in the method for automatically adjusting a ship's main engine control parameters, Figure 2 is a flow diagram of the process for performing main engine control based on the results of reinforcement learning (execution stage) in the method for automatically adjusting a ship's main engine control parameters, Figure 3 is a diagram showing equipment used for reinforcement learning in the system for automatically adjusting a ship's main engine control parameters, and Figure 4 is a diagram showing equipment used to execute main engine control in the system for automatically adjusting a ship's main engine control parameters. In the following description, the flow shown in FIG. 1 may be referred to as the reinforcement learning stage, and the flow shown in FIG. 2 may be referred to as the execution stage.

[0039] The method for automatically adjusting the control parameters of a ship's main engine is a method for automatically adjusting the control parameters of a ship's main engine by reinforcement learning using a computer. The system for automatically adjusting the control parameters of a ship's main engine is a system that uses the method for automatically adjusting the control parameters of a ship's main engine, and includes a system main body for the reinforcement learning stage shown in Figure 3 and a system main body for the execution stage shown in Figure 4. The system main body for the reinforcement learning stage includes a computer 10, and an input means 20 and an output means 30 for the computer 10. The computer 10 has a wave condition acquisition unit 10A, a simulation unit 10B, a reward value calculation unit 10C, a value function calculation unit 10D, a strategy update unit 10E, a repeat unit 10F, and an optimal strategy derivation unit 10G, and is connected to a first memory unit 10H, a second memory unit 10J, and a control unit 10K. The first memory unit 10H and the second memory unit 10J store an automatic adjustment program for the main engine control parameters of the ship, a propulsion model, a reinforcement learning model, and the like. The control unit 10K performs overall control when the computer 10 performs the processing of the reinforcement learning stage. The input means 20 is, for example, a keyboard, a mouse, a touch panel, etc., and is used to input the wave conditions, the initial values ​​of the control parameters, etc. The output means 30 is a communication device, an external storage device, etc., and is used to output the optimal strategy, etc. to the outside. The system main body for the execution stage includes a computer 40, an input means 50 for inputting information such as operation data and optimal measures to the computer 40, and a control means 60 for controlling the main engine. The computer 40 has an operation data acquisition unit (operation data acquisition means) 40A, a data estimation unit 40B, a buffering unit 40C, a data analysis unit 40D, a main engine state evaluation unit 40E, an input value generation unit 40F, a control agent unit 40G, and an optimal control parameter derivation unit 40H, and is connected to a first memory unit 40J, a second memory unit 40K, and a control unit 40L. The first memory unit 40J stores an automatic adjustment program for the main engine control parameters of the ship, a propulsion model, a reinforcement learning model, and the like. The second memory unit 40K is a buffer (buffer storage device). The control unit 40L performs overall control when the computer 40 performs processing of the execution stage. The input means 50 is, for example, a keyboard or a mouse, and is used to input operation data, optimal measures, and the like. Incidentally, the input means 50 does not necessarily have to be mechanical (hardware) and may be realized by software.

[0040] Because the sea conditions and environments that ships encounter are extremely diverse, the reinforcement learning stage uses a type of machine learning method called reinforcement learning, which is suitable for problems in which the control agent learns to maximize rewards (improving ship safety) and problems that take into account long-term rewards (reducing GHG emissions from ships). The goal of a reinforcement learning algorithm is to determine the control action u that maximizes the reward collected over time. t = π * (S t The goal is to find the optimal policy for choosing the * determines the decision-making ability of the agent, and the system is in state S t = Action A if s t=a is the probability that α = α. The agent and the environment interact with each other to generate an experience consisting of a set of states, actions, and rewards. The agent uses it to improve its policy. There are many different algorithms for learning optimal policies, but recently, algorithms that combine ideas from deep learning, i.e., artificial neural networks (ANNs) and reinforcement learning, have become popular. The differences between these deep reinforcement learning (DRL) algorithms depend on how the agent learns and how the actions are selected. In the field of DRL, trust region policy optimization (TRPO) and proximal policy optimization (PPO) are two popular techniques that belong to the family of actor-critic algorithms. In TRPO, a critic that evaluates actions estimates a value function (state value function), and an actor that decides on an action updates the policy based on the trust region constraints to ensure the stability of learning. On the other hand, PPO also incorporates a similar actor-critic setup as TRPO, but PPO aims to balance stability and sample efficiency. This is achieved by clipping the probability ratio between the new policy and the old policy within a specified range. Both the TRPO and PPO algorithms are applicable to control problems with continuous / discrete action and state space. In the actor-critic algorithm, both the policy function, called the actor, and the value function (state-value function), called the critic, are approximated and learned using an ANN. A key element of any reinforcement learning implementation is the design of the reward function, which plays a key role in the learning process by providing the primary feedback signal for the reinforcement learning agent.

[0041] In the reinforcement learning stage shown in FIG. 1, first, a propulsion model consisting of a ship's hull, propeller, and main engine is set as an environment for reinforcement learning. Then, the set wave conditions are acquired by the wave condition acquisition unit 10A (S1: wave condition acquisition step). The acquired wave conditions are preferably the frequency, wave direction, and wave height of the waves that the ship encounters. This allows the wave conditions used in reinforcement learning to be set with high accuracy, taking into account their effects on the ship and the main engine. Note that the wave conditions can be obtained by processing those obtained from actual sea state information, but it is also possible to use wave conditions measured on an actual ship, or wave conditions estimated from the waves acting on the propulsion model and the response of the propulsion model. Next, the computer 10 acquires the initial values ​​of the control parameters input from the input means 20 (S2: parameter initial value acquisition step). The control parameters are, for example, the proportional, integral, and derivative gains of a PID governor. Next, the simulation unit 10B simulates the propulsion model (S3: simulation step). In the simulation, it takes several minutes of analysis time per step (one evaluation step), and several hours of analysis time per episode.

[0042] Next, the reward value calculation unit 10C calculates a reward value based on the reward function for the result of the simulation for each evaluation step (S4: reward value calculation step). The reward function used in the reward value calculation step S4 is designed before learning begins. The reward function defines the mapping between state-action pairs and the corresponding rewards. Designing an appropriate reward function is a key challenge in reinforcement learning, as it directly affects the behavior and learning performance of the control agent. A well-defined reward function should provide meaningful and useful feedback, encouraging the control agent to achieve desired goals or objectives. The reward function is preferably determined based on at least one of maintaining vessel speed, suppressing fluctuations in fuel input, suppressing main engine load fluctuations, and optimizing the main engine fuel consumption rate. This allows a reward function specific to a ship to be designed from the perspective of maintaining vessel speed, suppressing fluctuations in fuel input, suppressing main engine load fluctuations, or optimizing the main engine fuel consumption rate, and a reward value based on the reward function can be calculated. In this case, the reward function (adjustment objective) is designed as follows. A) Maintaining ship speed: The higher the ship speed, the higher the reward. B) Suppression of fluctuations in fuel input amount (fuel pump mark): The smaller the fluctuations in the fuel pump mark, the higher the reward. C) Suppression of main engine load fluctuations: In order to improve fuel consumption rate and enhance safety near the load limit, both main engine torque fluctuations and rotation speed fluctuations are suppressed, and the smaller the main engine load fluctuation, the higher the reward. D) Optimization of main engine fuel consumption: By changing the fuel injection timing and injection rate, the lower the fuel consumption rate, the higher the reward. E) The above A) to D) are appropriately combined. In addition, it is more preferable to consider the main engine torque, the main engine speed, and the propeller inflow speed when designing the reward function. This makes it possible to design a reward function that takes the main engine torque, the main engine speed, and the propeller inflow speed into consideration, and to calculate the reward value.

[0043] Next, value function calculation unit 10D calculates a value function based on the reward value and the main engine state of the propulsion model (S5: value function calculation step). Next, the policy update unit updates the policy based on the calculation result of the value function to change the control parameters (S6: policy update step). As a policy for changing the control parameters, it is preferable to use trust region policy optimization (TRPO) or proximal policy optimization (PPO). This makes it possible to ensure the stability of learning by imposing constraints on the update of the policy function.

[0044] Next, the computer 10 judges whether the evaluation step is completed or not (S7: evaluation step completion judgment step). For example, if the analysis time of one episode reaches a set value (e.g., 240 minutes), it judges that the evaluation step is completed, and if not, it judges that the evaluation step is not completed. If it is determined in the evaluation step completion determination step S7 that the evaluation step is not completed, the repeating unit 10F repeats the simulation step, the reward value calculation step S4, the value function calculation step S5, and the policy update step S6 based on the changed control parameters (S8: repeating step). On the other hand, when it is determined in the evaluation step completion determination step S7 that the evaluation step is completed, the computer 10 determines whether or not the learning is completed. If it is determined that the learning is not completed, the simulation time is reset and initial values ​​of the wave conditions and the like are set, and the process returns to the simulation step S3. If it is determined that the learning is completed, the optimal policy derivation unit 10G derives an optimal policy that maximizes the reward value (S9: optimal policy derivation step). In the optimal policy derivation step S9, it is preferable to use a threshold value of the reward value (accumulated reward) or a limited number of episodes when deriving the optimal policy. Thereby, when the reward value reaches the threshold value or the number of episodes reaches the limited value, it is determined that the learning is completed, and the optimal policy can be appropriately derived. In this way, in the reinforcement learning stage, the control agent can derive an optimal policy by taking into account a reward function specific to the ship through reinforcement learning in an environment that reflects the circumstances specific to the ship. The derived optimal policy is output from the output means 30 to the system main body for the execution stage.

[0045] Here, we will describe PID control of an electronic governor using reinforcement learning to suppress main engine load fluctuations. Main engine speed control is important, especially for ships sailing in waves, because fluctuations in propeller torque caused by waves can lead to load fluctuations in the main engine, which can have undesirable effects such as increased wear on main engine parts and reduced performance efficiency. Proportional-integral-derivative (PID) governors are widely used to regulate the main engine speed. In order to keep the main engine speed constant, PID governors continuously adjust the amount of fuel injected into the engine cylinder in response to the change in speed caused by the fluctuations in propeller torque. Therefore, the quality of control, and therefore the performance of the main engine, is determined by a preset control parameter, K p , K i , K d It depends on the proportional, integral and derivative gains of the PID governor. Currently, these PID gains are tuned during main engine commissioning and remain fixed, which can result in sub-optimal control in the presence of wave-induced disturbances. To address this issue, this embodiment uses a new reinforcement learning method to simultaneously reduce main engine speed fluctuations and fuel supply fluctuations. In this reinforcement learning method, the control agent learns optimal PID gains taking into account the objective of suppressing main engine load fluctuations, the main engine state, and various wave conditions. The novelties of this reinforcement learning method include an optimal policy (gain setting, etc.), a design policy for the objective function (reward function), a way of providing input information that can explicitly consider the main engine state and disturbances due to waves, and the setting of a reinforcement learning environment for learning a set of variable parameters.

[0046] Specific examples of reinforcement learning stages are shown in Figures 5 and 6. Figure 5 is a schematic diagram of the entire reinforcement learning environment, and Figure 6 is a flow chart of the contents of Figure 5. In this example, we consider a simulation model of a Hull-Propeller-Engine (HPE) propulsion system, disturbances due to wave conditions, and the mean, variance, and period (S t ), reward calculation (r t ), and a set of PID gains (a t) The control agent is trained in a reinforcement learning environment, with each episode including the following steps, where an "episode" is a time series of simulations of the propulsion system, with each episode lasting several hours in this example. Step 1) After starting learning, first initialize any initial conditions of the propulsion system. The initial conditions are the main engine speed setting and the ship speed, etc. Step 2) Initialize a set of arbitrary ocean states Λ (the ocean states switch at regular time intervals λ1, λ2, … λ_(tn) during an episode). Step 3) Initial observation state S t Receive. Step 4) At each step, - Action a according to current policy t Select . - Action a t Execute and get reward r t and new state s t+1 is observed. - Save each intermediate state (s t , a t , r t , s t+1 ). Step 5) Update the value function. Step 6) Update the policy of the control agent. Note that the process of updating the policy and value function will vary depending on the reinforcement learning algorithm used. Step 7) Calculate the reward value (accumulated reward). Step 8) Learning is completed when the reward value (accumulated reward) reaches the target or the number of episodes reaches a limit. Note that, while this specific example is based on a simulation, it is also possible to apply learning using actual measurement data from an actual ship in operation, or to use a hybrid of actual measurement data from the actual ship and calculation data from the main engine digital twin.

[0047] An example of the progress of reinforcement learning in this specific example is shown in Figure 7. The vertical axis of Figure 7 is the reward for each episode (Episode Reward), and the horizontal axis is the number of episodes (Episode No.). Figure 7 shows the results of performing reinforcement learning three times with different initial conditions using the TRPO algorithm. It can be seen that at the end of the learning stage, the reward obtained is about 2 to 3 times larger than in the early learning stage.

[0048] Next, a specific example of reward function design will be described. The reward function is based on the effort of the electronic governor to suppress both the main engine torque fluctuation and the main engine speed fluctuation due to the encounter wave disturbance, i.e., to suppress the main engine power output fluctuation, and the following transformation is performed to express the contribution of the main engine state variable fluctuation to the power output fluctuation. A) Engine output is P e =2πQ e n e It is expressed as: Here, Q e is the main engine torque, n e is the main engine RPM. B) Apply the 'log' function to both sides of the above equation. log(P e )=log(2π)+log(Q e )+log(n e ) C) take all derivatives with respect to the corresponding variables. dP e / P e0 =dQ e / Q e0 +dn e / n e0 D) If a small difference Δ is substituted for the differential d, we get the following equation (1).

number

number

[0049] Figure 8 shows the output items (h p and n e ) effect of the response to h p Response,Figure 8(b) shows n e is the response. In Fig. 8, the propeller inflow velocity fluctuation δu p Fuel pump mark fluctuation due to δh p and rotation speed fluctuation δn e The response of the governor gain setting K p The effect of changing the n sp is the speed set point of the propeller shaft, e.g., n sp _70 is n sp =70%. Fuel pump response δh p is the response of the main engine speed δn e Since the contributions to the output response are not equal, and if the output response is used as the reward, the learning algorithm will eventually reach the conclusion that switching off the governor is the best option. Therefore, in this example, the reward function is considered as the separate contributions of both components, as shown in equation (3) below.

number

[0050] FIG. 9 is a diagram showing the transformation function f(x) with different scales. To equalize the contributions of the components to the reward function, we use the following transformation (4) which is based on the use of the standard Gaussian distribution:

number

number

[0051] The execution stage will now be described. In the execution stage shown in Fig. 2, the control parameters of the main engine are automatically adjusted to meet the objectives based on the results of learning by the reinforcement learning method. As described above, the objectives are suppression of main engine load fluctuations, optimization of fuel consumption rate, maintenance of set ship speed, etc., and are expressed by a formula called a reward function, and the control parameters are automatically adjusted optimally so that the reward function value becomes high. First, the computer 40 acquires the ship operation data input from the input means 50 by the operation data acquisition unit 40A (S11: operation data acquisition step). As the operation data acquired in the operation data acquisition step S11, it is preferable to use one or a combination of two or more of the following: calculation results using a simulation model of the ship, actual measurement data of the actual ship, and calculation results using a digital twin of the ship's main engine based on the measurement data of the actual ship. This makes it possible to estimate data of the propulsion model using the operation data and perform data analysis to improve the accuracy of evaluating the main engine condition. Next, data estimation unit 40B estimates data of the propulsion model using the propulsion model and the operation data (S12: data estimation step). Then, the buffering unit 40C stores the operation data and the propulsion model data together at predetermined time intervals (S13: buffering step).

[0052] Next, the data analysis unit 40D performs data analysis based on the stored navigation data and propulsion model data (S14: data analysis step). The data analysis uses statistical analysis, Fourier transformation, etc. The analysis time is set to, for example, 5 to 20 minutes, so that changes in encounter waves can be taken into account. Next, the main engine status evaluation unit 40E evaluates the main engine status at predetermined time intervals based on the results of the data analysis (S15: main engine status evaluation step). The main engine status to be evaluated includes the response of the fuel pump and the response of the main engine rotation speed. Next, the input value generating unit 40F generates input values ​​to be input to the control agent based on the evaluation result of the main engine state (S16: input value generating step). The input values ​​are information (average value, fluctuation period, fluctuation amplitude characteristics) obtained by analyzing the main engine state within a set time, and this allows the influence of encounter waves, including fluctuation factors, to be implicitly taken into account. Next, the control agent unit 40G and the optimal control parameter derivation unit 40H derive optimal control parameters for the main engine based on the optimal policy derived in the optimal policy derivation step S9 and the input values ​​(S17: optimal control parameter derivation step). When deriving the optimal control parameters, it is preferable to consider the fuel pump response and the main engine rotation speed response as the main engine state, and the change status of the control parameters. This makes it possible to appropriately derive the optimal control parameters for the main engine. In addition, by setting the optimal control parameters for the main engine to one or more of the parameters targeted by electronic control, such as the PID gain of the electronic governor of the main engine, the main engine command speed, and the fuel injection timing, the control parameters of the main engine can be automatically adjusted with high accuracy according to the purpose. Next, the control means 60 repeatedly reflects the optimum control parameters in the control of the main engine at predetermined time intervals (S18: control reflection step).

[0053] After the control reflection step S18, it is determined whether or not a condition for ending the execution stage has been reached (S19: condition reaching determination step). In the automatic adjustment of the main engine control parameters in the execution stage, as long as input values ​​are generated for each evaluation step, operation data is acquired in the operation data acquisition step S11, and the main engine status step to the control reflection step S18 are repeated. Therefore, the condition in the condition reaching determination step S19 is when the main engine stops, when operation data cannot be acquired due to low load, etc. If it is determined in the condition achievement determination step S19 that the condition has not been achieved, the process returns to the operation data acquisition step S11. On the other hand, if it is determined in the condition reaching determination step S19 that the condition has been reached, the process of the execution stage is terminated.

[0054] In addition, in the operation data acquisition step S11, actual measurement data of the actual ship is acquired as operation data, the actual measurement data is applied to the propulsion model, the wave conditions acting on the propulsion model are estimated, and the wave conditions are estimated from the main engine state using the propulsion model. When deriving the optimal control parameters of the main engine in the optimal control parameter derivation step S17, the wave conditions are estimated by applying the actual measurement data to the propulsion model, and the wave conditions are estimated from the main engine state, so that the optimal control parameters can be derived more appropriately.

[0055] FIG. 11 is a diagram comparing the simulation results of the main engine control according to this embodiment (adaptive control) and the conventional main engine control (standard fixed control). The upper dotted line in Figure 11 shows the results of conventional main engine control (standard fixed control: standard PID fixed control), and the lower dotted line in Figure 11 shows the results of main engine control of this embodiment (adaptive control: control using adjusted control parameters). Simulation results show that the main engine control of this embodiment, using PID gains adjusted according to sea conditions, simultaneously suppresses fluctuations in main engine speed and fuel volume (pump mark), realizing safe operation of the main engine. It is also expected to contribute to improving the fuel efficiency of the main engine.

[0056] Figure 12 is an image diagram of the automatic adjustment system for a ship's main engine control parameters applied to an actual ship, and Figure 13 is a schematic flow chart of the automatic adjustment algorithm used by the automatic adjustment system shown in Figure 12. The data analysis section 40D shown in FIG. 12 provides estimates of a set of state variables S that characterize the engine state and the wave disturbances. The input information consists of sensor items and estimated items. If measurements cannot be made using sensors, a Kalman filter can be applied to perform complementary state estimation. Table 1 below lists the items used in data analysis, where "〇" indicates an item that can be measured using a sensor and "×" indicates an estimated item. Once the auto-tuning system is trained, the control agent automatically tunes the control parameters online according to the information (input values) provided by the data analysis unit 40D. [Table 1]

[0057] As described above, the method for automatically adjusting control parameters of a ship's main engine is a method for automatically adjusting the control parameters of the ship's main engine through reinforcement learning using a computer, in which a propulsion model consisting of the ship's hull, propeller, and main engine is set as an environment for reinforcement learning, wave conditions acting on the propulsion model are obtained, the initial values ​​of the control parameters set by the control agent and the wave conditions are applied to the propulsion model, a simulation of the propulsion model for each of a plurality of wave conditions is performed in a plurality of evaluation steps in one episode, a reward value based on a reward function for the simulation results is calculated, and the policy is updated based on the reward value and a value function based on the main engine state to change the control parameters, and the optimal control parameters are derived based on the optimal policy that maximizes the reward value to control the main engine. This enables reinforcement learning in an environment that reflects the specific circumstances of a ship, and automatically adjusts the main engine control parameters based on optimal control parameters derived by taking into account a reward function specific to the ship, thereby contributing to improving ship safety and reducing GHG emissions.

[0058] Furthermore, a part of the method for automatically adjusting the main engine control parameters of a ship can be executed using a program. The program for automatically adjusting the main engine control parameters of a ship is a program for automatically adjusting the control parameters of the main engine of the ship by reinforcement learning, and causes a computer 10 to execute a wave condition acquisition step S1, a simulation step S3, a reward value calculation step S4, a value function calculation step S5, a policy update step S6, a repeat step S8, and an optimal policy derivation step S9 in the method for automatically adjusting the main engine control parameters of a ship. This enables the wave condition acquisition step S1 and simulation step S3, etc., performed by the control agent during learning to be accurately executed by the program, enabling the optimal policy to be derived.

[0059] In addition, the automatic adjustment program for the ship's main engine control parameters causes the computer 40 to execute the operation data acquisition step S11, the data estimation step S12, the buffering step S13, the data analysis step S14, the main engine condition evaluation step S15, the input value generation step S16, the optimal control parameter derivation step S17, and the control reflection step S18 in the method for automatically adjusting the ship's main engine control parameters. This enables the program to accurately execute operations data acquisition step S11, data estimation step S12, and the like when automatically adjusting the main engine control parameters according to the purpose based on the results of the control agent's learning.

[0060] 14 is a diagram showing another embodiment of the automatic adjustment system for the main engine control parameters of a ship. Note that the same functional members as those in the above-mentioned embodiment are given the same reference numerals and the description thereof will be omitted. In the automatic adjustment system of this embodiment, a computer 70 is integrally provided with a function related to the machine learning stage and a function related to the execution stage, and has a first storage unit 71, a second storage unit 72, and a control unit 73. An input means 80, an output means 30, and a control means 60 are connected to the computer 70. Functions related to the machine learning stage include a wave condition acquisition unit 10A, a simulation unit 10B, a reward value calculation unit 10C, a value function calculation unit 10D, a strategy update unit 10E, a repeat unit 10F, and an optimal strategy derivation unit 10G, and functions related to the execution stage include a navigation data acquisition unit 40A, a data estimation unit 40B, a buffering unit 40C, a data analysis unit 40D, a main engine state evaluation unit 40E, an input value generation unit 40F, a control agent unit 40G, and an optimal control parameter derivation unit 40H. The first storage unit 71 stores an automatic adjustment program for the main engine control parameters of the ship, a propulsion model, a reinforcement learning model, and the like, and the second storage unit 72 is a buffer (buffer storage device). In addition, the control unit 73 performs overall control when the computer 70 performs the processing of the machine learning stage and the execution stage. Note that it is possible to use one function or omit one function of the overlapping functions related to the machine learning stage and the execution stage. The automatic adjustment system of this embodiment is mounted on a ship, for example, and both the reinforcement learning stage and the execution stage can be carried out on the ship.

[0061] 15 is a diagram showing still another embodiment of the automatic adjustment system for main engine control parameters of a ship. Note that the same functional members as those in the above-mentioned embodiment are given the same reference numerals and the description thereof will be omitted. In the automatic adjustment system of this embodiment, a system main body equipped with a computer 10 that performs processing related to the machine learning stage is installed at a land-based service center 1, and system main bodies equipped with a computer 40 that performs processing related to the execution stage are installed on ships A and B, which are connected via an information and communication network 90. ​​In addition, a database 100 in which data such as wave information and similar ship types is accumulated is also connected to the information and communication network 90, and the data stored in the database 100 can be used in reinforcement learning, etc. According to this embodiment, the control parameters of the main engines of ship A and ship B can be automatically adjusted according to purpose based on the results of reinforcement learning by the control agent at service center 1 using a method that maximizes rewards. It is also possible for service center 1 to obtain information on aging from ship A or ship B on a regular or irregular basis, revise the propulsion model, and provide a new optimal policy at each time. [Industrial Applicability]

[0062] INDUSTRIAL APPLICABILITY The present invention can be used for automatically adjusting main engine control parameters according to purpose, such as main engine speed control, in a ship in operation. [Explanation of symbols]

[0063] 10. Computers 20 Input Method 30 Output Method 40 Computer 40A Flight data acquisition unit (flight data acquisition means) 50 Input Method 60 Control Means 70 Computer 80 Input Method S1 Wave condition acquisition step S2 Parameter initial value acquisition step S3 Simulation Step S4 Reward value calculation step S5 Value function calculation step S6 Policy update step S8 Repeat step S9 Optimal policy derivation step S11 Operation data acquisition step S12 Data Estimation Step S13 Buffering Step S14 Data analysis step S15 Main engine condition evaluation step S16 Input value generation step S17 Optimal control parameter derivation step S18 Control reflection step

Claims

1. A method for automatically adjusting control parameters of a ship's main engine through reinforcement learning using a computer, comprising the steps of: setting a propulsion model consisting of the hull, propeller, and main engine of the ship as an environment for the reinforcement learning; obtaining wave conditions acting on the propulsion model; applying initial values ​​of the control parameters set by a control agent and the wave conditions to the propulsion model; simulating the propulsion model for each of a plurality of wave conditions in a plurality of evaluation steps in one episode; calculating a reward value based on a reward function for the results of the simulation; updating a policy based on a value function based on the reward value and the main engine state of the main engine, and repeatedly changing the control parameters; and deriving optimal control parameters based on the optimal policy that maximizes the reward value to control the main engine.

2. A wave condition acquisition step for acquiring the set wave conditions; a parameter initial value acquisition step of acquiring initial values ​​of the control parameters; a simulation step of performing the simulation of the propulsion model; a reward value calculation step of calculating the reward value for the result of the simulation for each of the evaluation steps; a value function calculation step of calculating the value function based on the reward value and the engine status of the propulsion model; a policy updating step of updating the policy based on a calculation result of the value function to change the control parameters; a repeating step of repeating the simulation step, the reward value calculation step, the value function calculation step, and the policy update step based on the changed control parameters; 2. The method for automatically adjusting main engine control parameters of a ship according to claim 1, further comprising an optimal policy deriving step of deriving the optimal policy that maximizes the reward value.

3. A navigation data acquisition step of acquiring navigation data of the ship; a data estimation step of estimating data of the propulsion model using the propulsion model and the operation data; a buffering step of collectively storing the operation data and the propulsion model data at predetermined time intervals; a data analysis step of performing a data analysis based on the stored operation data and the propulsion model data; a main engine status evaluation step of evaluating the main engine status at each predetermined time based on a result of the data analysis; an input value generating step of generating an input value to be input to the control agent based on the evaluation result of the main engine state; an optimal control parameter derivation step of deriving the optimal control parameters of the main engine based on the optimal policy derived in the optimal policy derivation step and the input values; 3. The method for automatically adjusting main engine control parameters of a ship according to claim 2, further comprising a control reflection step of repeating reflecting the optimal control parameters in the control of the main engine at every predetermined time.

4. 3. The method for automatically adjusting main engine control parameters of a ship according to claim 1 or 2, wherein the wave conditions include a frequency, a wave direction, and a wave height of waves that the ship encounters.

5. 3. The method for automatically adjusting main engine control parameters of a ship according to claim 1 or claim 2, characterized in that the reward function is determined based on at least one of maintaining ship speed, suppressing fluctuations in fuel input amount, suppressing fluctuations in main engine load, and optimizing the main engine fuel consumption rate.

6. 6. The method for automatically adjusting main engine control parameters of a ship according to claim 5, wherein main engine torque, main engine rotation speed, and propeller inlet speed are taken into consideration when designing the reward function.

7. The method for automatically adjusting main engine control parameters of a ship according to claim 1 or 2, characterized in that one of trust region policy optimization (TRPO) and proximity policy optimization (PPO) is used as the policy for changing the control parameters.

8. 3. The method for automatically adjusting main engine control parameters of a ship according to claim 2, wherein a threshold value of the reward value or a limited number of episodes is used when deriving the optimal policy in the optimal policy derivation step.

9. The method for automatically adjusting main engine control parameters of a ship as described in claim 3, characterized in that at least one of calculation results using a simulation model of the ship, actual measurement data of the actual ship, and calculation results using a main engine digital twin of the ship based on the measurement data of the actual ship is used as the operation data.

10. 4. The method for automatically adjusting main engine control parameters of a ship according to claim 3, characterized in that, when deriving the optimal control parameters, the response of the fuel pump and the response of the main engine rotation speed as the main engine state, and the change status of the control parameters are taken into consideration.

11. 4. The method for automatically adjusting main engine control parameters of a ship according to claim 3, characterized in that the control parameters of the main engine are one or more of a PID gain of an electronic governor of the main engine, a main engine command speed, and a fuel injection timing.

12. A method for automatically adjusting main engine control parameters of a ship as described in claim 3, characterized in that actual measurement data of the actual ship is obtained as the operation data, the actual measurement data is applied to the propulsion model, the wave conditions acting on the propulsion model are estimated, and the main engine state is estimated using the propulsion model to derive the optimal control parameters of the main engine.

13. A program for automatically adjusting a control parameter of a main engine of a ship by reinforcement learning, On the computer, 3. An automatic adjustment program for a ship's main engine control parameters, characterized in that it executes a wave condition acquisition step, a simulation step, a reward value calculation step, a value function calculation step, a policy update step, a repetition step, and an optimal policy derivation step in the automatic adjustment method for a ship's main engine control parameters as described in claim 2.

14. A program for automatically adjusting a control parameter of a main engine of a ship by reinforcement learning, On the computer, 4. An automatic adjustment program for a ship's main engine control parameters, characterized in that it executes an operation data acquisition step, a data estimation step, a buffering step, a data analysis step, a main engine condition evaluation step, an input value generation step, an optimal control parameter derivation step, and a control reflection step in the automatic adjustment method for a ship's main engine control parameters as described in claim 3.

15. A system for automatically adjusting control parameters of a ship's main engine by reinforcement learning, 3. An automatic adjustment system for a ship's main engine control parameters, comprising a computer, and input means and output means for the computer, and causing the computer to execute the wave condition acquisition step, simulation step, reward value calculation step, value function calculation step, strategy update step, repetition step, and optimal strategy derivation step in the method for automatically adjusting a ship's main engine control parameters as described in claim 2, and outputting the optimal strategy.

16. A system for automatically adjusting control parameters of a ship's main engine by reinforcement learning, 4. An automatic adjustment system for a ship's main engine control parameters, comprising a computer, an operation data acquisition means for the ship which acquires input information for the computer, and a control means for controlling the main engine, characterized in that the computer executes the operation data acquisition step, data estimation step, buffering step, data analysis step, main engine condition evaluation step, input value generation step, optimal control parameter derivation step, and control reflection step in the method for automatically adjusting a ship's main engine control parameters as described in claim 3, and applies the derived optimal control parameters to the control means for each evaluation step time.

Images