Dual-drive modeling method for intelligent ship maneuvering based on improved deep reinforcement learning
By improving the deep reinforcement learning algorithm and hybrid drive model, combining mechanism drive and data drive model, the problem of describing motion characteristics in complex environments in intelligent ship manipulation motion modeling is solved, and higher modeling accuracy and adaptability are achieved. Suitable for complex environments of smart ships.
Patent Information
- Application Number
- CN202411016735.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-29
AI Technical Summary
Existing ship manipulation motion modeling methods cannot effectively deal with complex nonlinear and uncertain factors, resulting in the model being inaccurate and adaptable enough, making it difficult to accurately describe the motion characteristics of smart ships in a changing environment.
Using a hybrid drive model that combines a mechanism-driven model with a data-driven model, an intelligent ship manipulation motion model is designed, and the accuracy and robustness of modeling is improved through deep reinforcement learning identification algorithm and Stacking integrated model. , able to better handle motion recognition in complex environments.
Achieve more accurate intelligent ship motion modeling, able to accurately characterize and predict motion states under different environmental conditions, overcome the shortcomings of single drive modeling, improve the adaptability and robustness of the model, and reduce the cost of actual ship tests .
Smart Images

Figure CN118551190B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent ship maneuvering motion modeling and identification, and specifically to an intelligent ship maneuvering dual-drive modeling method based on improved deep reinforcement learning. Background Art
[0002] The rapid rise of intelligent ships, as well as their higher safety and intelligence, have made them quickly become the focus of attention and development of the shipbuilding industry, ship operators and even the entire shipping industry. The development of intelligent ships is inseparable from the support of key technologies such as autonomous navigation and collision avoidance control. Therefore, it is urgent to develop a more advanced ship maneuvering motion modeling method to meet the needs of motion characteristic analysis and motion planning control during the operation of intelligent ships. In the past few decades, ship maneuvering motion modeling has played a key role in many fields such as ship controllers, water traffic flow simulation, and ship design performance evaluation. Autonomous navigation technology is the core of realizing advanced ship motion functions, and building an accurate mathematical model of ship maneuvering motion is the basis for the development of autonomous navigation systems. An accurate ship motion model is crucial for designing a reliable navigation controller and accurately predicting ship maneuvering actions. Therefore, designing a robust and effective ship maneuvering motion modeling method is of far-reaching significance for improving the level of ship intelligence.
[0003] When an intelligent ship sails in an open environment at the junction of water and air, it will be affected by various environmental factors such as wind, waves, and ocean currents, and its maneuvering motion state will also change due to changes in navigation conditions such as load and speed. In view of the influence characteristics of these factors, current research often considers the influence of a certain factor or a few uncertain factors, and uses a white noise model for simplified simulation processing. This method cannot accurately and effectively describe these uncertain interferences, resulting in insufficient consideration of the impact of related factors on the maneuvering motion of intelligent ships. In addition, in terms of the construction of ship maneuvering motion models, current research mainly focuses on a single drive modeling method, that is, a mechanism-driven modeling method or a data-driven modeling method. Due to the strong nonlinearity of water resistance and the randomness of turbulence, it is extremely difficult to establish an accurate mechanism model, and it is impossible to effectively deal with highly nonlinear and complex ship dynamics problems. In addition, the irregularity of the shapes of various unmanned surface ships makes it difficult to fully match the traditional parametric model based on classic ship types. Although the data-driven modeling method does not express the hydrodynamic structure of the ship, it does not consider the mechanism of the ship's maneuvering motion and lacks physical explanation. In a specific parameter framework, the unmodeled dynamics caused by external disturbances and noise greatly affect the parameter estimation. Therefore, it is difficult for a single-structure model to quickly adapt to the changes in external environmental disturbances and the internal dynamic characteristics of the power system. It is urgent to study a dual-drive model for intelligent ship maneuvering motion to meet these challenges. Summary of the invention
[0004] The purpose of the present invention is to solve the problems raised by the above-mentioned background technology, and to propose a dual-drive modeling method for intelligent ship manipulation based on improved deep reinforcement learning; the present invention simulates the manipulation motion of the intelligent ship based on a preset high-precision intelligent ship motion model, based on the derivation of the three-degree-of-freedom motion model of the intelligent ship. At the same time, considering the influence of uncertainty interference on the motion model of the intelligent ship, a hybrid drive model combining the mechanism-driven model and the data-driven model is designed. The model can more accurately reflect the motion characteristics of the intelligent ship under the influence of uncertainty, improve the accuracy, adaptability and robustness of the modeling, save the cost of the actual ship test, and improve the efficiency of ship modeling. At the same time, based on the deep reinforcement learning algorithm as the basic identification method, combined with the advantages of the Stacking integrated model, the deep reinforcement learning identification modeling method is improved, and an improved deep reinforcement learning identification algorithm is designed. The algorithm can more effectively handle the motion identification problem of intelligent ships in complex environments, and can effectively capture and describe the changes in the motion model of the ship when it is disturbed by the outside world, and improve the accuracy and efficiency of identification. The present invention provides an innovative intelligent ship motion modeling and identification method, which can more accurately characterize and predict the motion state of intelligent ships under different environmental conditions, overcome the problems existing in the existing modeling technology, such as high dependence on mechanism models, insufficient consideration of uncertainty effects, and poor single-drive modeling, and provide strong support for the development of a ship maneuvering motion modeling method system.
[0005] The purpose of the present invention can be achieved by the following technical solutions: A dual-drive modeling method for intelligent ship maneuvering based on improved deep reinforcement learning, comprising:
[0006] S1: According to the six-degree-of-freedom intelligent ship motion model and motion characteristics, a planar three-degree-of-freedom motion model for describing the motion characteristics of the intelligent ship is obtained;
[0007] S2: Based on the planar three-degree-of-freedom motion model used to describe the motion characteristics of intelligent ships, the maneuvering motion of intelligent ships is simulated, and combined with the motion change characteristics of intelligent ships under uncertain interference, a hybrid driving model combining the mechanism-driven model and the data-driven model is designed;
[0008] S3: Based on the selected deep reinforcement learning algorithm as the basic identification method, combined with the advantages of the Stacking ensemble model, the deep reinforcement learning identification modeling method is improved, and an improved deep reinforcement learning identification algorithm is designed;
[0009] S4: Train and test the hybrid drive model according to the designed improved deep reinforcement learning identification algorithm to obtain the identification results;
[0010] S5: According to the preset evaluation criteria, the model with the best comprehensive evaluation indicators is selected from the test results as the three-degree-of-freedom motion characteristic characterization model of the intelligent ship.
[0011] As a preferred embodiment of the present invention, the process of obtaining the planar three-degree-of-freedom motion model in step S1 is specifically as follows:
[0012] On the horizontal plane, it is regarded as a rigid body with six degrees of freedom, including sway, surge, heave, roll, pitch and bow. Its action center is set to be the same as the center of the center line of the ship's water surface line. At the same time, two right-handed Cartesian coordinate systems, namely inertial coordinate systems, are introduced. and the attached coordinate system ;
[0013] Assume that G is the center of gravity of the ship, u is the sway velocity, v is the sway velocity, r is the bow angular velocity, w is the heave velocity, p is the roll angular velocity, q is the pitch angular velocity, U is the total translation velocity, δ is the rudder angle, Ψ is the heading angle, β is the drift angle, is the heel angle, θ is the pitch angle, x 0 ,y 0 and z 0 is the position coordinate of the intelligent ship, X, Y, Z, K, M and N are the projections of the total forces and moments of the ship on the six degrees of freedom;
[0014] The six-degree-of-freedom maneuvering motion equation of an intelligent ship is:
[0015]
[0016] The three-degree-of-freedom maneuvering motion equation of the intelligent ship in the appendage coordinate system can be expressed as:
[0017]
[0018] Where, m is the mass of the ship, I z is the yaw moment of inertia about the z-axis, is the longitudinal coordinate of the ship’s center of gravity, , and They represent surge acceleration, sway acceleration and yaw angular acceleration respectively. X, Y and N are the projections of the total forces and moments of the ship on the three degrees of freedom.
[0019] As a preferred embodiment of the present invention, the specific design process of the hybrid drive model in step S2 is: the ship, propeller and rudder are regarded as a whole, and the hydrodynamic force F acting on the ship is regarded as a function of motion input and control input, that is, ; Assuming that the ship is sailing at a uniform speed, the Taylor series expansion retains the third-order term, and the overall mathematical model of the intelligent ship is expressed as:
[0020]
[0021] in, , and is a nonlinear function containing first to third order hydrodynamic derivatives between ship speed or angular velocity and rudder angle, , , , and is the derivative of the ship's angular acceleration or the fluid acceleration;
[0022] , and They are expressed as:
[0023]
[0024] In the formula, the subscript represents the differential of the subscript parameter with respect to the superscript. , , are the surge, sway and yaw force or moment components at the initial moment respectively.
[0025] As a preferred embodiment of the present invention, the mechanism-driven model in step S2 is an expression that represents the dynamic change characteristics of the system with parameters having clear physical meanings based on the corresponding theorem, and is specifically expressed as:
[0026]
[0027] The accelerations of X, Y, and N are , and ; X is , , , , The even function of , , , , The odd-order coefficients are zero, and Y and N are , , , , Odd functions of
[0028] The data-driven model in step S2 is a model based on the mechanism-driven model framework, which represents the mapping relationship between the cause and the variable that causes the difference according to the difference between the mechanism-driven model and the system dynamic change motion measurement data, and is specifically expressed as:
[0029]
[0030] in, , , , and is related to acceleration , and The corresponding fluid dynamic coefficient, , and Representation and , , and Related nonlinear functions;
[0031] According to the formula, we can get the nonlinear mapping function: , and and , , and The relationship between is used as the model to be identified:
[0032]
[0033] The equation of the hybrid drive model for intelligent ship maneuvering motion is:
[0034]
[0035] Set different weight ratios: ;in, It is represented as a hybrid drive model. It is represented as a mechanism-driven model, Represented as a data-driven model, and is the model weight ratio; set , then the overall structure of the hybrid drive model is expressed as:
[0036] .
[0037] As a preferred embodiment of the present invention, the specific process of designing an improved deep reinforcement learning identification algorithm is as follows:
[0038] The movement of the intelligent ship under environmental interference is regarded as a standard reinforcement learning system, which consists of an intelligent agent that interacts with the external navigation environment.
[0039] Based on the intelligent ship motion model, at each time step, DDPG uses a Markov decision process To describe, , where is the state space, is the action space, is the reward function, Represents the probability of state transition process; has initial state distribution , Action transfer status And the reward function , the agent's behavior is defined by a policy that maps states to a probability distribution of actions, namely , the expression is:
[0040]
[0041] For the mapping process from the state to the action of the intelligent ship agent, in a time series , the probability of this sequence occurring is:
[0042]
[0043] Set the reward for each moment of the sequence The sum is , when the strategy is When , the expected value of the reward is:
[0044]
[0045] Maximize the expected value obtained by the Agent, that is, the optimal strategy:
[0046]
[0047] In DDPG, the Agent consists of four network structures and a replay buffer. The four networks are the Critic target network, the Critic current network, the Actor target network, and the Actor current network.
[0048]
[0049] in, The input of the network is , the weight is , the output is ; The main network and the target network have the same structure, but different weights; The input of the Critic network is the state and action value output by the Actor network, and the network output is the value of evaluating the following actions:
[0050]
[0051] The target network then slowly follows the learned network to update the target network’s weights:
[0052]
[0053] in, is the soft update factor and ; This means that the constraint target value changes slowly, greatly improving the stability of learning; then the Critic network can be updated by minimizing the loss:
[0054]
[0055] in: , For the current reward, is the discount factor, and ; Take the current ship's motion variables and rudder angle as input, and the next moment's motion variables as output; let , the above formula can be rewritten as: ;
[0056] When the replay buffer is full, the initial samples are discarded; DDPG evenly samples a small batch from the replay buffer to update the Actor network and the Critic network;
[0057] Adding Ornstein Uhlenbeck noise to the actions output during the Actor network training phase gives the Agent the ability to explore: , It is an OU function.
[0058] As a preferred embodiment of the present invention, there are two layers of models in series in the stacking integrated model. First, the original data is input into the first layer model for training. After the training is completed, the algorithm in the first layer model outputs the corresponding prediction results; these prediction results are organized into a new feature matrix and then input into the second layer algorithm for training; the result output by the second layer model is the prediction result finally output by the Stacking integrated model.
[0059] As a preferred embodiment of the present invention, the input of the hybrid drive model of intelligent ship maneuvering motion includes the state variables and control variables of the current time step, namely u, v, r and , the output includes the state variables for the next time step, i.e. , and ; ,in, is a state variable, is the controlled variable, i.e., the rudder angle, is the matrix form of nonlinear mapping, Indicates the trajectory of the ship and heading angle , the ship's trajectory and heading angle Then use Euler's formula forward difference to solve:
[0060]
[0061] , and The calculation formula is:
[0062]
[0063] in, is the sampling time;
[0064] The constructor of the hydrodynamic parameter estimation sample for the hybrid drive model of intelligent ship maneuvering motion is expressed as:
[0065]
[0066] Through error Error, root mean square error RMSE and determination coefficient R 2 The indicators are used for evaluation and analysis. The Error calculation formula is as follows:
[0067]
[0068] Where t is the sample time, For real data, To predict data, For the three ship motion variables, RMSE and R 2 The calculation formula is as follows:
[0069]
[0070]
[0071] in, is the real data at time n, is the data predicted by the model, is the mean of the real data, and N is the data sequence.
[0072] As a preferred embodiment of the present invention, the specific process of selecting the model with the best comprehensive evaluation indexes from the test results as the intelligent ship three-degree-of-freedom motion characteristic characterization model is as follows:
[0073] Firstly, the collected motion data of the intelligent ship are divided into two groups: training set data and test set data. Both the training set data and the test set data include the sway velocity, sway velocity, bow angular velocity and command rudder angle. At the same time, the input of the model is the sway velocity, sway velocity, bow angular velocity and command rudder angle at the current moment, and the output of the model is the sway velocity, sway velocity and bow angular velocity at the next moment.
[0074] Then, these training set data and test set data are applied to the hybrid drive model of intelligent ship maneuvering motion and input into the improved deep reinforcement learning model. By using the actual motion data of the intelligent ship and the improved deep reinforcement learning algorithm, the three-degree-of-freedom hybrid drive model of the intelligent ship is identified.
[0075] Finally, by comparing and analyzing the test results and based on the preset evaluation criteria, the model with the highest accuracy is selected as the final intelligent ship three-degree-of-freedom motion characteristic characterization model.
[0076] Compared with the prior art, the present invention has the following beneficial effects:
[0077] 1. Aiming at the model uncertainty caused by the hydrodynamic changes brought about by the constantly changing external environmental factors and internal dynamic characteristics of intelligent ships during navigation, the present invention mixes the mechanism-driven model with mechanism support and explainability with the data-driven model with operability and flexibility, and designs an intelligent ship maneuvering motion identification model driven by a combination of mechanism and data, which avoids the shortcomings caused by the use of a single driving method and has extremely strong accuracy and applicability while ensuring the model mechanism;
[0078] 2. The recognition algorithm designed by the present invention combines the decision-making ability of reinforcement learning with the computing power of neural networks, and has strong nonlinear learning and self-organizing capabilities. In combination with the idea of the Stacking integrated model, it can provide interval prediction results with probability distribution significance while ensuring the accuracy of the model, overcoming the problems of traditional algorithms being sensitive to interference and prone to falling into local optimality, and has good innovation. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0080] Figure 1 A flowchart of a smart ship maneuvering motion mechanism and data dual-driven modeling method based on improved deep reinforcement learning provided by the present invention;
[0081] Figure 2 It is a flow chart for implementing the identification modeling method in an embodiment of the present invention;
[0082] Figure 3 A schematic diagram of a three-degree-of-freedom coordinate system for the motion of an intelligent ship according to an embodiment of the present invention;
[0083] Figure 4 A schematic diagram of a hybrid drive model for intelligent ship maneuvering motion according to an embodiment of the present invention;
[0084] Figure 5 A schematic diagram of an improved deep reinforcement learning algorithm according to an embodiment of the present invention;
[0085] Figure 6 It is a specific implementation case effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION
[0086] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0087] The purpose of the present invention is to address the shortcomings of existing intelligent ship motion modeling technology and provide a method for modeling intelligent ship maneuvering motion mechanism and data dual-driven based on improved deep reinforcement learning, thereby achieving a technical solution that is not highly dependent on the mechanism model, fully considers uncertainty interference, and has a good identification algorithm effect.
[0088] In order to achieve the above technical effects, the main concepts of the present invention are as follows:
[0089] According to the determined motion model and motion characteristics of the intelligent ship, a planar three-degree-of-freedom motion model for describing the motion characteristics of the intelligent ship is obtained; then, based on the planar three-degree-of-freedom motion model for describing the motion characteristics of the intelligent ship, the maneuvering motion of the intelligent ship is simulated, and combined with the change characteristics of the intelligent ship motion model under uncertainty interference, a hybrid drive model combining the mechanism-driven model and the data-driven model of the intelligent ship is designed; at the same time, based on the selected deep reinforcement learning algorithm as the basic identification method, combined with the advantages of the Stacking integrated model, the deep reinforcement learning identification modeling method is improved, and an improved deep reinforcement learning identification algorithm is designed; according to the designed improved deep reinforcement learning algorithm, the three-degree-of-freedom motion hybrid drive model of the intelligent ship is tested to obtain the identification results; finally, according to the preset evaluation criteria, the model with the best evaluation indicators is selected from the test results as the three-degree-of-freedom intelligent ship motion characteristics analysis model.
[0090] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0091] This embodiment provides a method for modeling intelligent ship maneuvering motion mechanism and data dual drive based on improved deep reinforcement learning, see Figure 1 , the method comprising:
[0092] Step 1: According to the six-degree-of-freedom intelligent ship motion model and motion characteristics, a planar three-degree-of-freedom motion model is obtained to describe the motion characteristics of the intelligent ship.
[0093] Considering all coupled motions, the smart ship can be regarded as a rigid body with six degrees of freedom on the horizontal plane, including sway, surge, heave, roll, pitch and bow. In order to describe the motion state of the ship, it is assumed that the position of its action center is the same as the center position of the center line of the ship's water surface line, and two right-handed Cartesian coordinate systems, namely the inertial coordinate system, are introduced. ( is the true north direction) and the attached coordinate system ( is the ship’s heading). is the ship's center of gravity, u is the surge velocity, v is the sway velocity, r is the bow angular velocity, w is the heave velocity, p is the roll angular velocity, q is the pitch angular velocity, is the total translational velocity, is the rudder angle, is the heading angle, For the drift angle, is the heel angle, is the longitudinal inclination angle, x0, y0 and z0 are the position coordinates of the intelligent ship, and X, Y, Z, K, M and N are the projections of the total forces and moments of the ship on the six degrees of freedom.
[0094] The six-degree-of-freedom maneuvering motion equation of an intelligent ship can be expressed as:
[0095]
[0096] From the perspective of ship manipulation, for most movements and controls, the ship motion can be simplified to three-degree-of-freedom horizontal plane motion, that is, the vertical heave motion, roll motion and pitch motion of the ship are ignored, and only the horizontal plane sway motion, pitch motion and bow motion are considered. Assuming that the center of gravity G of the ship coincides with the origin of the appendage coordinate system O0, combined with Newton's second law, the three-degree-of-freedom maneuvering motion equation of the intelligent ship in the appendage coordinate system can be expressed as:
[0097]
[0098] in, is the mass of the ship (including additional mass), for The yaw moment of inertia about the axis, is the longitudinal coordinate of the ship’s center of gravity, , and denote the swell acceleration, the roll acceleration and the pitch angular acceleration, respectively. , and It is the projection of the total forces and moments of the ship on the three degrees of freedom.
[0099] Step 2: Based on the planar three-degree-of-freedom motion model used to describe the motion characteristics of intelligent ships, simulate the maneuvering motion of intelligent ships, and design a hybrid driving model that combines the mechanism-driven model and the data-driven model based on the motion change characteristics of intelligent ships under uncertain interference.
[0100] The ship, propeller and rudder are considered as a whole, and the hydrodynamic force F acting on the ship is considered as a function of motion input and control input, that is, Assuming that the ship is sailing at a constant speed, the Taylor series expansion retains the third-order term, and the overall mathematical model of the intelligent ship can be expressed as follows:
[0101]
[0102] in, , and is a nonlinear function containing first to third order hydrodynamic derivatives between ship speed or angular velocity and rudder angle, , , , and is the derivative of the ship's angular acceleration or the fluid acceleration. , and Can be expressed as:
[0103]
[0104] In the formula, the subscript represents the differential of the subscript parameter with respect to the superscript. , , are the surge, sway and yaw force or moment components at the initial moment respectively.
[0105] Fix the hull coordinate system As the origin, the hydrodynamic force and torque acting on the ship are expressed as a function of the motion variable and the rudder angle. The velocity component is used as the state variable, and the rudder angle is used as the control variable. Only the hydrodynamic force and torque acting on the ship are considered. When , and The accelerations are , and , it is expressed as the ship motion state variable , , and control variables Since the ship is symmetrical about the longitudinal direction, X is , , , , The even function of , , , , The odd-order coefficients are zero. And Y and N are , , , , Odd function, relative , , , , The even-order expansion terms of are not involved in the expansion. At the same time, only the first-order coefficient of acceleration is considered, and the intelligent ship mechanism driving model can be written as:
[0106]
[0107] The goal of data-driven modeling of intelligent ships is to establish a mathematical model that uses the ship's motion state, thrust and rudder angle as input to produce the best fit between the model prediction and the actual response of the ship. The more accurate the mapping between input and output, the closer the established model is to the real model. It is an effective method to automatically fit the dynamic response of the ship to the environment inside the black box model by using an identification algorithm without considering the influence of the external environment and only considering the changes in the ship's motion parameters and control equipment under the influence of the external environment.
[0108] Taking into account the actual voyage process of the ship , and It will be affected by external environmental factors and the ship's own factors. , and The motion variables will have uncertain interference, and the data-driven model of intelligent ships can be written as:
[0109]
[0110] in, , , , and is related to acceleration , and The corresponding fluid dynamic coefficient, , and Representation and , , and The nonlinear function involved.
[0111] In the data-driven model, according to the formula, following the nonlinear mapping function, we can get , and and , , and The relationship between is used as the model to be identified:
[0112]
[0113] In order to make up for the shortcomings of the mechanism-driven model, a correction term is added , taking into account the unknown or undefined hydrodynamic errors caused by idealized motion. Approximations such as Taylor series expansion, unknown or estimated parameters, speed and load conditions, and environmental changes are added to the mathematical model to improve the accuracy of the simulation. The equation of the hybrid drive model for intelligent ship maneuvering motion can be described as:
[0114]
[0115] In order to weigh the weights of the mechanism-driven model and the data-driven model in the hybrid model, different weight ratios are set as follows:
[0116]
[0117] in, It is represented as a hybrid drive model. It is represented as a mechanism-driven model, Represented as a data-driven model, and is the model weight ratio, which is set in this application The overall structure of the hybrid drive model is expressed as:
[0118]
[0119] The mechanism and data-driven models are integrated into a whole through summation approximation. The high-order residuals predicted by the data-driven model are approximated by deep reinforcement learning, and a direct summation connection is established between the residual prediction value and the mechanism-driven model prediction value, thereby realizing the combination of mechanism-driven and data-driven models. Only the initial velocity is required, and then the result of the previous prediction is used as the input of the next prediction, and the test result is used as the input of the next prediction. Substituting into, we get the hydrodynamic correction term The hybrid drive model of intelligent ship maneuvering motion can be obtained.
[0120] Step 3: Based on the selected deep reinforcement learning algorithm as the basic identification method, combined with the advantages of the Stacking ensemble model, improve the deep reinforcement learning identification modeling method and design an improved deep reinforcement learning identification algorithm. The specific implementation steps are as follows:
[0121] The movement of smart ships under environmental interference is regarded as a standard reinforcement learning system. The system consists of an agent (the smart ship itself) that interacts with the external navigation environment. By continuously interacting and learning with the environment, the smart ship can gradually optimize its strategy and improve its ability to cope with environmental interference. Based on the smart ship motion model, at each time step, DDPG can use the Markov decision process To describe, where is the state space, is the action space, is the reward function, Represents the probability of state transition process:
[0122]
[0123] It has an initial state distribution , Action transfer status And the reward function The behavior of an agent is defined by a policy that maps states to a probability distribution of actions, i.e. , the expression is:
[0124]
[0125] For the mapping process from the state to the action of the intelligent ship agent, in a time series , the probability of this sequence occurring is:
[0126]
[0127] Assume that the reward obtained by the sequence at each moment is The sum is , then when the strategy is When , the expected value of the reward is:
[0128]
[0129] The goal of DRL is to learn a strategy that maximizes the expected value obtained by the Agent, that is, the optimal strategy:
[0130]
[0131] In DDPG, the Agent is mainly composed of four network structures and a replay buffer. The four networks are the Critic target network, the Critic current network, the Actor target network, and the Actor current network. The Actor network inputs the current state and outputs the speed index of the ship in the current state. The Critic network is used to evaluate whether the action corresponding to the state is appropriate, and inputs and outputs the value function of the action.
[0132]
[0133] in, The input of the network is , the weight is , the output is The main network and the target network have the same structure, but different weights. The input of the Critic network is the state and action value output by the Actor network, and the network output is the value of evaluating the following actions:
[0134]
[0135] These target networks then slowly track the learned network to update the target network weights:
[0136]
[0137] in, is the soft update factor and This means that the target value is constrained to change slowly, greatly improving the stability of learning. The Critic network can then be updated by minimizing the loss:
[0138]
[0139] in:
[0140]
[0141] in, For the current reward, is the discount factor, and Since the model established in this application takes the motion variables and rudder angle of the ship at the current moment as input and the motion variables at the next moment as output, the training process only involves one-step interaction, and the goal of the optimal strategy is to maximize the cumulative result of the corresponding instant reward of each single-step interaction, therefore, let , the above formula can be rewritten as:
[0142]
[0143] The replay buffer is a finite-sized container that samples data from the environment according to the sampling strategy and uses it to train the neural network. Stored in. When the replay buffer is full, the initial samples are discarded. DDPG uniformly samples a small batch from the replay buffer to update the Actor network and the Critic network. In addition, exploration is crucial for the Agent, and the deterministic strategy lacks exploration capabilities, so it is necessary to add Ornstein Uhlenbeck (OU) noise to the actions output during the Actor network training phase to enable the Agent to have exploration capabilities:
[0144]
[0145] in, It is an OU function.
[0146] There are two layers of models in the Stacking model. First, the original data will be input into the first layer model for training. After the training is completed, the algorithm in the first layer model will output the corresponding prediction results. These prediction results are organized into a new feature matrix and then input into the second layer algorithm for training. The result output by the second layer model is the prediction result finally output by the Stacking model. It can be understood that the first layer model is responsible for feature transformation, and the second layer model is responsible for the final result prediction.
[0147] Assume that the original data set is expressed as: . and are the number of samples in the training set and the test set respectively. The input of the first layer With output It can be expressed as:
[0148]
[0149]
[0150] Among them, the training set used for the first layer model input and test set It is expressed as:
[0151]
[0152]
[0153] Will After being used as the input of the first layer model, the first prediction result is obtained , output It is expressed as:
[0154]
[0155]
[0156] Based on the results of the first prediction, reconstruct the training set and test set. As the input of the second layer model, the corresponding original data As output, it is used to train the second layer model, that is, the output of the first layer model is the input of the second layer model. and output It can be expressed as:
[0157]
[0158]
[0159] in:
[0160]
[0161]
[0162]
[0163] Then, take As the input of the trained second layer model, the second prediction result is obtained , which is the final prediction result of the hybrid model, is used to evaluate the performance of the model. It is expressed as:
[0164]
[0165] Stacking can use the base model to learn the initial data, and then pass the data learned by the base model to the meta-model for fitting. If the prediction errors of these models have weak correlation, then these errors will compensate each other. The fundamental advantage of this method is that it allows the meta-model to be trained in the direction of minimizing the loss function.
[0166] Step 4: According to the designed improved deep reinforcement learning algorithm, the three-degree-of-freedom motion hybrid drive model of the intelligent ship is tested to obtain the identification results.
[0167] The input of the hybrid drive model of intelligent ship maneuvering motion considered in this application includes the state variables and control variables of the current time step, namely u, v, r and , the output includes the state variables for the next time step, namely , and The process can be described as:
[0168]
[0169] in, is a state variable, is the controlled variable, i.e., the rudder angle, is the matrix form of nonlinear mapping, Indicates the trajectory of the ship and heading angle .for , the ship's trajectory and heading angle Then use Euler's formula forward difference to solve:
[0170]
[0171] , and The calculation formula is:
[0172]
[0173] in, is the sampling time. The constructor of the hydrodynamic parameter estimation sample for the hybrid drive model of intelligent ship maneuvering motion can be expressed as:
[0174]
[0175] The results are , and Three motion variables are displayed, through error Error, root mean square error RMSE and determination coefficient R 2 The error calculation formula is as follows:
[0176]
[0177] Where t is the sample time, For real data, To predict data, are the three ship motion variables. RMSE and R 2 The calculation formula is as follows:
[0178]
[0179]
[0180] in, is the real data at time n, is the data predicted by the model, is the mean of the real data, and N is the data sequence.
[0181] Step 5: According to the preset evaluation criteria, the model with the best comprehensive evaluation indicators is selected from the test results as the three-degree-of-freedom motion characteristic characterization model of the intelligent ship.
[0182] First, the collected motion data of the intelligent ship are divided into two groups: training set data and test set data. Both the training set data and the test set data include the longitudinal velocity, sway velocity, bow angular velocity and command rudder angle. At the same time, the input of the model is the longitudinal velocity, sway velocity, bow angular velocity and command rudder angle at the current moment, and the output of the model is the longitudinal velocity, sway velocity and bow angular velocity at the next moment. Then, these training set data and test set data are applied to the hybrid drive model of the intelligent ship maneuvering motion and input into the improved deep reinforcement learning model. By using the actual motion data of the intelligent ship and the improved deep reinforcement learning algorithm, the hybrid drive model of the three-degree-of-freedom motion of the intelligent ship is identified. Finally, by comparing and analyzing the test results and according to the preset evaluation criteria, the model with the highest accuracy is selected as the final intelligent ship three-degree-of-freedom motion characteristic characterization model.
[0183] In order to fully verify the excellent performance of the present invention in terms of accuracy, robustness and generalization, specific implementation case tests are carried out. Through the implementation cases, the efficiency and reliability of the present invention in practical applications will be demonstrated, further proving its innovation and practicality in related fields.
[0184] Taking the Mariner ship model as the object, set its initial longitudinal velocity u=7.72 m / s, initial transverse velocity v=0 m / s, initial bow angular velocity r=0 deg / s, and limit rudder angle . Continuous data collection was performed in Z-shaped maneuvering tests at different angles at a sampling frequency of 10 Hz, and a data set with a time length of 700 s and a total of 7000 samples was obtained. The first 70% of the data set was designated as a training data set, and the remaining 30% of the data was regarded as a validation data set. In order to evaluate the generalization of the hybrid model of the identified intelligent ship maneuvering motion, the trained model was used to test and verify the experiments not involved in the training data.
[0185] The 30° / 30° Z-shaped manipulation test was selected as the training set, and the 20° / 20° Z-shaped manipulation test was tested for generalization. Figure 6 As shown in the results, it can be seen that the intelligent ship maneuvering motion mechanism and data dual-driven modeling method designed by the present invention based on improved deep reinforcement learning can effectively identify the maneuvering motion state parameters of intelligent ships, and has better prediction accuracy and robustness.
[0186] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A dual-drive modeling method for intelligent ship maneuvering based on improved deep reinforcement learning, characterized in that: The method includes: S1: According to the six-degree-of-freedom intelligent ship motion model and motion characteristics, a planar three-degree-of-freedom motion model for describing the motion characteristics of the intelligent ship is obtained; G is preset as the center of gravity of the ship, u is the pitch velocity, v is the sway velocity, r is the bow angular velocity, w is the heave velocity, p is the roll angular velocity, q is the pitch angular velocity, U is the total translation velocity, δ is the rudder angle, Ψ is the heading angle, β is the drift angle, is the heel angle, θ is the pitch angle; and represent the surge acceleration, sway acceleration and yaw angular acceleration respectively, X, Y and N are the projections of the total forces and moments of the ship on the three degrees of freedom; S2: Based on the planar three-degree-of-freedom motion model used to describe the motion characteristics of intelligent ships, the maneuvering motion of intelligent ships is simulated, and combined with the motion change characteristics of intelligent ships under uncertainty interference, a hybrid driving model combining the mechanism-driven model and the data-driven model is designed; the mechanism-driven model is an expression that represents the dynamic change characteristics of the system with parameters with clear physical meanings based on the corresponding theorem, which is specifically expressed as: The accelerations of X, Y, and N are and X is v, r, Even function of δ, corresponding to v, r, The odd-order coefficients of δ are zero, and Y and N are v, r, Odd functions of δ; and is the derivative of the ship's angular acceleration or the fluid acceleration; The data-driven model is a model based on the mechanism-driven model framework, which represents the mapping relationship between the cause and the variable that causes the difference according to the difference between the mechanism-driven model and the system dynamic change motion measurement data. It is specifically expressed as: in, and is related to acceleration and The corresponding fluid dynamic coefficients, Δf1, Δf2, and Δf3, represent nonlinear functions related to u, v, r, and δ; According to the formula, following the nonlinear mapping function, we can get and The relationship between u, v, r and δ is used as the model to be identified: The equation of the hybrid drive model for intelligent ship maneuvering motion is: Set different weight ratios: F i =α1f i +α2Δf i Among them, F i =[F1,F2,F3] T Expressed as a hybrid drive model, f i =[f1,f2,f3] T Expressed as a mechanism-driven model, Δf i =[Δf1,Δf2,Δf3] T It is represented as a data-driven model, α1 and α2 are model weight ratios; if α1 = α2 = 1, the overall structure of the hybrid-driven model is expressed as: S3: Based on the selected deep reinforcement learning algorithm as the basic identification method, combined with the advantages of the Stacking ensemble model, the deep reinforcement learning identification modeling method is improved, and an improved deep reinforcement learning identification algorithm is designed; S4: Train and test the hybrid drive model according to the designed improved deep reinforcement learning identification algorithm to obtain the identification results; S5: According to the preset evaluation criteria, the model with the best comprehensive evaluation indicators is selected from the test results as the three-degree-of-freedom motion characteristic characterization model of the intelligent ship.
2. According to the method for modeling intelligent ship maneuvering dual drive based on improved deep reinforcement learning in claim 1, it is characterized in that: The process of obtaining the planar three-degree-of-freedom motion model in step S1 is specifically as follows: On the horizontal plane, it is regarded as a rigid body with six degrees of freedom, including sway, surge, heave, roll, pitch and bow, and its action center is set to be the same as the center of the center line of the ship's water surface line. At the same time, two right-handed Cartesian coordinate systems are introduced, namely the inertial coordinate system O0-x0y0z0 and the appendage coordinate system O-xyz; x0, y0 and z0 are the position coordinates of the intelligent ship, X, Y, Z, K, M and N are the projections of the total force and moment of the ship on the six degrees of freedom, I x ,I y ,I z are the yaw moments of inertia about the x, y, and z axes respectively; The six-degree-of-freedom maneuvering motion equation of an intelligent ship is: The three-degree-of-freedom maneuvering motion equation of the intelligent ship in the appendage coordinate system can be expressed as: Where m is the mass of the ship, x G is the longitudinal coordinate of the ship's center of gravity.
3. The intelligent ship maneuvering dual-drive modeling method based on improved deep reinforcement learning according to claim 1 is characterized in that: The specific process of designing the hybrid drive model in step S2 is: considering the ship, propeller and rudder as a whole, and considering the hydrodynamic force F acting on the ship as a function of motion input and control input, that is, Assuming that the ship is sailing at a uniform speed, the Taylor series expansion retains the third-order term, and the overall mathematical model of the intelligent ship is expressed as: Among them, f1, f2 and f3 are nonlinear functions containing first to third order hydrodynamic derivatives between the ship speed or angular velocity and the rudder angle; f1, f2 and f3 are expressed as: Wherein, the subscript represents the differential of the subscript parameter with respect to the superscript, and X0, Y0, and N0 are the surge, sway, and yaw force or moment components at the initial moment, respectively.
4. The intelligent ship maneuvering dual-drive modeling method based on improved deep reinforcement learning according to claim 1 is characterized in that: The specific process of designing an improved recognition algorithm for deep reinforcement learning is: The movement of the intelligent ship under environmental interference is regarded as a standard reinforcement learning system, which consists of an intelligent agent that interacts with the external navigation environment. Based on the intelligent ship motion model, at each time step, DDPG is described by a Markov decision process (s, a, p, r), p(s′|s, a) = p[s t+1 =s′|s t =s,a t =a], where s is the state space, a is the action space, r is the reward function, and p represents the probability of the state transition process; With initial state distribution p(s1), action transition state p(s t+1 |s t ,a t ) and the reward function r t (s t ,a t ), the agent’s behavior is defined by a policy that maps states to the probability distribution of actions, i.e., π:S→P(A), expressed as: π(a|s)=p[a t =a|s t =s] For the mapping process from the state to the action of the intelligent ship agent, in a time series t = {s1, a1, s2, a2…s T }, the probability of this sequence occurring is: Set the sum of the rewards τ obtained by the sequence at each moment to R t , when the strategy is π, the expected value of the reward is: Maximize the expected value obtained by the Agent, that is, the optimal strategy: p * =arg max π E(π) In DDPG, the Agent consists of four network structures and a replay buffer. The four networks are the Critic target network, the Critic current network, the Actor target network, and the Actor current network. Among them, Q(a|b) means that the input of the network is a, the weight is b, and the output is Q; the main network and the target network have the same structure but different weights; the input of the Critic network is the state and action value output by the Actor network, and the network output is the value of evaluating the following actions: Q(s,a)=E[R t |s t =s,a t =a] The target network then slowly follows the learned network to update the target network’s weights: τθ+(1-τ)θ′→θ′ Among them, τ is the soft update factor and τ < 1; this means that the constraint target value changes slowly, which greatly improves the stability of learning; then the Critic network can be updated by minimizing the loss: Where: y i =r i +γQ′(S i+1 ,μ′(s i+1 |θ μ′ )θ Q′ ), r i is the current reward, γ is the discount factor, and γ∈[0,1]; the motion variables and rudder angle of the ship at the current moment are used as input, and the motion variables at the next moment are used as output; let γ=0, the above formula can be rewritten as: i =r i ; When the replay buffer is full, the initial samples are discarded; DDPG evenly samples a small batch from the replay buffer to update the Actor network and the Critic network; Adding Ornstein Uhlenbeck noise to the actions output during the Actor network training phase gives the Agent the ability to explore: It is an OU function.
5. The intelligent ship maneuvering dual-drive modeling method based on improved deep reinforcement learning according to claim 1 is characterized in that: There are two layers of models in series in the Stacking integrated model. First, the original data is input into the first layer model for training. After the training is completed, the algorithm in the first layer model outputs the corresponding prediction results; these prediction results are organized into a new feature matrix and then input into the second layer algorithm for training; the output result of the second layer model is the final prediction result of the Stacking integrated model.
6. The intelligent ship maneuvering dual-drive modeling method based on improved deep reinforcement learning according to claim 1 is characterized in that: The input of the hybrid drive model of intelligent ship maneuvering motion includes the state variables and control variables of the current time step, namely u, v, r and δ, and the output includes the state variables of the next time step, namely and Among them, x s (t) is the state variable, δ(t) is the control variable, i.e., the rudder angle, f(·) is the matrix form of the nonlinear mapping, z(t) represents the trajectory (x, y) and heading angle ψ of the ship, and the trajectory (x, y) and heading angle ψ of the ship are solved by forward difference using the Euler formula: and The calculation formula is: Where, Δt is the sampling time; The constructor of the hydrodynamic parameter estimation sample for the hybrid drive model of intelligent ship maneuvering motion is expressed as: Through error Error, root mean square error RMSE and determination coefficient R 2 The indicators are used for evaluation and analysis, and the Error calculation formula is as follows: Error t (·)=predict t (·)-real t (·) Where t is the sample time, real is the real data, predict is the predicted data, (·) are the three ship motion variables, RMSE and R 2 The calculation formula is as follows: Among them, y n is the real data at time n, is the data predicted by the model, is the mean of the real data, and N is the data sequence.
7. The intelligent ship maneuvering dual-drive modeling method based on improved deep reinforcement learning according to claim 1 is characterized in that: The specific process of selecting the model with the best comprehensive evaluation indexes from the test results as the three-degree-of-freedom motion characteristic characterization model of the intelligent ship is as follows: Firstly, the collected motion data of the intelligent ship are divided into two groups: training set data and test set data. Both the training set data and the test set data include the sway velocity, sway velocity, bow angular velocity and command rudder angle. At the same time, the input of the model is the sway velocity, sway velocity, bow angular velocity and command rudder angle at the current moment, and the output of the model is the sway velocity, sway velocity and bow angular velocity at the next moment. Then, these training set data and test set data are applied to the hybrid drive model of intelligent ship maneuvering motion and input into the improved deep reinforcement learning model. By using the actual motion data of the intelligent ship and the improved deep reinforcement learning algorithm, the three-degree-of-freedom hybrid drive model of the intelligent ship is identified. Finally, by comparing and analyzing the test results and based on the preset evaluation criteria, the model with the highest accuracy is selected as the final intelligent ship three-degree-of-freedom motion characteristic characterization model.
