Ship motion trajectory optimization control method based on deep learning

Abstract

The present invention proposes a deep learning-based ship motion trajectory optimization control method, which relates to the field of ship control technology. The method includes: constructing a ship motion simulation environment and establishing a dynamic simulation system that includes ocean environment parameters; collecting ship operation data; constructing a dual-network structure model, including a macro-strategy network and a micro-control network, wherein the macro-strategy network generates a heading planning strategy at longer time intervals, and the micro-control network generates specific control instructions at shorter time intervals; using a hierarchical training strategy, the dual-network structure model is trained in the ship motion simulation environment, first training the macro-strategy network to generate an overall heading strategy, and then training the micro-control network to achieve precise control; deploying the trained dual-network structure model to the ship control system to generate optimal control instructions, thereby achieving real-time optimization control of the ship's motion trajectory. The present invention achieves real-time optimization control of the ship's motion trajectory while ensuring control accuracy.

Description

Technical Field

[0001] The present invention relates to the field of ship control technology, and in particular to a ship motion trajectory optimization control method based on deep learning. Background Art

[0002] Optimal control of ship motion trajectory is a key technology for ensuring safe and efficient maritime navigation. It is crucial for improving shipping efficiency, reducing energy consumption, and ensuring safe navigation. Ships navigate complex and changing ocean environments, impacted by factors such as wind, wave forces, and currents. Furthermore, consideration must be given to the ship's own motion characteristics and maneuverability. Therefore, achieving precise control of a ship's motion trajectory in complex sea conditions is a crucial research topic in the shipping industry.

[0003] Currently, ship trajectory optimization control primarily relies on traditional PID control and model predictive control methods. These methods typically design control strategies based on simplified ship motion models. With the development of artificial intelligence (AI) technology, some research has begun to explore the application of deep learning methods to ship trajectory control, using neural network models to learn control strategies or reinforcement learning methods to optimize control decisions.

[0004] However, existing control methods still have many shortcomings in practical applications: First, traditional control methods have difficulty coping with complex and changing marine environments, and control performance is easily affected by external interference. Second, methods based on a single control strategy cannot simultaneously take into account the integrity of trajectory planning and the accuracy of control instructions. Third, existing deep learning methods have limitations in coordinating the optimization of long-term time series prediction and short-term control, and lack adaptability to extreme sea conditions. These problems seriously restrict the practical application of ship motion trajectory optimization control technology. Summary of the Invention

[0005] In view of this, the present invention proposes a ship motion trajectory optimization control method based on deep learning. By constructing a dual network structure model and a hierarchical training strategy, the coordinated optimization of long-term planning and short-term control of the ship motion trajectory is realized, while the control system's adaptability to complex and changeable sea conditions is improved, thereby achieving real-time optimization control of the ship motion trajectory while ensuring control accuracy.

[0006] The technical solution of the present invention is achieved as follows:

[0007] The present invention provides a ship motion trajectory optimization control method based on deep learning, comprising:

[0008] S1. Constructing a ship motion simulation environment and establishing a dynamic simulation system including ocean environment parameters; collecting ship operation data, wherein the operation data includes ship state parameters and ocean environment parameters, wherein the ship state parameters are obtained through a shipboard sensing system, and the ocean environment parameters are obtained through an environmental monitoring system;

[0009] S2. Construct a dual-network structure model, including a macro-strategy network and a micro-control network. The macro-strategy network generates a course planning strategy at longer intervals, while the micro-control network generates specific control instructions at shorter intervals. The dual-network structure model takes as input ship state parameters and ocean environment parameters, and the two networks interact with each other through a temporal attention mechanism.

[0010] S3. A hierarchical training strategy is used to train the dual-network structure model in a ship motion simulation environment. First, the macro-strategy network is trained to generate the overall heading strategy, and then the micro-control network is trained to achieve precise control. The macro-strategy network is optimized based on long-term rewards, and the micro-control network is optimized based on short-term control accuracy. The two networks are maintained consistent through a gradient alignment mechanism.

[0011] S4. Deploy the trained dual-network structure model to the ship control system, implement predictive control through the sliding time window method, receive ship state parameters and ocean environment parameters in real time, dynamically adjust the control strategy based on the predicted environmental changes, generate optimal control instructions, and realize real-time optimization control of the ship's motion trajectory.

[0012] Preferably, the ship status parameters include: the ship's position coordinates, heading angle, speed, pitch angle, roll angle, yaw angle, bow angular velocity, lateral speed and longitudinal speed; the ocean environment parameters include: wind speed, wind direction, wave height, wave direction, water flow speed, water flow direction, water depth and sea temperature.

[0013] Preferably, in step S1, constructing a ship motion simulation environment includes:

[0014] Establish a three-dimensional ship model, including the hull geometry model and mass distribution model;

[0015] Build a hydrodynamics module to calculate various hydrodynamic forces acting on ships;

[0016] Set up the propulsion system model, including the main thruster and steering gear system;

[0017] Establish an environmental force model to simulate the effects of wind, wave and current forces;

[0018] The dynamic simulation system is integrated into the ship motion simulation environment, including:

[0019] Construct an ocean environment parameter generation module to generate time-varying environment parameters;

[0020] Establish a ship motion equation solver to calculate the ship's motion response in complex sea conditions;

[0021] Set up a sensor data simulation module to simulate the data acquisition process of the shipborne sensor system and environmental monitoring system;

[0022] Establish a data storage and analysis module to record and process various types of data during the simulation process.

[0023] Preferably, the dual network structure model includes:

[0024] Macro Strategy Network:

[0025] An input layer, comprising a first input sublayer and a second input sublayer for receiving ship state parameters and ocean environment parameters respectively;

[0026] The temporal coding layer sets a temporal coding module after each input sublayer to perform feature marking on input signals with long time intervals;

[0027] The feature extraction layer combines a bidirectional long short-term memory network with a one-dimensional convolutional network to process the input ship state parameters and ocean environment parameters in parallel and extract time series features respectively;

[0028] The feature fusion layer fuses the features extracted by the bidirectional long short-term memory network and the one-dimensional convolutional network through a gating mechanism to form a comprehensive feature representation;

[0029] The strategy generation layer connects two fully connected networks, using a learnable residual connection between each network layer and the activation function Swish to generate the heading planning strategy vector;

[0030] Micro-control network:

[0031] The input layer sets a multi-channel convolution input structure for data with shorter time intervals, mapping the high-frequency sampled ship state parameters and ocean environment parameters into a multi-dimensional feature space;

[0032] The multi-scale convolution layer combines dilated convolution with variable kernel convolution to capture the temporal features of the micro-control stage at multiple scales.

[0033] The attention enhancement layer uses a multi-head causal attention mechanism to assign feature weights of different historical moments to the control output at the current moment;

[0034] The control output layer is connected to a two-branch fully connected network. One branch is used to output the rudder angle distribution, and the other branch is used to output the propulsion force distribution. The activation function is Mish.

[0035] Cross-network information interaction mechanism:

[0036] The temporal interactive attention module is set between the policy generation layer of the macro policy network and the input layer of the micro control network to achieve dynamic alignment of information between the output of the macro policy network and the input of the micro control network.

[0037] Preferably, step S3 includes:

[0038] S31. Construct comprehensive evaluation function R t , based on the comprehensive evaluation function R t Define the long-term return R long-term and short-term immediate reward R short-term ;

[0039] S32. Define the state space S and action space A of the ship's motion, and the state vector s t ∈S is defined as:

[0040]

[0041] In the formula, (x t ,y t ) is the position coordinate of the ship in the global coordinate system; θ t is the heading angle; v t is the ship speed; pit is the pitch angle; φ roll is the roll angle; φ yaw is the head swing angle; ω yaw is the yaw angular velocity; v lat is the lateral speed; v long is the longitudinal velocity; v wind is the wind speed; θ wind is the wind direction; h wave is the wave height; θ wave is the wave direction; v cur is the water flow velocity; θ cur is the direction of water flow; d depth is the water depth; T water is the sea temperature; (x tar ,y tar ) is the target position coordinate; φ task The current task stage;

[0042] Motion vector a t ∈A is defined as:

[0043] a t =[n t ,δ t ] T

[0044] Where n tis the propeller speed ratio, which represents the ratio of the desired propeller speed to the rated speed; δ t is the rudder angle;

[0045] S33, first train the macro strategy network, based on the long-term return R long-term Iteratively optimize the course planning strategy; use a multi-batch experience replay mechanism to screen key training samples covering various sea conditions;

[0046] S34, after the macro strategy network initially converges, switch to the training phase of the micro control network; based on the short-term immediate reward R short-term , conduct reinforcement learning iteration of the micro-control network to generate specific control strategies;

[0047] S35. By dynamically allocating tasks, different environmental complexity scenarios are introduced to enable the macro-strategy network and the micro-control network to adapt to different sea conditions.

[0048] S36. During training, a gradient alignment module is established to monitor the policy differences between the macro policy network and the micro control network. Through the gradient alignment mechanism, the parameters of the macro policy network and the micro control network are adjusted synchronously.

[0049] S37. After the macro-strategy network and the micro-control network are trained to the preset convergence standard, they are jointly fine-tuned. During the joint fine-tuning process, the macro-strategy network and the micro-control network are simultaneously optimized by minimizing the comprehensive loss function. When the fine-tuning is completed, a trained dual-network structure model is obtained.

[0050] Preferably, the calculation formula of the comprehensive evaluation function is as follows:

[0051] R t =w1·f traj (s t )+w2·f energy (a t )+w3·f smooth (s t ,a t )

[0052] Where R t is the reward value at time step t; f traj (s t ) represents state s t The trajectory tracking error when f energy (a t ) means taking action a t Energy consumption when smooth (s t ,a t ) represents state s t Next take action a tw1, w2, w3 are dynamic weight coefficients;

[0053] Long-term return R long-term The calculation formula is as follows:

[0054]

[0055] Where, ρ t is the timing discount factor; g(H t ) is the sea state adaptation function, according to the current sea state H t Adjust reward intensity;

[0056] Short-term immediate reward R short-term The calculation formula is: R short-term =R t , that is, the short-term immediate reward directly uses the reward value calculated by the comprehensive evaluation function at the current time step t.

[0057] Preferably, in step S36, the gradient alignment mechanism is as follows:

[0058] For the macro strategy network and micro control network, the loss function L is defined respectively macro and L micro ;

[0059] Calculate the respective gradient vectors and Among them, θ macro and θ micro represent the parameter sets of the macro-strategy network and the micro-control network respectively;

[0060] Use the gradient alignment formula to align the gradient vectors of the two networks and adjust the network parameters synchronously. The gradient alignment formula is as follows:

[0061]

[0062] Where, L align is the gradient alignment loss, λ is the trade-off coefficient, η(t) is the time-varying adjustment factor, and ‖·‖ represents the Euclidean norm.

[0063] Preferably, the comprehensive loss function is as follows:

[0064] L total =L macro +L micro +α·L align

[0065] L macro =L policy +β1·L value +β2·L entropy

[0066]

[0067]

[0068] L micro =L TD +μ1·L smooth +μ2·L constraint

[0069]

[0070] L smooth =‖a t -a t-1 ‖ 2

[0071] L constraint =max(0,‖a t ‖-a max ) 2

[0072] Where, L total is the comprehensive loss function, L macro is the macro strategy network loss function, L micro is the micro-control network loss function, α is the trade-off coefficient, L align is the gradient alignment loss; L policy is the strategy loss, L value is the value function loss, L entropy is the entropy regularization loss, β1 and β2 are weight coefficients; π θ is the parameterized policy function, π θ (a t |s t ) means that in a given state s t Next take action a t The probability of π θ (a|s t ) means that in a given state s t The probability distribution of taking any action a under t =R long-term -V(s t ) is the advantage function, V(s t ) is the state value function, In the strategy π θ The expected value under , ‖·‖ represents the Euclidean norm; L TD is the timing difference loss, L smooth is the smoothing loss, L constraint is the constraint loss, μ1 and μ2 are weight coefficients; Q is the action value function, Q(s t ,a t) means that in a given state s t Next take action a t The expected return that can be obtained, Q(s t+1 ,a t+1 ) means that in a given state s t+1 Next take action a t+1 The expected return that can be obtained, γ is the discount factor; a t 、a t-1 are the action instructions for the current time step and the previous time step respectively; ‖a t ‖ indicates action instruction a t The modulus length, that is, a t The Euclidean norm of a max is the maximum allowed modulus of the action space.

[0073] Preferably, the following attention calculation formula is used in the temporal interaction attention module:

[0074]

[0075] Where Q, K, and V are query, key, and value vectors respectively; d k is the dimension of the key vector; M(t) is the temporal mask function; G(ΔT) is the time interval adaptation function, which dynamically adjusts the attention intensity according to the predicted time span.

[0076] Preferably, step S4 includes:

[0077] S41, loading the trained dual network structure model into the ship control system, initializing the model parameters and setting the operating environment;

[0078] S42. Receive ship status parameters and ocean environment parameters in real time using a sliding time window method, wherein the sliding time window method includes dynamic updating of historical data and current data;

[0079] S43, based on the received ship state parameters and ocean environment parameters, using the macro strategy network to generate a heading planning strategy;

[0080] S44, generating specific control instructions, including rudder angle distribution and propulsion force distribution, based on the heading planning strategy generated by the macro strategy network and combined with the micro control network;

[0081] S45. Monitor the ship's motion status and environmental changes in real time, dynamically adjust the control strategy based on the predicted environmental changes, generate optimal control instructions, and achieve real-time optimized control of the ship's motion trajectory.

[0082] The present invention has the following beneficial effects compared to the prior art:

[0083] (1) The present invention realizes the real-time optimization control of the ship's motion trajectory by constructing a dual-network structure model and a hierarchical training strategy. This method can dynamically adjust the control strategy in a complex and changeable marine environment, improve the safety and economy of ship navigation, and enhance the overall performance and response speed of the ship control system; (2) The dual-network structure combining the macro-strategy network and the micro-control network is adopted to effectively distinguish between the generation of long-term heading planning and short-term control instructions. The macro-strategy network is responsible for generating the overall heading planning strategy to ensure that the navigation direction of the ship is reasonable within a large range; the micro-control network generates specific rudder angles and propulsion force instructions based on real-time status and environmental parameters, thereby achieving precise control of the ship's motion. This structural design enhances the coordination ability of the control system at different time scales, improves the accuracy of trajectory planning and the execution accuracy of control instructions;

[0084] (3) Through a hierarchical training strategy, the macro-strategy network and the micro-control network are trained sequentially to ensure that each network independently improves performance under its own optimization goal. At the same time, a gradient alignment mechanism is introduced to maintain parameter consistency between the two networks and avoid policy deviations caused by different training sequences or optimization goals. This strategy effectively improves the stability and consistency of the collaborative work of the two networks, ensuring that the entire control system can coordinate and improve performance during the optimization process.

[0085] (4) The designed comprehensive evaluation function comprehensively considers the trajectory tracking error, energy consumption and control command smoothness. Through the adaptive adjustment of the dynamic weight coefficient, the dynamic balance of multi-objective optimization is achieved, so that the control system can take into account energy efficiency and control stability while ensuring trajectory accuracy.

[0086] (5) The temporal interactive attention module is introduced to achieve dynamic information alignment between the output of the macro-strategy network and the input of the micro-control network through the temporal mask function and the time interval adaptation function, thereby improving the real-time performance and accuracy of the dual-network collaborative control. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0088] Figure 1 is a flow chart of the method of the present invention;

[0089] Figure 2 It is a framework diagram of the dual network structure model of the present invention;

[0090] Figure 3 This is the model training process diagram of the present invention DETAILED DESCRIPTION

[0091] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0092] like Figure 1 As shown, the present invention provides a ship motion trajectory optimization control method based on deep learning, comprising:

[0093] S1. Constructing a ship motion simulation environment and establishing a dynamic simulation system including ocean environment parameters; collecting ship operation data, wherein the operation data includes ship state parameters and ocean environment parameters, wherein the ship state parameters are obtained through a shipboard sensing system, and the ocean environment parameters are obtained through an environmental monitoring system;

[0094] S2. Construct a dual-network structure model, including a macro-strategy network and a micro-control network. The macro-strategy network generates a course planning strategy at longer intervals, while the micro-control network generates specific control instructions at shorter intervals. The dual-network structure model takes as input ship state parameters and ocean environment parameters, and the two networks interact with each other through a temporal attention mechanism.

[0095] S3. A hierarchical training strategy is used to train the dual-network structure model in a ship motion simulation environment. First, the macro-strategy network is trained to generate the overall heading strategy, and then the micro-control network is trained to achieve precise control. The macro-strategy network is optimized based on long-term rewards, and the micro-control network is optimized based on short-term control accuracy. The two networks are maintained consistent through a gradient alignment mechanism.

[0096] S4. Deploy the trained dual-network structure model to the ship control system, implement predictive control through the sliding time window method, receive ship state parameters and ocean environment parameters in real time, dynamically adjust the control strategy based on the predicted environmental changes, generate optimal control instructions, and realize real-time optimization control of the ship's motion trajectory.

[0097] In one embodiment of the present invention, the ship state parameters include: ship position coordinates, heading angle, speed, pitch angle, roll angle, yaw angle, bow angular velocity, lateral velocity, and longitudinal velocity; the ocean environment parameters include: wind speed, wind direction, wave height, wave direction, water flow velocity, water flow direction, water depth, and sea temperature. Step S1 includes:

[0098] Building a ship motion simulation environment:

[0099] Establish a 3D ship model: Use the NURBS surface modeling method to construct the hull geometry model and describe the hull shape characteristics; establish a mass distribution model based on the finite element meshing method to calculate the mass, center of gravity position and moment of inertia of each mesh unit.

[0100] Construct a hydrodynamics module: use three-dimensional potential flow theory to calculate the radiation and diffraction forces of the hull; use the Morison formula to calculate the viscous hydrodynamic force; establish a Green's function method solver that takes into account the free surface effect to calculate the wave excitation force.

[0101] Set up the propulsion system model: establish a fixed-pitch propeller model, using the B-series propeller characteristic curve; construct a servo system model; use four-quadrant thrust coefficients and torque coefficients to describe the hydrodynamic characteristics of the propeller-rudder system.

[0102] Establish an environmental force model: the Isherwood wind coefficient method is used for wind force calculation; the wave force is calculated based on the second-order wave theory, taking into account the first-order and second-order wave forces; the water flow force is calculated using the relative velocity method, taking into account the distribution of water flow velocity on the hull surface.

[0103] Integrated dynamic simulation system:

[0104] (1) Marine environmental parameter generation module: The JONSWAP spectrum model is used to generate irregular wave fields; the logarithmic wind speed profile model is used to generate wind field data; and the current field data is generated based on the tidal current forecast model.

[0105] (2) Ship motion equation solver: Establish a numerical solver based on the six-degree-of-freedom motion equation; use the fourth-order Runge-Kutta method for time domain integration; set motion constraints.

[0106] (3) Shipborne sensing system: simulates the data acquisition process of GPS receivers, fiber optic gyroscopes, motion sensors, Doppler speed meters and other equipment; environmental monitoring system: simulates the data acquisition process of ultrasonic anemometers, wave radars, acoustic Doppler current profilers, temperature sensors and other equipment.

[0107] (4) Data storage and analysis module: Use a time series database to store collected sensor data; set up a data caching mechanism; implement data preprocessing functions, including outlier detection, data smoothing, and resampling; establish a data analysis interface to support real-time data statistics and historical data queries.

[0108] In one embodiment of the present invention, Figure 2 As shown, in step S2, the dual network structure model includes:

[0109] Macro Strategy Network:

[0110] The input layer is provided with a first input sublayer and a second input sublayer for receiving ship status parameters and ocean environment parameters respectively; the input sublayers are provided in parallel.

[0111] The timing coding layer sets a timing coding module after each input sub-layer to perform feature marking on the input signal with a long time interval to maintain the timing information.

[0112] The feature extraction layer combines a bidirectional long short-term memory network with a one-dimensional convolutional network to process the input ship state parameters and ocean environment parameters in parallel, and extract time series features respectively. The bidirectional long short-term memory network contains two hidden layers to capture long-term dependencies. The one-dimensional convolutional network uses multiple convolutional layers with different convolution kernel sizes to extract local time series features.

[0113] The feature fusion layer fuses the features extracted by the bidirectional long short-term memory network and the one-dimensional convolutional network through a gating mechanism to form a comprehensive feature representation; the gating mechanism includes an update gate and a reset gate to control the selective fusion of features.

[0114] The strategy generation layer connects two layers of fully connected networks. Each layer of the network uses a learnable residual connection and the activation function is Swish. It is used to generate the heading planning strategy vector.

[0115] Micro-control network:

[0116] The input layer sets a multi-channel convolution input structure for data with shorter time intervals, mapping the high-frequency sampled ship state parameters and ocean environment parameters into a multi-dimensional feature space.

[0117] The multi-scale convolution layer combines dilated convolution and variable kernel convolution to capture the temporal features of the micro-control stage at multiple scales. Dilated convolution expands the receptive field by setting different expansion rates, and variable kernel convolution adapts to features of different scales by learning different convolution kernel shapes.

[0118] The attention enhancement layer adopts a multi-head causal attention mechanism, setting multiple attention heads. Each attention head independently calculates the feature correlation between the current moment and the historical moment, and performs weighted aggregation to assign feature weights of different historical moments to the control output of the current moment.

[0119] The control output layer is connected to a two-branch fully connected network. One branch is used to output the rudder angle distribution, and the other branch is used to output the propulsion force distribution. Both branches use the Mish activation function.

[0120] Cross-network information interaction mechanism:

[0121] The temporal interactive attention module is set between the policy generation layer of the macro policy network and the input layer of the micro control network to achieve dynamic alignment of information between the output of the macro policy network and the input of the micro control network. This module uses the self-attention mechanism to achieve dynamic alignment of information. Specifically, the heading planning strategy vector output by the macro policy network is used as the query vector, and the historical state characteristics of the micro control network are used as the key vector and value vector. By calculating the similarity between the query vector and the key vector, and combining the temporal mask function and the time interval adaptation function, the selective use of historical information is achieved, ensuring that the micro control network can generate appropriate control instructions based on the planning results of the macro policy network. This module uses the following attention calculation formula:

[0122]

[0123] Where Q, K, and V are query, key, and value vectors respectively; d k is the dimension of the key vector; M(t) is the temporal mask function; G(ΔT) is the time interval adaptation function, which dynamically adjusts the attention intensity according to the predicted time span.

[0124] Through the design and connection of the above network structure, the macro-strategy network optimizes long-term heading planning, and the micro-control network accurately generates short-term control instructions. At the same time, the temporal interactive attention module ensures information coordination between the two networks.

[0125] In one embodiment of the present invention, Figure 3 As shown, step S3 includes:

[0126] S31. Construct comprehensive evaluation function R t , based on the comprehensive evaluation function R t Define the long-term return R long-term and short-term immediate reward R short-term .

[0127] Before model training begins, a comprehensive evaluation function is first constructed as the optimization objective for training. This evaluation function comprehensively considers three aspects: trajectory tracking error, energy consumption, and control command smoothness, and achieves a balance of multiple objectives through dynamic weight coefficients. The calculation formula of the comprehensive evaluation function is as follows:

[0128] R t =w1·f traj (s t )+w2·f energy (a t )+w3·f smooth (s t ,a t )

[0129] Where R tis the reward value at time step t; f traj (s t ) represents state s t The trajectory tracking error when f energy (a t ) means taking action a t Energy consumption when smooth (s t ,a t ) represents state s t Next take action a t w1, w2, and w3 are dynamic weight coefficients. w1, w2, and w3 are adaptively adjusted based on the current sea conditions, mission phase, and control objectives. Specifically, in adverse sea conditions, the system increases the weight w1 for trajectory tracking error to ensure control accuracy. In smooth sea conditions, the system increases the weight w2 for energy consumption to improve energy efficiency. During missions requiring rapid maneuvers, the system decreases the weight w3 for control smoothness to improve response speed. During stable navigation, w3 is increased to ensure control smoothness.

[0130] Specifically, f traj (s t ) represents the deviation between the current ship position and the target trajectory, and the calculation formula is:

[0131]

[0132] In the formula, (x t ,y t ) is the current ship position; (x tar ,y tar ) is the target position; θ t is the current heading angle; θ tar is the target heading angle; ω1 and ω2 are weight coefficients; the negative sign indicates that the larger the error, the smaller the reward value.

[0133] f energy (a t ) represents the energy consumption required to execute the control action, and the calculation formula is:

[0134]

[0135] Where n t is the propeller speed ratio, which represents the ratio of the desired propeller speed to the rated speed; δ t is the rudder angle; c1 and c2 are energy consumption coefficients; the negative sign means that the greater the energy consumption, the smaller the reward value.

[0136] f smooth (s t ,a t) represents the degree of change between adjacent control instructions, and the calculation formula is:

[0137] f smooth (s t ,a t )=-[(n t -n t-1 ) 2 +k·(δ t -δ t-1 ) 2 ]

[0138] Where n t 、n t-1 are the propeller speed ratios at the current and previous moments respectively; δ t , δ t-1 are the rudder angles at the current and previous moments respectively; k is the weight coefficient of the rudder angle change; the negative sign indicates that the greater the change in the control command, the smaller the reward value.

[0139] Based on the comprehensive evaluation function, long-term rewards and short-term immediate rewards are defined. The calculation of long-term rewards takes into account the timing discount and sea state adaptability, and its calculation formula is:

[0140]

[0141] Where, ρ t is the timing discount factor; g(H t ) is the sea state adaptation function, according to the current sea state H t Adjusted reward strength.

[0142] Short-term immediate reward R short-term The calculation formula is: R short-term =R t , that is, the short-term immediate reward directly uses the reward value calculated by the comprehensive evaluation function at the current time step t.

[0143] S32. Define the state space S and action space A of the ship's motion, and the state vector s t ∈S is defined as a 21-dimensional vector:

[0144]

[0145] In the formula, (x t ,y t ) is the position coordinate of the ship in the global coordinate system; θ t is the heading angle; v t is the ship speed; pit is the pitch angle; φ roll is the roll angle; φ yaw is the head swing angle; ω yaw is the yaw angular velocity; v latis the lateral speed; v long is the longitudinal velocity; v wind is the wind speed; θ wind is the wind direction; h wave is the wave height; θ wave is the wave direction; v cur is the water flow velocity; θ cur is the direction of water flow; d depth is the water depth; T water is the sea temperature; (x tar ,y tar ) is the target position coordinate; φ task The current task stage;

[0146] Motion vector a t ∈A is defined as a two-dimensional vector of propeller speed ratio and rudder angle:

[0147] a t =[n t ,δ t ] T

[0148] Where n t is the propeller speed ratio, which represents the ratio of the desired propeller speed to the rated speed; δ t is the rudder angle.

[0149] S33, first train the macro strategy network, based on the long-term return R long-term Conduct iterative optimization of the course planning strategy; adopt a multi-batch experience replay mechanism to screen key training samples covering various sea conditions.

[0150] Specifically, the model training adopts a hierarchical strategy, first training the macro-policy network. In this stage, a multi-batch experience replay mechanism is used to select key samples containing various sea conditions from historical data for training. The loss function of the macro-policy network includes policy loss, value function loss, and entropy regularization loss:

[0151] L macro =L policy +β1·L value +β2·L entropy

[0152] in:

[0153]

[0154]

[0155] Where, L macro is the macro strategy network loss function, L policy is the strategy loss, L value is the value function loss, Lentropy is the entropy regularization loss, β1 and β2 are weight coefficients; π θ is the parameterized policy function, π θ (a t |s t ) means that in a given state s t Next take action a t The probability of π θ (a|s t ) means that in a given state s t The probability distribution of taking any action a under t =R long-term -V(s t ) is the advantage function, V(s t ) is the state value function, In the strategy π θ The expected value under , ‖·‖ represents the Euclidean norm.

[0156] Strategy loss L policy Used to optimize the policy function by maximizing the log-likelihood weighted by the advantage function. The value function loss L value The state value function is optimized by minimizing the mean squared error between the predicted value and the actual return. Entropy regularization loss L entropy Used to maintain the exploratory nature of the strategy and prevent premature convergence to a suboptimal strategy.

[0157] In this embodiment, the multi-batch experience playback mechanism is implemented as follows:

[0158] First, define each experience sample e in the experience replay pool D as a five-tuple:

[0159] e=(s t ,a t ,R t ,s t+1 ,H t )

[0160] Among them, s t is the current state, a t For the action to be performed, R t For the reward, s t+1 is the next state, H t The current sea conditions.

[0161] The core calculation formula of the experience replay mechanism is as follows:

[0162] Sample importance weight calculation: Among them, g(H t ) is the sea state adaptation function, which is used to increase the diversity of samples in different sea conditions. max The maximum reward value in the experience pool.

[0163] Batch division and sampling probability: P(e)=w(e) / ∑w(e i ), where P(e) represents the probability that sample e is selected.

[0164] Multi-batch sampling strategy: Batch i ={e j |e j ~P(e),j=1,2,...,M}, where Batch i Indicates the i-th batch, M is the batch size. Each training cycle samples N batches to form a batch set: Batch = {Batch1, Batch2, ..., Batch N}.

[0165] Calculation of batch difference: D(Batch i ,Batch i )=‖μ(Batch i )-μ(Batch j )‖ 2 ,μ(Batch i ) represents batch i The feature mean of all samples in .

[0166] Batch screening criteria: Batch * =argmax Batch [∑ i ∑ j D(Batch i ,Batch i )]; subject to |Batch| = K, where K is the number of batches retained, and the diversity of samples is ensured by maximizing the difference between batches.

[0167] S34, after the macro strategy network initially converges, switch to the training phase of the micro control network; based on the short-term immediate reward R short-term , conduct reinforcement learning iterations of the micro-control network and generate specific control strategies.

[0168] Specifically, the loss function of the micro-control network includes temporal difference loss, smoothing loss and constraint loss:

[0169] L micro =L TD +μ1·L smooth +μ2·L constraint

[0170] in:

[0171]

[0172] L smooth =‖a t -a t-1 ‖ 2

[0173] L constraint =max(0,‖a t ‖-a max ) 2

[0174] Where, L micro is the loss function of the micro-control network, L TD is the timing difference loss, L smooth is the smoothing loss, L constraint is the constraint loss, μ1 and μ2 are weight coefficients; Q is the action value function, Q(s t ,a t ) means that in a given state s t Next take action a t The expected return that can be obtained, Q(s t+1 ,a t+1 ) means that in a given state s t+1 Next take action a t+1 The expected return that can be obtained, γ is the discount factor; a t 、a t-1 are the action instructions for the current time step and the previous time step respectively; ‖a t ‖ indicates action instruction a t The modulus length, that is, a t The Euclidean norm of a max is the maximum allowed modulus of the action space.

[0175] Timing difference loss L TD Used to optimize the action value function by minimizing the difference between the current Q value and the target Q value. Smooth loss L smooth Used to ensure the continuity of control instructions and constrain loss L constraint Ensure that control instructions are within reasonable range.

[0176] S35. Through dynamic task allocation, different environmental complexity scenarios are introduced to enable the macro strategy network and micro control network to adapt to different sea conditions respectively.

[0177] Specifically, we first establish an environmental complexity assessment system and define the environmental complexity index C:

[0178] C=w1·f(H wave )+w2·f(H wind )+w3·f(H current )

[0179] Where, f(H wave ) represents the wave complexity, which is calculated from the wave height and direction; f(H wind ) represents the complexity of wind conditions, which is calculated from wind speed and direction; f(H current ) represents the complexity of water flow, which is calculated by flow velocity and flow direction; w1, w2, and w3 are weight coefficients.

[0180] Task allocation formula: T macro =f macro (C); T micro =f micro (C); where T macro and T micro denote the training tasks assigned to the macro policy network and the micro control network, respectively, and f macro 、f micro Function for dynamically adjusting task allocation.

[0181] S36. A gradient alignment module is established during training to monitor the policy differences between the macro-policy network and the micro-control network. Through the gradient alignment mechanism, the parameters of the macro-policy network and the micro-control network are adjusted synchronously.

[0182] The gradient alignment mechanism is as follows:

[0183] For the macro strategy network and micro control network, the loss function L is defined respectively macro and L micro ;

[0184] Calculate the respective gradient vectors and Among them, θ macro and θ micro represent the parameter sets of the macro-strategy network and the micro-control network respectively;

[0185] Use the gradient alignment formula to align the gradient vectors of the two networks and adjust the network parameters synchronously. The gradient alignment formula is as follows:

[0186]

[0187] Where, L align is the gradient alignment loss, λ is the trade-off coefficient, η(t) is the time-varying adjustment factor, and ‖·‖ represents the Euclidean norm.

[0188] S37. After the macro-strategy network and the micro-control network are trained to the preset convergence standard, they are jointly fine-tuned. During the joint fine-tuning process, the macro-strategy network and the micro-control network are simultaneously optimized by minimizing the comprehensive loss function. When the fine-tuning is completed, a trained dual-network structure model is obtained.

[0189] The joint fine-tuning process uses a comprehensive loss function for overall optimization:

[0190] L total =L macro +L micro +α·L align

[0191] α is the trade-off coefficient.

[0192] By minimizing the combined loss function, we achieve collaborative optimization of the macro-strategy network and the micro-control network. During the joint optimization process, we dynamically adjust the weight coefficients of each loss term to balance different training objectives. Training ends when model performance no longer significantly improves.

[0193] In one embodiment of the present invention, step S4 includes:

[0194] S41. Load the trained dual network structure model into the ship control system, initialize the model parameters and set the operating environment.

[0195] First, the trained dual-network model, including the network structure and weight parameters of the macro-policy network and the micro-control network, is loaded into the ship control system. The model's runtime environment is initialized, setting the parameter ranges for the state space and action space. The state space consists of a 21-dimensional state vector, and the action space contains two control variables: the propeller speed ratio and the rudder angle. Furthermore, the parameters of the temporal interactive attention module are configured, including the dimensions of the query vector, key vector, and value vector, as well as the initial parameters of the temporal masking function and the time interval adaptation function.

[0196] S42. Receive ship status parameters and ocean environment parameters in real time through a sliding time window method, wherein the sliding time window method includes dynamic updating of historical data and current data.

[0197] A sliding time window approach is used to receive and update data in real time, with the window length dynamically adjusted based on the control cycle. The ship's state parameters, including position coordinates, heading angle, speed, pitch angle, roll angle, yaw angle, bow angular velocity, lateral velocity, and longitudinal velocity, are collected in real time via the onboard sensing system. Marine environmental parameters, including wind speed, direction, wave height, wave direction, current velocity, current direction, water depth, and sea temperature, are collected via the environmental monitoring system. The system maintains a certain length of historical data and dynamically updates this data through a sliding window mechanism, ensuring the model captures continuous time-series information.

[0198] S43. Based on the received ship state parameters and ocean environment parameters, a heading planning strategy is generated using a macro strategy network.

[0199] Based on data collected over a sliding time window, a macro-strategy network is activated to generate a heading planning strategy. First, a bidirectional long short-term memory network and a one-dimensional convolutional network are used to process the input ship state parameters and ocean environment parameters in parallel to extract temporal features. These features are then fused through a gating mechanism to form a comprehensive feature representation. Finally, a two-layer fully connected network, combined with the Swish activation function, generates a heading planning strategy vector to provide guidance for micro-control.

[0200] S44. Based on the heading planning strategy generated by the macro-strategy network, specific control instructions are generated in combination with the micro-control network, including rudder angle distribution and propulsion force distribution.

[0201] The micro-control network receives the heading planning strategy generated by the macro-strategy network and dynamically aligns this information through a temporal interactive attention module. A multi-channel convolutional input structure processes high-frequency sampled data, and a combination of dilated and variable kernel convolutions captures multi-scale temporal features. A multi-head causal attention mechanism assigns feature weights at different historical moments to the current control output. Finally, a two-branch fully connected network, combined with a Mish activation function, outputs specific control commands for rudder angle distribution and thrust force distribution, respectively.

[0202] S45. Monitor the ship's motion status and environmental changes in real time, dynamically adjust the control strategy based on the predicted environmental changes, generate optimal control instructions, and achieve real-time optimized control of the ship's motion trajectory.

[0203] The system continuously monitors the ship's motion and environmental changes, predicting environmental trends through the time interval adaptive function G(ΔT) within the temporal interactive attention module. When significant environmental changes are detected, the system automatically adjusts the control strategy and recalculates the optimal control instructions. During this adjustment process, the macro-strategy network updates the course plan based on the predicted environmental changes, and the micro-control network adjusts specific control parameters accordingly, achieving real-time optimal control of the ship's trajectory, ensuring the ship can safely and efficiently complete its scheduled navigation mission.

[0204] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A ship motion trajectory optimization control method based on deep learning, characterized in that: include: S1. Build a ship motion simulation environment and establish a dynamic simulation system including ocean environment parameters; Collecting ship operation data, including ship status parameters and ocean environment parameters, wherein the ship status parameters are obtained through a shipboard sensing system and the ocean environment parameters are obtained through an environmental monitoring system; S2. Construct a dual-network structure model, including a macro-strategy network and a micro-control network. The macro-strategy network generates a course planning strategy at longer intervals, while the micro-control network generates specific control instructions at shorter intervals. The dual-network structure model takes as input ship state parameters and ocean environment parameters, and the two networks interact with each other through a temporal attention mechanism. S3. A hierarchical training strategy is used to train the dual-network structure model in a ship motion simulation environment. First, the macro-strategy network is trained to generate the overall heading strategy, and then the micro-control network is trained to achieve precise control. The macro-strategy network is optimized based on long-term rewards, and the micro-control network is optimized based on short-term control accuracy. The two networks are maintained consistent through a gradient alignment mechanism. S4. Deploy the trained dual-network structure model to the ship control system, implement predictive control through a sliding time window method, receive ship state parameters and ocean environment parameters in real time, dynamically adjust the control strategy based on the predicted environmental changes, generate optimal control instructions, and achieve real-time optimized control of the ship's motion trajectory; The gradient alignment mechanism is as follows: For the macro strategy network and micro control network, the loss function L is defined respectively macro and L micro ; Calculate the respective gradient vectors and in, and represent the parameter sets of the macro-strategy network and the micro-control network respectively; Use the gradient alignment formula to align the gradient vectors of the two networks and adjust the network parameters synchronously. The gradient alignment formula is as follows: Where, L align is the gradient alignment loss, λ is the trade-off coefficient, η(t) is the time-varying adjustment factor, and ‖·‖ represents the Euclidean norm.

2. The ship motion trajectory optimization control method based on deep learning according to claim 1, characterized in that: The ship status parameters include: the ship's position coordinates, heading angle, speed, pitch angle, roll angle, yaw angle, bow angular velocity, lateral speed and longitudinal speed; the ocean environment parameters include: wind speed, wind direction, wave height, wave direction, water flow speed, water flow direction, water depth and sea temperature.

3. The ship motion trajectory optimization control method based on deep learning according to claim 2, characterized in that: In step S1, building a ship motion simulation environment includes: Establish a three-dimensional ship model, including the hull geometry model and mass distribution model; Build a hydrodynamics module to calculate various hydrodynamic forces acting on ships; Set up the propulsion system model, including the main thruster and steering gear system; Establish an environmental force model to simulate the effects of wind, wave and current forces; The dynamic simulation system is integrated into the ship motion simulation environment, including: Construct an ocean environment parameter generation module to generate time-varying environment parameters; Establish a ship motion equation solver to calculate the ship's motion response in complex sea conditions; Set up a sensor data simulation module to simulate the data acquisition process of the shipborne sensor system and environmental monitoring system; Establish a data storage and analysis module to record and process various types of data during the simulation process.

4. The ship motion trajectory optimization control method based on deep learning according to claim 2, characterized in that: The dual network structure model includes: Macro Strategy Network: An input layer, comprising a first input sublayer and a second input sublayer for receiving ship state parameters and ocean environment parameters respectively; The temporal coding layer sets a temporal coding module after each input sublayer to perform feature marking on input signals with long time intervals; The feature extraction layer combines a bidirectional long short-term memory network with a one-dimensional convolutional network to process the input ship state parameters and ocean environment parameters in parallel and extract time series features respectively; The feature fusion layer fuses the features extracted by the bidirectional long short-term memory network and the one-dimensional convolutional network through a gating mechanism to form a comprehensive feature representation; The strategy generation layer connects two fully connected networks, using a learnable residual connection between each network layer and the activation function Swish to generate the heading planning strategy vector; Micro-control network: The input layer sets a multi-channel convolution input structure for data with shorter time intervals, mapping the high-frequency sampled ship state parameters and ocean environment parameters into a multi-dimensional feature space; The multi-scale convolution layer combines dilated convolution with variable kernel convolution to capture the temporal features of the micro-control stage at multiple scales. The attention enhancement layer uses a multi-head causal attention mechanism to assign feature weights of different historical moments to the control output at the current moment; The control output layer is connected to a two-branch fully connected network. One branch is used to output the rudder angle distribution, and the other branch is used to output the propulsion force distribution. The activation function is Mish. Cross-network information interaction mechanism: The temporal interactive attention module is set between the policy generation layer of the macro policy network and the input layer of the micro control network to achieve dynamic alignment of information between the output of the macro policy network and the input of the micro control network.

5. The ship motion trajectory optimization control method based on deep learning according to claim 4 is characterized in that: Step S3 includes: S31. Construct comprehensive evaluation function R t , based on the comprehensive evaluation function R t Define the long-term return R long-term and short-term immediate reward R short-term ; S32. Define the state space S and action space A of the ship's motion, and the state vector s t ∈S is defined as: In the formula, (x t ,y t ) is the position coordinate of the ship in the global coordinate system; θ t is the heading angle; v t is the ship speed; pit is the pitch angle; φ roll is the roll angle; φ yaw is the head swing angle; ω yaw is the yaw angular velocity; v lat is the lateral speed; v long is the longitudinal velocity; v wind is the wind speed; θ wind is the wind direction; h wave is the wave height; θ wave is the wave direction; v cur is the water flow velocity; θ cur is the direction of water flow; d depth is the water depth; T water is the sea temperature; (x tar ,y tar ) is the target position coordinate; φ task The current task stage; Motion vector a t ∈A is defined as: a t =[n t ,d t ] T Where n t is the propeller speed ratio, which represents the ratio of the desired propeller speed to the rated speed; δ t is the rudder angle; S33, first train the macro strategy network, based on the long-term return R long-term Iteratively optimize the course planning strategy; use a multi-batch experience replay mechanism to screen key training samples covering various sea conditions; S34, after the macro strategy network initially converges, switch to the training phase of the micro control network; based on the short-term immediate reward R short-term , conduct reinforcement learning iteration of the micro-control network to generate specific control strategies; S35. By dynamically allocating tasks, different environmental complexity scenarios are introduced to enable the macro-strategy network and the micro-control network to adapt to different sea conditions. S36. During training, a gradient alignment module is established to monitor the policy differences between the macro policy network and the micro control network. Through the gradient alignment mechanism, the parameters of the macro policy network and the micro control network are adjusted synchronously. S37. After the macro-strategy network and the micro-control network are trained to the preset convergence standard, they are jointly fine-tuned. During the joint fine-tuning process, the macro-strategy network and the micro-control network are simultaneously optimized by minimizing the comprehensive loss function. When the fine-tuning is completed, a trained dual-network structure model is obtained.

6. The ship motion trajectory optimization control method based on deep learning according to claim 5, characterized in that: The calculation formula of the comprehensive evaluation function is as follows: R t =w1·f traj (s t )+w2·f energy (a t )+w3·f smooth (s t ,a t ) Where R t is the reward value at time step t; f traj (s t ) represents state s t The trajectory tracking error when f energy (a t ) means taking action a t Energy consumption when smooth (s t ,a t ) represents state s t Next take action a t w1, w2, w3 are dynamic weight coefficients; Long-term return R long-term The calculation formula is as follows: Where, ρ t is the timing discount factor; g(H t ) is the sea state adaptation function, according to the current sea state H t Adjust reward intensity; Short-term immediate reward R short-term The calculation formula is: R short-term =R t , that is, the short-term immediate reward directly uses the reward value calculated by the comprehensive evaluation function at the current time step t.

7. The ship motion trajectory optimization control method based on deep learning according to claim 5, characterized in that: The comprehensive loss function is as follows: L total =L macro +L micro +α·L align L macro =L policy +β1·L value +β2·L entropy L micro =L TD +μ1·L smooth +μ2·L constraint L smooth =‖a t -a t-1 ‖ 2 L constraint =max(0,‖a t ‖-a max ) 2 Where, L total is the comprehensive loss function, L macro is the macro strategy network loss function, L micro is the micro-control network loss function, α is the trade-off coefficient, L align is the gradient alignment loss; L policy is the strategy loss, L value is the value function loss, L entropy is the entropy regularization loss, β1 and β2 are weight coefficients; is a parameterized policy function, Indicates that in a given state s t Next take action a t The probability of Indicates that in a given state s t The probability distribution of taking any action a under t =R long-term -V(s t ) is the advantage function, V(s t ) is the state value function, Indicates that in the strategy The expected value under , ‖·‖ represents the Euclidean norm; L TD is the timing difference loss, L smooth is the smoothing loss, L constraint is the constraint loss, μ1 and μ2 are weight coefficients; Q is the action value function, Q(s t ,a t ) means that in a given state s t Next take action a t The expected return that can be obtained, Q(s t+1 ,a t+1 ) means that in a given state s t+1 Next take action a t+1 The expected return that can be obtained, γ is the discount factor; a t 、a t-1 are the action instructions for the current time step and the previous time step respectively; ‖a t ‖ indicates action instruction a t The modulus length, that is, a t The Euclidean norm of a max is the maximum allowed modulus of the action space.

8. The ship motion trajectory optimization control method based on deep learning according to claim 4, characterized in that: The following attention calculation formula is used in the temporal interaction attention module: Where Q, K, and V are query, key, and value vectors respectively; d k is the dimension of the key vector; M(t) is the temporal mask function; G(ΔT) is the time interval adaptation function, which dynamically adjusts the attention intensity according to the predicted time span.

9. The ship motion trajectory optimization control method based on deep learning according to claim 5, characterized in that: Step S4 includes: S41, loading the trained dual network structure model into the ship control system, initializing the model parameters and setting the operating environment; S42. Receive ship status parameters and ocean environment parameters in real time using a sliding time window method, wherein the sliding time window method includes dynamic updating of historical data and current data; S43, based on the received ship state parameters and ocean environment parameters, using the macro strategy network to generate a heading planning strategy; S44, generating specific control instructions, including rudder angle distribution and propulsion force distribution, based on the heading planning strategy generated by the macro strategy network and combined with the micro control network; S45. Monitor the ship's motion status and environmental changes in real time, dynamically adjust the control strategy based on the predicted environmental changes, generate optimal control instructions, and achieve real-time optimized control of the ship's motion trajectory.

Images