Unmanned ship path following system and method based on deep reinforcement learning

Through deep reinforcement learning technology, the neural network is trained using simulation platform and DDPG algorithm, the problems of poor complexity and portability of traditional unmanned ship path following methods are solved, and efficient and stable path following of unmanned ships in complex environments are achieved.

CN114859910BActive Publication Date: 2024-11-12WUHAN UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210470023.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-28
Publication Date
2024-11-12
Estimated Expiration
2042-04-28

AI Technical Summary

Technical Problem

The traditional unmanned ship path following method has complex calculations and poor portability, making it difficult to deal with changes in navigation conditions and environmental interference, resulting in poor path following effect.

Method used

Adopting an unmanned ship path following system based on deep reinforcement learning, the ship motion model and control algorithm are separated through simulation platform, Markov decision modeling, neural network construction and line-of-sight guidance algorithm, and deep neural network is trained using DDPG algorithm to achieve no The path of the man and ship follows the control.

Benefits of technology

The control strategy design is simplified, the adaptability and real-time nature of path follow-up is improved, the dependence on professional knowledge is reduced, and the navigation stability and portability of unmanned ships in complex environments is significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114859910B_ABST
    Figure CN114859910B_ABST
Patent Text Reader

Abstract

The present invention discloses an unmanned ship path following system based on deep reinforcement learning, wherein the simulation platform construction module is used to construct an unmanned ship motion interactive simulation platform; the Markov decision modeling module is used to model the Markov decision process using the unmanned ship motion control task; the neural network construction module is used to design a deep neural network based on the DDPG algorithm architecture according to the state space, action space and reward function in the Markov decision process; the strategy model construction module uses the DDPG algorithm to train the deep neural network on the simulation platform to obtain the unmanned ship path following control strategy model; the path following control module is used to combine the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control. The present invention separates the ship motion model from the control algorithm, simplifies the design process of the control strategy, and significantly reduces or eliminates the reliance on professional knowledge in the field of ship motion control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned ship motion control, and in particular to an unmanned ship path following system and method based on deep reinforcement learning. Background Art

[0002] Highly autonomous and intelligent unmanned ships are an inevitable trend in the development of the shipbuilding and shipping industries. Unmanned ships aim to achieve unmanned and intelligent ships, which can effectively improve the safety of equipment and ship operations, optimize navigation strategies, and reduce operating costs. As a result, unmanned ships have become a key development direction for major shipbuilding countries and maritime powers. Path following, as one of the basic tasks of unmanned ship motion control, is the key to achieving autonomous and intelligent navigation of unmanned ships.

[0003] The traditional unmanned ship path following method is based on mathematical estimation analysis. The controller parameters are determined through mathematical analysis and derivation. The controller design and parameter setting process rely on strong professional knowledge. At present, the effective performance of the path following method based on mathematical estimation analysis has been proven, but it has great limitations, such as high computational complexity, poor portability, and great influence of environmental interference. In particular, it is easily affected by factors such as changes in navigation conditions and environmental interference during navigation, showing strong uncertainty, nonlinearity and time-varying. It is difficult to establish an accurate mathematical model to express the changing state of the ship, which brings great challenges to the unmanned ship path following task. Unmanned ships need to adapt to changes in navigation conditions and adjust control strategies in time during real-time navigation, but traditional control methods cannot handle uncertainty problems well, resulting in poor path following effect. Summary of the invention

[0004] The purpose of the present invention is to provide an unmanned ship path following system and method based on deep reinforcement learning. The present invention separates the ship motion model from the control algorithm, simplifies the design process of the control strategy, and significantly reduces or eliminates the dependence on professional knowledge in the field of ship motion control.

[0005] To achieve this purpose, the present invention designs an unmanned ship path following system based on deep reinforcement learning, which includes a simulation platform building module, a Markov decision modeling module, a neural network building module, a strategy model building module and a path following control module;

[0006] The simulation platform construction module is used to construct an unmanned ship motion interactive simulation platform, initialize the target path and navigation environment of the unmanned ship in the unmanned ship motion interactive simulation platform, and define the unmanned ship motion control task according to the requirements of the unmanned ship path following;

[0007] The Markov decision modeling module is used to model the Markov decision process using the unmanned ship motion control task. The Markov decision process is used to describe the interactive process of the unmanned ship path following control. The state space of the Markov decision process is determined according to the information required by the ship to complete the path following task, the action space of the Markov decision process is determined according to the ship control instruction, and the reward function of the Markov decision process is determined according to the target task of the ship control.

[0008] The neural network building module is used to design a deep neural network based on the DDPG (deep deterministic policy gradient) algorithm architecture according to the state space, action space and reward function in the Markov decision process;

[0009] The strategy model building module uses the DDPG algorithm to train the deep neural network on the simulation platform to obtain the unmanned ship path following control strategy model;

[0010] The path following control module is used to combine the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control.

[0011] Deep reinforcement learning has the characteristics of autonomous learning and strong self-adaptation, and has been widely used in navigation, robot control, parameter optimization and other fields. At the same time, the end-to-end learning method in deep reinforcement learning significantly reduces or eliminates the dependence on professional knowledge and has a good performance that surpasses human professionals in many fields, which provides new ideas for research in the field of decision-making and control. The path following method based on deep reinforcement learning requires the construction of an environment for interactive learning, which can be a real environment or a simulation environment. The actual environment has problems such as high risk, long cycle, and high cost, and the numerical simulation method cannot intuitively reflect the movement process of the ship.

[0012] Beneficial effects of the present invention:

[0013] The unmanned ship motion interactive simulation platform constructed by the present invention provides a training and testing environment close to the actual environment, separates the ship motion control from the control algorithm, and can directly design the unmanned ship control algorithm without understanding the underlying principle of ship motion control. The ship motion process is visualized with Unity3D, which intuitively reflects the control effect of the control strategy, and provides a safe, efficient and low-cost approach for the research and verification of the decision-making and control strategy of unmanned ships.

[0014] The present invention proposes an unmanned ship path following method based on deep reinforcement learning, in which a neural network trained by a DDPG algorithm is used as the unmanned ship path following controller, which does not require complex mathematical derivation and simplifies the design process of the controller; in the training process, interactive data consisting of control input and the corresponding dynamic response of the unmanned ship are used as training samples of the neural network, implicitly considering the modeling uncertainty problem and the influence of various environmental factors, and better handling the uncertainty problem; the trained control strategy model can be applied to real-time path following tasks after fine-tuning, and has high portability. The control strategy model takes the real-time state of the unmanned ship as input, directly outputs the optimal control strategy of the ship, and has strong adaptability and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural schematic diagram of the present invention;

[0016] Figure 2 A method flow chart of the unmanned ship path following method based on deep reinforcement learning provided by the present invention;

[0017] Figure 3 This is a system architecture diagram of the unmanned ship motion interactive simulation platform in the present invention;

[0018] Figure 4 This is a schematic diagram of the unmanned ship path following controller based on the DDPG algorithm in the present invention;

[0019] Figure 5 This is the principle diagram of the line of sight method;

[0020] Figure 6 This is a training framework diagram of the unmanned ship path following based on the DDPG algorithm in the present invention;

[0021] Figure 7 is an execution flow chart of the control strategy model in the present invention; DETAILED DESCRIPTION

[0022] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0023] like Figure 1 The unmanned ship path following system based on deep reinforcement learning is characterized in that: it includes a simulation platform building module, a Markov decision modeling module, a neural network building module, a strategy model building module and a path following control module;

[0024] The simulation platform building module is used to build an unmanned ship motion interactive simulation platform, such as Figure 3As shown, the target path and navigation environment of the unmanned ship are initialized in the unmanned ship motion interactive simulation platform. The navigation environment includes the target path, the initial position of the ship, the heading and obstacle information. The unmanned ship motion control task is defined according to the requirements of the unmanned ship path following;

[0025] The Markov decision modeling module is used to model the Markov decision process using the unmanned ship motion control task. The Markov decision process is used to describe the interactive process of the unmanned ship path following control. The state space of the Markov decision process is determined according to the information required by the ship to complete the path following task, the action space of the Markov decision process is determined according to the ship control instruction, and the reward function of the Markov decision process is determined according to the target task of the ship control. The ship control instruction is the action that the ship can execute, and the ship control task is what kind of task the ship is to complete; the target task of the ship control is the target of the ship path following, specifically, to minimize the difference between the ship's heading angle and the target heading angle, minimize the position error of the path following, and smoothly follow the target path; the Markov decision process is used to describe the interactive process of the unmanned ship path following control, and converts the ship path following control problem into a discrete sequential decision problem. The state space, action space, and reward function are all components of the Markov decision process;

[0026] The Markov decision process is described by (S, A, P, R, γ), and the state space, action space, and reward function are all its components; the state space is a collection of descriptions of the ship's state, describing the ship's state information; the expected heading angle in the state space is obtained by the line of sight method; the action space is a set of ship control actions, which defines what actions the ship can take; the reward function is the feedback signal from the environment after the ship takes a certain action, which is used to evaluate the quality of the action taken and optimize the control strategy through this signal.

[0027] The neural network building module is used to design a deep neural network based on the DDPG algorithm architecture according to the state space, action space and reward function in the Markov decision process;

[0028] The strategy model building module uses the DDPG algorithm to train the deep neural network on the simulation platform to obtain the unmanned ship path following control strategy model. The training process is that the unmanned ship collects experience data according to the Markov decision process, stores it in the experience pool, and then randomly samples from the experience pool to train the neural network;

[0029] The path following control module is used to combine the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control, such as Figure 4 .

[0030] In the above technical solution, the simulation platform construction module constructs a virtual 3D navigation environment through the Unity3D engine, and constructs a control model of the ship's heading and speed in the Pycharm software. The ship's navigation speed and steering are controlled through an interface in PyCharm to give the ship a longitudinal thrust and torque. The communication toolkit of the Airsim software is used to realize data interaction between the virtual 3D navigation environment and the control model of the ship's heading and speed, thereby forming an unmanned ship motion interactive simulation platform.

[0031] In the above technical scheme, an unmanned ship motion interactive simulation platform is constructed for the training and evaluation of unmanned ship motion control strategies; the simulation platform includes a virtual environment layer, an application program interface layer, and an external environment layer, wherein the virtual environment layer constructs a virtual environment based on the ship motion model, the unmanned ship navigation scene, the environmental model, and the sensor model, and is visualized with Unity3D, which can realize ship motion simulation and sensor information content perception; the external environment layer can be used to design the decision-making and control strategy of the unmanned ship; the application program interface layer connects the ship motion simulation with the control strategy through the data transmission protocol, and realizes the transmission of environmental perception information and unmanned ship control signals between the virtual environment layer and the external environment layer.

[0032] Optionally, the external environment layer is isolated from the unmanned ship motion model in the virtual environment. The core of the external environment layer is the ship motion control strategy, which only involves the design and implementation of the ship motion control strategy. Therefore, the ship control method of the external environment layer can be the method proposed by the present invention or the technology proposed by other institutions.

[0033] In the above technical solution, the specific method of modeling the Markov decision process is:

[0034] The Markov decision process is described by the tuple (S, A, P, R, γ), where S is the state space of the unmanned ship, A is the action space of the unmanned ship, P is the state transition probability, R is the reward function, and γ is the discount factor used to weigh the relationship between immediate rewards and future long-term rewards. At time t, the state information of the unmanned ship s t ∈S, select action a from the action space according to the corresponding action strategy t Execute, then the state of the unmanned ship is transferred to a new state s t+1 , and get the feedback reward value r t ,The mission goal of the unmanned ship is to maximize the cumulative reward value in the ,process of completing the interaction;

[0035] Construct the state space of the unmanned ship; the state space S is expressed as: where ψ d represents the desired heading angle, χ is the difference between the ship heading angle and the desired heading, e yrepresents the lateral position error between the ship's position and the desired route, u is the ship's speed, is the derivative of the expected heading angle, heading error, and lateral position error; the expected heading is obtained by the line of sight method; the line of sight method obtains the expected heading of the ship based on the real-time position information of the ship and the target path information; the real-time position information is obtained through the ship perception system; the target path information is known information.

[0036] The line of sight method provides the desired heading for the unmanned ship. The principle diagram is as follows Figure 5 As shown, assuming P k+1 (x k+1 ,y k+1 ) is the current target waypoint, P k (x k ,y k ) is the previous waypoint, P k+2 (x k+2 ,y k+2 ) is the next target waypoint, the current position of the ship is P(x,y), P los (x los ,y los ) is the LOS point, and the equation for solving the LOS point is:

[0037]

[0038] Among them, α k is the path angle, R is the turning radius;

[0039] Therefore, the expected heading is:

[0040] ψ d =atan2(y los -y,x los -x)

[0041] Among them, atan2 function is the inverse tangent function.

[0042] Constructing an unmanned ship action space; the action space A is expressed as: A = {δ}, where δ is the ship rudder angle;

[0043] Construct a reward function; the reward function includes the relative position and relative heading of the target path, expressed as: r = w e r e +w χ r χ , where r e is the position error reward, r χ is the heading error reward, w e is the position error reward weight, w χ is the heading error reward weight; the position error reward and heading error reward are:

[0044]

[0045] Among them, χ(k) is the heading error at the current moment, χ(k-1) represents the heading error at the previous moment, k1 is the first weight coefficient, which is 0.1, k2 is the second weight coefficient, which is 0.01, k3 is the third weight coefficient, which is 0.1, e is a natural constant, and rad represents radians.

[0046] In the above technical solution, the unmanned ship path following learning framework based on DDPG algorithm is as follows: Figure 6 As shown, the deep neural network includes: a current strategy network, a current evaluation network, a target strategy network, and a target evaluation network; the current strategy network uses the current state information s of the unmanned ship t As input, it corresponds to the ship state information at time t in the description of the Markov decision process, that is, the vector composed of the specific values ​​corresponding to each variable in the state space, and outputs the current strategy action μ(s t ), μ(s t ) is the action value output by the strategy network at the current moment, for action μ(s t ) After adding OU (Ornstein-Uhlenback) exploration noise, we get the action a to be executed t The output of the policy network corresponds to a value in the action space. Specifically, at time t, the policy network inputs the ship state s t When the action value is output, that is, the ship rudder angle; the current evaluation network is based on the state and action pair (s t ,a t ) is the input, state and action pair, from the state s t and action a t Composition, a t is the current policy network output action value μ(s t ) Join OU to explore noise Then get the action to be performed Output calculation of the current Q value (Q value is a proper noun, indicating the state s t Execute action a t The current Q value is the output value of the current evaluation network); the target policy network has the same structure as the current policy network, and the target policy network has the next state s t+1 is the input, and the output is the optimal action μ'(s t+1 |θ μ' ), the target evaluation network is in the state s at the next moment t+1 and the optimal action μ'(s t+1 |θ μ' ) is the input, and the output is the estimated Q value Q'(s t+1,μ'(s t+1 |θ μ' )|θ Q' ); the target evaluation network has the same structure as the current evaluation network.

[0047] In the above technical solution, the dimension of the state space defines the input of the current policy network; the dimension of the action space defines the output of the current policy network; the state space dimension and the action space dimension define the input of the current evaluation network; the number of input layer nodes and the number of output layer nodes of the neural network are determined by the state space and the action space.

[0048] In the above technical solution, the network parameter of the current strategy network is θ μ , taking the current state of the unmanned ship s t As input, output current policy action μ(s t ), after adding OU exploration noise, we get the action a to be executed t ;The network parameter of the current evaluation network is θ Q , with state and action pairs (s t ,a t ) is input, and the output is the current Q value Q(s t ,a t |θ Q );The network parameters of the target strategy network are θ μ′ , the state s at the next moment t+1 is the input, and the output is the optimal action μ'(s t+1 |θ μ' );The network parameter of the target evaluation network is θ Q′ , the state s at the next moment t+1 and the optimal action μ'(s t+1 |θ μ' ) is the input, and the output is the estimated Q value Q'(s t+1 ,μ'(s t+1 |θ μ' )|θ Q' ).

[0049] In the above technical solution, the specific method for obtaining the unmanned ship path following control strategy model is:

[0050] Step 4.1, initialize the number of training rounds M, the maximum number of steps per round T, the soft update rate τ, the learning rate of the current policy network and the current evaluation network, and the decay factor γ in the deep neural network;

[0051] Step 4.2: Initialize the current policy network parameters θ μ , current evaluation network parameters θ Q 、Target strategy network parameters θ μ′, target evaluation network parameters θ Q′ And experience replay pool D;

[0052] Step 4.3, for each training round, randomly initialize the ship state s0;

[0053] Step 4.4: For each time step, obtain the current state information s of the unmanned ship from the virtual environment layer of the unmanned ship motion interactive simulation platform t , and then input the current policy network to get the policy action μ(s t ), plus the noise exploration signal to synthesize the unmanned boat action a t , the unmanned ship performs unmanned ship actions in the virtual environment a t , according to the reward function, the unmanned ship is in state s t (The current state of the unmanned ship corresponds to the ship state information at time t in the description of the Markov decision process, that is, the vector composed of the specific values ​​corresponding to each variable in the state space) and executes action a t The reward value r t , and get the new state s of the unmanned ship t+1 ;

[0054] Step 4.5: Transform the interactive experience data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool D, and the interactive experience data includes the reward function to obtain the unmanned ship state s t , unmanned boat action a t , reward value r t , the new state of unmanned ships t+1 ,When the amount of interactive experience data exceeds the capacity of the experience pool, the earliest experience data is replaced by the latest experience data;

[0055] Step 4.6: Randomly sample batches of N tuples of data (s i ,a i ,r i ,s i+1 ) as training data, s i is the current state of the i-th sample, a i is the current state of the i-th sample, r i is the current moment reward in the i-th sample, s i+1 For the next state in the i-th sample, calculate the target value function:

[0056] y i =r i +γQ'(s i+1 ,μ'(si+1 |θ μ' )|θ Q' )

[0057] y i is the target Q value, r i represents the reward value, γ represents the decay factor, Q'(s i+1 ,μ'(s i+1 |θ μ' )|θ Q' ) is the Q value output by the target evaluation network, s i+1 represents the new state of the unmanned ship, μ'(s i+1 |θ μ' ) is the output action of the target policy network, θ μ' represents the target policy network parameters, θ Q' represents the target evaluation network parameters;

[0058] Step 4.7: Update the current evaluation network by minimizing the loss function:

[0059]

[0060] L(θ Q ) represents the loss function of the current evaluation network, Q(s i ,a i |θ Q ) represents the output Q value of the current evaluation network, s i Indicates the state of unmanned ship, a i represents the action of the unmanned ship, θ Q Represents the parameters of the current evaluation network;

[0061] Step 4.8: Update the current policy network using policy gradient:

[0062]

[0063] in, represents the gradient of the objective function of the current policy network, E represents the mean function, represents the gradient of the Q value output by the current evaluation network to action a, a = μ(s i ) represents the output action value of the current strategy network when the state of the unmanned ship is si, Represents the action value pair θ output by the current policy network μ The gradient of

[0064] Step 4.9: Update the target network parameters by soft update:

[0065] θ Q’ ←τθ Q +(1-τ)θ Q’

[0066] θ μ’ ←τθ μ +(1-τ)θ μ’

[0067] Where τ represents the soft update rate, θ Q Represents the current evaluation network parameters, θ Q' represents the target evaluation network parameters, θ μ represents the current policy network parameters, θ μ' represents the target strategy network parameters;

[0068] Step 4.10: Repeat steps 4.4 to 4.9 until the maximum number of steps T is reached;

[0069] Step 4.11, repeat steps 4.3 to 4.10 until the maximum number of rounds M is reached, and the unmanned ship path following control strategy model is obtained;

[0070] Through the above training process, an optimal control strategy model is obtained, and the control strategy model is the current strategy network parameters after training.

[0071] In the above technical solution, the path following control module combines the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control. The line of sight guidance algorithm is used to provide the desired heading for the control strategy model to guide the unmanned ship to navigate. The path following process is realized, specifically including, Figure 7 As shown:

[0072] Step 5.1, initializing the target path of the ship's navigation;

[0073] Step 5.2, obtain the initial state information and environmental information of the unmanned ship, and calculate the value of each variable in the state space by obtaining the ship's position, heading, speed information and target path information to obtain the current state of the ship;

[0074] Step 5.3, input the current state information of the unmanned ship into the unmanned ship path following control strategy model to obtain the current action strategy of the unmanned ship;

[0075] Step 5.4, convert the action strategy into a rudder angle control command, execute the rudder angle control command, and transfer the unmanned ship to a new state;

[0076] Step 5.5, obtain the real-time position information of the ship and calculate the current deviation distance of the ship;

[0077] Step 5.6: If the deviation distance exceeds the threshold, it means that the mission execution fails and the ship returns to the initial position; otherwise, proceed to the next step;

[0078] Step 5.7: If the unmanned ship reaches the destination, the path following control process ends; otherwise, proceed to the next step;

[0079] Step 5.8: If the unmanned ship completes following the current target path, the next target path is used as the current target path and the process returns to step 5.2; otherwise, the process returns directly to step 5.2.

[0080] A path following method for an unmanned ship, such as Figure 2 As shown, it includes the following steps:

[0081] Step 1: Build an unmanned ship motion interactive simulation platform, initialize the target path and navigation environment of the unmanned ship in the unmanned ship motion interactive simulation platform, and define the unmanned ship motion control task according to the requirements of the unmanned ship path following;

[0082] The system architecture diagram of the unmanned ship motion interactive simulation platform is as follows: Figure 3 As shown in the figure, the entire simulation platform includes a virtual environment layer, an application program interface layer, and an external environment layer. The virtual environment layer builds a virtual environment based on the ship motion model, the unmanned ship navigation scene, the environmental model, and the sensor model, and visualizes it with Unity3D. It can control the ship's heading and speed, and can sense the ship's position, speed, heading information, and navigation environment information; the external environment layer can be used to design the decision-making and control strategy of the unmanned ship; the application program interface layer connects the ship motion simulation with the control strategy through the data transmission protocol, and realizes the transmission of environmental perception information and unmanned ship control signals between the virtual environment layer and the external environment layer;

[0083] Step 2: Use the unmanned ship motion control task to model the Markov decision process. The Markov decision process is used to describe the interactive process of the unmanned ship path following control. The state space of the Markov decision process is determined according to the information required by the ship to complete the path following task. The action space of the Markov decision process is determined according to the ship control command. The reward function of the Markov decision process is determined according to the target task of the ship control.

[0084] Step 3: Based on the state space, action space and reward function in the Markov decision process, design a deep neural network based on the DDPG algorithm architecture;

[0085] Step 4: Use the DDPG algorithm on the simulation platform to train the deep neural network and obtain the unmanned ship path following control strategy model;

[0086] Step 5: Combine the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control.

[0087] Compared with the prior art, the unmanned ship motion interactive simulation platform constructed by the present invention separates the ship motion model and the control algorithm, shields the underlying implementation principle of the ship motion controller, greatly simplifies the design of the control strategy, and the simulation platform also provides a training and testing environment close to the actual environment, providing a safe, efficient and low-cost way for the research and verification of the decision-making and control strategy of the unmanned ship; at the same time, the unmanned ship path following method based on deep reinforcement learning proposed by the present invention uses the neural network trained by the DDPG algorithm as the unmanned ship path following controller, which does not require complex mathematical derivation and simplifies the controller design process; in the training process, the interactive data composed of the control input and the corresponding dynamic response of the unmanned ship is used as the training sample of the neural network, implicitly considering the modeling uncertainty problem and the influence of various environmental factors, and better handling the uncertainty problem; the trained control strategy model can be applied to real-time path following tasks after fine-tuning, with high portability, and the control strategy model takes the real-time state of the unmanned ship as input, directly outputs the optimal control strategy of the ship, and has strong adaptability and real-time performance.

[0088] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. An unmanned ship path following system based on deep reinforcement learning, characterized in that: It includes a simulation platform building module, a Markov decision modeling module, a neural network building module, a strategy model building module and a path following control module; The simulation platform construction module is used to construct an unmanned ship motion interactive simulation platform, initialize the target path and navigation environment of the unmanned ship in the unmanned ship motion interactive simulation platform, and define the unmanned ship motion control task according to the requirements of the unmanned ship path following; The Markov decision modeling module is used to model the Markov decision process using the unmanned ship motion control task. The Markov decision process is used to describe the interactive process of the unmanned ship path following control. The state space of the Markov decision process is determined according to the information required by the ship to complete the path following task, the action space of the Markov decision process is determined according to the ship control instruction, and the reward function of the Markov decision process is determined according to the target task of the ship control. The neural network building module is used to design a deep neural network based on the DDPG algorithm architecture according to the state space, action space and reward function in the Markov decision process; The strategy model building module uses the DDPG algorithm to train the deep neural network on the simulation platform to obtain the unmanned ship path following control strategy model; The path following control module is used to combine the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control; the specific method of modeling the Markov decision process is: The Markov decision process is described by the tuple (S, A, P, R, γ), where S is the state space of the unmanned ship, A is the action space of the unmanned ship, P is the state transition probability, R is the reward function, and γ is the discount factor used to weigh the relationship between immediate rewards and future long-term rewards. At time t, the state information of the unmanned ship s t ∈S, select action a from the action space according to the corresponding action strategy t Execute, then the state of the unmanned ship transfers to a new state s t+1 , and get the feedback reward value r t ,The mission goal of the unmanned ship is to maximize the cumulative reward value in the ,process of completing the interaction; Construct the state space of the unmanned ship; the state space S is expressed as: where ψ d represents the desired heading angle, χ is the difference between the ship heading angle and the desired heading, e y represents the lateral position error between the ship's position and the desired route, u is the ship's speed, is the derivative of the desired heading angle, heading error, and lateral position error; Constructing an unmanned ship action space; the action space A is expressed as: A = {δ}, where δ is the ship rudder angle; Construct a reward function; the reward function includes the relative position and relative heading of the target path, expressed as: r = w e r e +w χ r χ , where r e is the position error reward, r χ is the heading error reward, w e is the position error reward weight, w χ is the heading error reward weight; The position error reward and heading error reward are: Among them, χ(k) is the heading error at the current moment, χ(k-1) represents the heading error at the previous moment, k1 is the first weight coefficient, k2 is the second weight coefficient, k3 is the third weight coefficient, e is a natural constant, and rad represents radians.

2. The unmanned ship path following system based on deep reinforcement learning according to claim 1, characterized in that: The simulation platform construction module constructs a virtual 3D navigation environment through the Unity3D engine, and constructs a control model for the ship's heading and speed in the Pycharm software. The communication toolkit of the Airsim software is used to realize data interaction between the virtual 3D navigation environment and the control model for the ship's heading and speed, thereby forming an unmanned ship motion interactive simulation platform.

3. The unmanned ship path following system based on deep reinforcement learning according to claim 1, characterized in that: The deep neural network includes: a current strategy network, a current evaluation network, a target strategy network, and a target evaluation network; the current strategy network uses the current state information s of the unmanned ship t As input, output current policy action μ(s t ); The current evaluation network is a state and action pair (s t ,a t ) is input, and the output is the current Q value; the target strategy network has the same structure as the current strategy network, and the target strategy network is in the next state s t+1 is the input, and the output is the optimal action μ'(s t+1 θ μ '), the target evaluation network is in the state s at the next moment t+1 and the optimal action μ'(s t+1 θ μ ') is the input, and the output is the estimated Q value Q'(s t+1 ,μ'(s t+1 θ μ ')θ Q '); The target evaluation network has the same structure as the current evaluation network.

4. The unmanned ship path following system based on deep reinforcement learning according to claim 1, characterized in that: The dimension of the state space defines the input of the current policy network; the dimension of the action space defines the output of the current policy network; the state space dimension and the action space dimension define the input of the current evaluation network; the number of input layer nodes and the number of output layer nodes of the neural network are determined by the state space and the action space.

5. The unmanned ship path following system based on deep reinforcement learning according to claim 3 is characterized in that: The network parameters of the current policy network are θ μ , taking the current state of the unmanned ship s t As input, output current policy action μ(s t ), after adding OU exploration noise, we get the action a to be executed t ;The network parameter of the current evaluation network is θ Q , taking the state and action pair (st,at) as input, outputs the current Q value Q(s t ,a t |θ Q );The network parameter of the target strategy network is θ μ′ , the state s at the next moment t+1 is the input, and the output is the optimal action μ'(s t+1 θ μ '); the network parameter of the target evaluation network is θ Q′ , the state s at the next moment t+1 and the optimal action μ'(s t+1 θ μ ') is the input, and the output is the estimated Q value Q'(s t+1 ,μ'(s t+1 θ μ ')θ Q ').

6. The unmanned ship path following system based on deep reinforcement learning according to claim 1, characterized in that: The specific method for obtaining the unmanned ship path following control strategy model is: Step 4.1, initialize the number of training rounds M, the maximum number of steps per round T, the soft update rate τ, the learning rate of the current policy network and the current evaluation network, and the decay factor γ in the deep neural network; Step 4.2: Initialize the current policy network parameters θ μ , current evaluation network parameters θ Q 、Target strategy network parameters θ μ′ , target evaluation network parameters θ Q′ And experience replay pool D; Step 4.3, for each training round, randomly initialize the ship state s0; Step 4.4: For each time step, obtain the current state information s of the unmanned ship from the virtual environment layer of the unmanned ship motion interactive simulation platform t , and then input the current policy network to get the policy action μ(s t ), plus the noise exploration signal to synthesize the unmanned boat action a t , the unmanned ship performs unmanned ship actions in the virtual environment a t , according to the reward function, the unmanned ship is in state s t Next, perform action a t The reward value r t , and get the new state s of the unmanned ship t+1 ; Step 4.5: Store the interactive experience data in the experience replay pool D. The interactive experience data includes the reward function to obtain the unmanned ship state s t , unmanned boat action a t , reward value r t , the new state of unmanned ships t+1 ,When the amount of interactive experience data exceeds the capacity of the experience pool, the earliest experience data is replaced by the latest experience data; Step 4.6: Randomly sample batches of N tuples of data (s i ,a i ,r i ,s i+1 ) as training data, s i is the current state of the i-th sample, a i is the current state of the i-th sample, r i is the current moment reward in the i-th sample, s i+1 For the next state in the i-th sample, calculate the target value function: y i =r i +γQ'(s i+1 ,m'(s i+1 |θ μ' )|θ Q' ) y i is the target Q value, r i represents the reward value, γ represents the decay factor, Q'(s i+1 ,μ'(s i+1 θ μ ')θ Q ') is the Q value output by the target evaluation network, s i+1 represents the new state of the unmanned ship, μ'(s i+1 θ μ ') is the output action of the target policy network, θ μ' represents the target policy network parameters, θ Q' represents the target evaluation network parameters; Step 4.7: Update the current evaluation network by minimizing the loss function: L(θ Q ) represents the loss function of the current evaluation network, Q(s i ,a i θ Q ) represents the output Q value of the current evaluation network, s i Indicates the state of unmanned ship, a i represents the action of the unmanned ship, θ Q Represents the parameters of the current evaluation network; Step 4.8: Update the current policy network using policy gradient: in, represents the gradient of the objective function of the current policy network, E represents the mean function, represents the gradient of the Q value output by the current evaluation network to action a, a = μ(s i ) represents the output action value of the current strategy network when the state of the unmanned ship is si, Represents the action value pair θ output by the current policy network μ The gradient of Step 4.9: Update the target network parameters by soft update: i Q′ ←tth Q +(1-τ)θ Q′ i μ′ ←tth μ +(1-τ)θ μ′ Where τ represents the soft update rate, θ Q Represents the current evaluation network parameters, θ Q' represents the target evaluation network parameters, θ μ represents the current policy network parameters, θ μ' represents the target strategy network parameters; Step 4.10: Repeat steps 4.4 to 4.9 until the maximum number of steps T is reached; Step 4.11: Repeat steps 4.3 to 4.10 until the maximum number of rounds M is reached, and the unmanned ship path following control strategy model is obtained.

7. The unmanned ship path following system based on deep reinforcement learning according to claim 1, characterized in that: The path following control module combines the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control. The line of sight guidance algorithm is used to provide the control strategy model with the desired heading to guide the unmanned ship to navigate. Implement the path following process, specifically including: Step 5.1, initializing the target path of the ship's navigation; Step 5.2, obtain the initial state information and environmental information of the unmanned ship, and calculate the value of each variable in the state space by obtaining the ship's position, heading, speed information and target path information to obtain the current state of the ship; Step 5.3, input the current state information of the unmanned ship into the unmanned ship path following control strategy model to obtain the current action strategy of the unmanned ship; Step 5.4, convert the action strategy into a rudder angle control command, execute the rudder angle control command, and transfer the unmanned ship to a new state; Step 5.5, obtain the real-time position information of the ship and calculate the current deviation distance of the ship; Step 5.6: If the deviation distance exceeds the threshold, it means that the mission execution fails and the ship returns to the initial position; otherwise, proceed to the next step; Step 5.7: If the unmanned ship reaches the destination, the path following control process ends; otherwise, proceed to the next step; Step 5.8: If the unmanned ship completes following the current target path, the next target path is used as the current target path and the process returns to step 5.2; otherwise, the process returns directly to step 5.

2.

8. The unmanned ship path following system based on deep reinforcement learning according to claim 1, characterized in that: The desired heading is obtained through the line of sight method; the line of sight method obtains the desired heading of the ship based on the real-time position information of the ship and the target path information; the real-time position information is obtained through the ship perception system; the target path information is known information.

9. A method for following the path of an unmanned ship of the system of claim 1, characterized in that: It includes the following steps: Step 1: Build an unmanned ship motion interactive simulation platform, initialize the target path and navigation environment of the unmanned ship in the unmanned ship motion interactive simulation platform, and define the unmanned ship motion control task according to the requirements of the unmanned ship path following; Step 2: Use the unmanned ship motion control task to model the Markov decision process. The Markov decision process is used to describe the interactive process of the unmanned ship path following control. The state space of the Markov decision process is determined according to the information required by the ship to complete the path following task. The action space of the Markov decision process is determined according to the ship control command. The reward function of the Markov decision process is determined according to the target task of the ship control. Step 3: Based on the state space, action space and reward function in the Markov decision process, design a deep neural network based on the DDPG algorithm architecture; Step 4: Use the DDPG algorithm on the simulation platform to train the deep neural network and obtain the unmanned ship path following control strategy model; Step 5: Combine the unmanned ship path following control strategy model with the line of sight guidance algorithm to realize the unmanned ship path following control.

Citation Information

Patent Citations

  • Unmanned ship path tracking method based on reinforcement learning

    CN112947431A

  • AUV action plan and operation control method based on reinforcement learning

    JP2021034050A