Unmanned ship path tracking control method based on deep reinforcement learning

Abstract

The invention relates to an unmanned ship path tracking control method based on deep reinforcement learning. The method comprises the following steps: step 1, constructing a time-varying marine environment model fusing multiple environmental elements; step 2, based on the marine environment model in the step 1, modeling an unmanned ship path tracking control problem as a Markov decision process-based solution framework; step 3, on the basis of the solving framework in the step 2, carrying out optimization solving on the path tracking control of the unmanned ship by using an improved Twin Delayed Deed Defined Definition (TD3) algorithm, and carrying out optimization solving on the path tracking control of the unmanned ship by using the improved Twin Delayed Defined Definition (TD3) algorithm; and step 4, using the strategy neural network trained in the step 3 to control the unmanned ship to execute a path tracking task. According to the unmanned ship path tracking control method based on deep reinforcement learning, the improved TD3 algorithm is adopted for strategy training and optimization, the convergence speed of the control strategy is remarkably increased, and the accurate and reliable path tracking effect in the dynamic and complex marine environment is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous motion control of unmanned ships, and in particular relates to an unmanned ship path tracking control method based on deep reinforcement learning. Background Art

[0002] With the continuous advancement of the plan to build a strong maritime nation, marine resources have been continuously developed and utilized. At the same time, people have a greater demand for the intelligence and autonomy of marine equipment. Among them, unmanned ships are widely used in resource exploration, maritime search and rescue, intelligence reconnaissance and other fields due to their strong maneuverability, long range and high autonomy. In the study of motion control of unmanned ships, high-precision path tracking technology in complex marine environments is still an important and challenging problem. For path tracking control, its main function is to generate motion control commands so that the unmanned ship can track the reference path with minimal dynamic error.

[0003] The ocean environment for unmanned ship path tracking is dynamic and complex, and is affected by environmental factors such as ocean currents and wind. In actual navigation, these environmental factors may cause the unmanned ship to deviate from the reference path, resulting in huge tracking errors. However, early studies ignored the impact of environmental factors on the movement of unmanned ships, so the practical application value of their ocean environment models was low. The use of simple mathematical function models or uniform vector fields to simulate the ocean environment lacks environmental authenticity. In addition, the current ocean environment ignores the dynamic changes of environmental factors. With the development of marine meteorological data acquisition and prediction technology, real ocean observation data and prediction data have been widely used in other fields. Therefore, it is crucial to establish a real and accurate dynamic ocean environment model based on real ocean data for unmanned ship path tracking control.

[0004] In the past few decades, various studies have been devoted to solving the path tracking problem of unmanned ships. Traditional control methods, such as sliding mode control (SMC), model predictive control (MPC), linear active disturbance rejection controller (LADRC), etc., are widely used in path tracking control of unmanned ships. However, these model-based controllers are essentially based on mathematical estimation analysis, which leads to high computational complexity and poor portability. In order to overcome the problems of the above traditional controllers, intelligent path tracking controllers have been proposed, including fuzzy logic control and neural network-based control. However, these intelligent controllers face challenges such as high requirements for expert experience, poor adaptability to dynamic environments, and slow reasoning speed. However, the path tracking control of unmanned ships in complex marine environments places higher requirements on the model adaptability and real-time model reasoning capabilities of the methods.

[0005] In recent years, with the development of deep learning and reinforcement learning technologies, the introduction of deep reinforcement learning has provided the possibility to meet the above requirements. The deep reinforcement learning method can control extremely complex systems without the need for accurate system dynamics models and prior knowledge. It is widely used in intelligent robot control, recommendation systems and other fields. For the path tracking control of unmanned ships, the unmanned ship agent learns the optimal control strategy in dynamic and complex marine environments through trial and error interaction with the environment. Specifically, the deep reinforcement learning method controls the action output of the unmanned ship, allowing the unmanned ship to continuously interact with the simulation environment in real time, and learn and train a control strategy with the minimum path tracking error. The classic deep reinforcement learning methods DQN and DDPG have the problem of overestimating the Q value, which will lead to model instability and poor control strategy effect.

[0006] In summary, the control method of unmanned ship path tracking has made great progress in theoretical research, but many researchers have not carefully considered the impact of environmental factors on the movement of unmanned ships, resulting in unsatisfactory results in practical applications. In addition, traditional control methods are not robust and adaptable.

[0007] After searching, no prior art documents identical or similar to the present invention were found. Summary of the invention

[0008] In order to overcome the shortcomings of environmental authenticity and effectiveness of control strategies in unmanned ship path tracking control, the present invention proposes an unmanned ship path tracking control method based on deep reinforcement learning. By introducing real ocean data, a time-varying ocean environment model integrating multiple environmental elements is constructed; then, the unmanned ship path tracking control problem is modeled as a solution framework based on the Markov decision process, and a reasonable state action space and reward function are designed; the unmanned ship path tracking control is optimized and solved using an improved TD3 algorithm, and the trained strategy neural network is used to control the unmanned ship to perform the path tracking task. The present invention can achieve rapid and stable convergence of the control strategy, so that the unmanned ship can accurately track the reference path in a complex ocean environment.

[0009] The present invention solves the practical problem by adopting the following technical solutions:

[0010] A path tracking control method for an unmanned ship based on deep reinforcement learning comprises the following steps:

[0011] Step 1: Construct a time-varying marine environment model integrating multiple environmental elements;

[0012] Step 2: Based on the ocean environment model in step 1, the unmanned ship path tracking control problem is modeled as a solution framework based on the Markov decision process;

[0013] Step 3: Based on the solution framework of step 2, the improved Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is used to optimize the path tracking control of the unmanned ship;

[0014] Step 4: Use the strategy neural network trained in step 3 to control the unmanned boat to perform the path tracking task.

[0015] Moreover, the specific steps of constructing the marine environment model in step 1 include:

[0016] 1.1 Obtain ocean current and wind field environmental data of part of the South China Sea (113.5°E-115.5°E, 17.25°N to 19.25°N) from the National Marine Science Data Center;

[0017] 1.2 The area is divided into 120×120 grids using the grid method, and the ocean current and wind field environment data are interpolated into the grid environment using the inverse distance weighted interpolation method;

[0018] 1.3 When the USV is at position P, the environmental data at a certain time t is expressed as:

[0019]

[0020] Where I P (t) is the environmental information received by the USV in real time at position P and time t through sensors such as velocity profiler and anemometer, and represent the lateral and longitudinal velocities of the ocean current, respectively. and They represent the lateral and longitudinal speeds of the wind respectively.

[0021] The environmental data I(t) of the non-grid point location P can be obtained from the surrounding four adjacent grid points P i Environmental data I at (i=1,...,4) i (t) is obtained by distance weighting and is calculated by the following formula:

[0022]

[0023] Moreover, the step 2 specifically includes:

[0024] The unmanned ship path tracking control problem is modeled as a solution framework based on the Markov decision process, which specifically includes the state space S, action space A, reward function R, state transition relationship P, and discount factor γ of the Markov decision process:

[0025] Among them, the state space S represents the set of state information s perceived by the unmanned ship, including motion parameter information sm , environmental information e , LOS guidance information LOS , the state s at the tth control moment t It is expressed as:

[0026] s t ={s m ,s e ,s LOS}

[0027] Where: motion parameter information s m ={(x, y), U, ψ}, where (x, y) is the current position of the unmanned ship, U and ψ are the current speed and heading angle of the unmanned ship respectively; as described in step 1, the environmental data, environmental information s e =I P (t);LOS guidance information s LOS =(ψ d ,x e ,y e ), where ψ d is the ideal heading angle of the unmanned ship, x e and e They are longitudinal tracking error and lateral tracking error respectively. The LOS guidance method is used to calculate the expected heading angle and tracking error to improve the tracking accuracy. The expected heading angle can be expressed as:

[0028]

[0029] In the formula, ψ k is the slope of the tracking path, Δ is the forward distance of the unmanned ship, and β is the sideslip angle;

[0030] For the unmanned ship at position (x, y), the parameterized path P is tracked k (t), the tracking error can be expressed as:

[0031]

[0032] Among them, the action space A represents the set of actions a of the unmanned ship, and the action a at the tth control moment is t It is defined as the output speed and heading angle of the unmanned ship. The heading angle is controlled by controlling the direction of the rudder, and the speed of the unmanned ship is adjusted by controlling the thrust of the propeller, as shown in the following formula:

[0033]

[0034] Where U(t) represents the speed of the unmanned ship, Indicates the heading angle of the unmanned ship;

[0035] Among them, the reward function R defines the unmanned ship in different states st and action a t The reward signal r obtained under t , which is used to guide the learning and decision-making process of the control strategy, is defined as follows:

[0036]

[0037] Where K1 and K2 are the proportional parameters assigned to different error penalty terms, t step is the total control time; in addition, when the unmanned ship successfully completes the tracking task, it will receive a huge reward incentive. However, if the unmanned ship fails to track within the specified number of steps or deviates from the tracking area, it will receive a huge penalty.

[0038] Among them, the state transition relationship is defined as the probability of updating from the current environmental state to the next state after the unmanned ship makes an action decision, as follows:

[0039] P(S t+1 |S t ×[A] t )=ρ

[0040] Where ρ is the environmental transfer rate, which is 1 in the unmanned ship path tracking environment;

[0041] Among them, the discount factor is defined as γ, which is a value between 0 and 1, indicating the importance of future rewards; the larger the discount factor γ, the more attention is paid to the impact of long-term rewards on the control strategy.

[0042] Moreover, the specific steps of step 3 include:

[0043] 3.1 Initialize the ocean environment, initialize the network parameters of the actor strategy neural network and critic value neural network of the improved TD3 algorithm, and set the experience replay buffer.

[0044] 3.2 The unmanned ship agent interacts with the marine environment. The interaction process is the same as step 2. The state information of the unmanned ship is input into the actor strategy network. The output action of the actor strategy network is calculated through forward propagation. After the unmanned ship executes the action, the state is transferred and the reward is calculated. During this interaction process, the experience data is collected and stored in the experience replay buffer. Repeat this step until the experience replay buffer is full.

[0045] 3.3 The Critic value network uses the Q-value function to evaluate the performance of the actor policy network and guide its update in the next stage; the actor policy network determines to take better actions by changing the strategy π to obtain higher rewards;

[0046] 3.4 Use two state value networks to solve the problem of overestimation of Q value function. The next state s t+1 The Q value can be estimated by two target value networks:

[0047]

[0048] Where Q′1 and Q′2 are the target Q values ​​of the two value networks, π′ is the strategy of the target strategy network, and θ μ′ are the parameters of the target policy network, and are the parameters of the two target value networks respectively;

[0049] Select a smaller target Q value. According to the Bellman equation, the target value of the value network is calculated as:

[0050]

[0051] 3.5 According to the priority experience sampling mechanism, N experience samples are sampled from the experience replay buffer to update the network, the state set S and action set A are input into the value neural network, and the parameters of the neural network are updated through back propagation; the loss function of the value network is constructed by the following formula to guide the learning and optimization of the strategy:

[0052]

[0053] 3.6 Update the parameters of the critic value network by minimizing the loss function and gradient descent:

[0054]

[0055] In the formula is the gradient of the loss function, is the gradient of the Q value, and α1 is the learning rate of the value network;

[0056] 3.7 Use the policy delay update mechanism to improve the stability of the policy network update. The policy network is updated at a lower frequency than the value network. Through gradient ascent, the policy network updates the parameter θ in the direction of increasing the Q value. μ :

[0057]

[0058] In the formula, is the objective function J(θ μ ), is the strategy π(s t |θ μ ), α2 is the learning rate of the policy network;

[0059] 3.8 Use soft update mechanism to adjust the parameters of target network and θ μ′ Update the current network so that it gradually updates the target network at a smaller update rate, which is expressed by the following formula:

[0060]

[0061] θ μ′ ←εθ μ +(1-ε)θ μ′

[0062] Where, ε is the soft update rate;

[0063] 3.9 Repeat steps 3.2-3.8 for several training times to finally obtain a policy network with excellent control strategy.

[0064] Moreover, the specific steps of step 4 include:

[0065] 4.1 Initialize the ocean environment and load the trained strategy neural network parameters;

[0066] 4.2 Input the initial environment state of the unmanned ship into the policy network trained in step 3, and obtain the speed and heading control actions of the unmanned ship through forward propagation reasoning;

[0067] 4.3 The unmanned vessel performs output actions in the marine environment, tracks the reference path, and moves to a new state;

[0068] 4.4 Input the newly acquired state information into the policy network to generate new control actions for the unmanned ship;

[0069] 4.5 Repeat steps 4.3 and 4.4 until the unmanned vessel completes the path tracking task for the reference path.

[0070] Advantages and beneficial effects of the present invention:

[0071] 1. In view of the problems of low control accuracy and poor adaptability to dynamic environments of traditional control methods, the present invention proposes an unmanned ship path tracking control algorithm based on deep reinforcement learning, models the Markov decision process, and trains and optimizes the control strategy based on the improved TD3 algorithm to improve the path tracking control accuracy of the unmanned ship in complex marine environments, thereby improving the response speed and system stability of the unmanned ship.

[0072] 2. In order to solve the problem of poor authenticity of environmental models, the present invention introduces real marine environmental data and constructs a time-varying marine environmental model integrating multiple environmental elements, thereby improving its authenticity and reliability. In addition, the support of real data makes up for the shortcomings of existing work and improves its practical application value.

[0073] 3. When modeling the Markov decision process, the reward function proposed by the present invention comprehensively considers the tracking error, control cycle and task reward of the unmanned ship, effectively alleviating the problem of sparse rewards in complex marine environments, and helping the unmanned ship to quickly learn an effective and excellent control strategy.

[0074] 4. The present invention is easy to deploy on embedded terminal devices. After the trained control strategy is deployed on the terminal device, fast reasoning can be achieved to meet the needs of real-time decision-making, and it has great application value in real ship control. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 is a schematic diagram of the path tracking control task of the present invention;

[0076] Figure 2 It is a visual schematic diagram of the marine environment of the present invention;

[0077] Figure 3 is a Markov decision process diagram of the present invention;

[0078] Figure 4 It is a schematic diagram of the deep reinforcement learning unmanned ship path tracking control optimization algorithm based on the TD3 algorithm of the present invention;

[0079] Figure 5 It is a flow chart of the solution process of an unmanned ship path tracking control method based on deep reinforcement learning in the present invention. DETAILED DESCRIPTION

[0080] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0081] A path tracking control method for an unmanned ship based on deep reinforcement learning comprises the following steps:

[0082] Step 1: Construct a time-varying marine environment model integrating multiple environmental elements;

[0083] The specific steps of step 1 include:

[0084] 1.3 Obtain ocean current and wind field environmental data of part of the South China Sea (113.5°E-115.5°E, 17.25°N to 19.25°N) from the National Marine Science Data Center;

[0085] 1.4 The area is divided into 120×120 grids using the grid method, and the ocean current and wind field environment data are interpolated into the grid environment using the inverse distance weighted interpolation method;

[0086] 1.3 When the USV is at position P, the environmental data at a certain time t is expressed as:

[0087]

[0088] Where I P (t) is the environmental information received by the USV in real time at position P and time t through sensors such as velocity profiler and anemometer, and represent the lateral and longitudinal velocities of the ocean current, respectively. and They represent the lateral and longitudinal speeds of the wind respectively.

[0089] The environmental data I(t) of the non-grid point location P can be obtained from the surrounding four adjacent grid points P i Environmental data I at (i=1,...,4) i (t) is obtained by distance weighting and is calculated by the following formula:

[0090]

[0091] Step 2: Based on the ocean environment model in step 1, the unmanned ship path tracking control problem is modeled as a solution framework based on the Markov decision process;

[0092] The specific steps of step 2 include:

[0093] The unmanned ship path tracking control problem is modeled as a solution framework based on the Markov decision process, which specifically includes the state space S, action space A, reward function R, state transition relationship P, and discount factor γ of the Markov decision process:

[0094] Among them, the state space S represents the set of state information s perceived by the unmanned ship, including motion parameter information s m , environmental information e , LOS guidance information LOS , the state s at the tth control moment t It is expressed as:

[0095] s t ={s m ,s e ,s LOS}

[0096] Where: motion parameter information s m ={(x, y), U, ψ}, where (x, y) is the current position of the unmanned ship, U and ψ are the current speed and heading angle of the unmanned ship respectively; as described in step 1, the environmental data, environmental information s e =I P (t);LOS guidance information s LOS =(ψ d ,x e ,y e), where ψ d is the ideal heading angle of the unmanned ship, x e and e They are longitudinal tracking error and lateral tracking error respectively. The LOS guidance method is used to calculate the expected heading angle and tracking error to improve the tracking accuracy. The expected heading angle can be expressed as:

[0097]

[0098] In the formula, ψ k is the slope of the tracking path, Δ is the forward distance of the unmanned ship, and β is the sideslip angle;

[0099] For the unmanned ship at position (x, y), the parameterized path P is tracked k (t), the tracking error can be expressed as:

[0100]

[0101] Among them, the action space A represents the set of actions a of the unmanned ship, and the action a at the tth control moment is t It is defined as the output speed and heading angle of the unmanned ship. The heading angle is controlled by controlling the direction of the rudder, and the speed of the unmanned ship is adjusted by controlling the thrust of the propeller, as shown in the following formula:

[0102]

[0103] Where U(t) represents the speed of the unmanned ship, Indicates the heading angle of the unmanned ship;

[0104] Among them, the reward function R defines the unmanned ship in different states s t and action a t The reward signal r obtained under t , which is used to guide the learning and decision-making process of the control strategy, is defined as follows:

[0105]

[0106] Where K1 and K2 are the proportional parameters assigned to different error penalty terms, t step is the total control time; in addition, when the unmanned ship successfully completes the tracking task, it will receive a huge reward incentive. However, if the unmanned ship fails to track within the specified number of steps or deviates from the tracking area, it will receive a huge penalty.

[0107] Among them, the state transition relationship is defined as the probability of updating from the current environmental state to the next state after the unmanned ship makes an action decision, as follows:

[0108] P(S t+1 |S t ×[A]t )=ρ

[0109] Where ρ is the environmental transfer rate, which is 1 in the unmanned ship path tracking environment;

[0110] Among them, the discount factor is defined as γ, which is a value between 0 and 1, indicating the importance of future rewards; the larger the discount factor γ, the more attention is paid to the impact of long-term rewards on the control strategy.

[0111] Step 3: Based on the solution framework of step 2, the improved TD3 algorithm is used to optimize the path tracking control of the unmanned ship;

[0112] The specific steps of step 3 include:

[0113] 3.1 Initialize the ocean environment, initialize the network parameters of the actor strategy neural network and critic value neural network of the improved TD3 algorithm, and set the experience replay buffer.

[0114] 3.2 The unmanned ship agent interacts with the marine environment. The interaction process is the same as step 2. The state information of the unmanned ship is input into the actor strategy network. The output action of the actor strategy network is calculated through forward propagation. After the unmanned ship executes the action, the state is transferred and the reward is calculated. During this interaction process, the experience data is collected and stored in the experience replay buffer. Repeat this step until the experience replay buffer is full.

[0115] 3.3 The Critic value network uses the Q-value function to evaluate the performance of the actor policy network and guide its next stage update; the actor policy network determines to take better actions by changing the strategy π to obtain higher rewards;

[0116] 3.4 Use two state value networks to solve the problem of overestimation of Q value function. The next state s t+1 The Q value can be estimated by two target value networks:

[0117]

[0118] Where Q′1 and Q′2 are the target Q values ​​of the two value networks, π′ is the strategy of the target strategy network, and θ μ′ are the parameters of the target policy network, and are the parameters of the two target value networks respectively;

[0119] Select a smaller target Q value. According to the Bellman equation, the target value of the value network is calculated as:

[0120]

[0121] 3.5 According to the priority experience sampling mechanism, N experience samples are sampled from the experience replay buffer to update the network, the state set S and action set A are input into the value neural network, and the parameters of the neural network are updated through back propagation; the loss function of the value network is constructed by the following formula to guide the learning and optimization of the strategy:

[0122]

[0123] 3.6 Update the parameters of the critic value network by minimizing the loss function and gradient descent:

[0124]

[0125] In the formula is the gradient of the loss function, is the gradient of the Q value, and α1 is the learning rate of the value network;

[0126] 3.7 Use the policy delay update mechanism to improve the stability of the policy network update. The policy network is updated at a lower frequency than the value network. Through gradient ascent, the policy network updates the parameter θ in the direction of increasing the Q value. μ :

[0127]

[0128] In the formula, is the objective function J(θ μ ), is the strategy π(s t |θ μ ), α2 is the learning rate of the policy network;

[0129] 3.8 Use soft update mechanism to adjust the parameters of target network and θ μ′ Update the current network so that it gradually updates the target network at a smaller update rate, which is expressed by the following formula:

[0130]

[0131] θ μ′ ←εθ μ +(1-ε)θ μ′

[0132] Where ε is the soft update rate;

[0133] 3.9 Repeat steps 3.2-3.8 for several training times to finally obtain a policy network with excellent control strategy.

[0134] Step 4: Use the strategy neural network trained in step 3 to control the unmanned boat to perform the path tracking task. The specific steps of step 4 include:

[0135] 4.1 Initialize the ocean environment and load the trained strategy neural network parameters;

[0136] 4.2 Input the initial environment state of the unmanned ship into the policy network trained in step 3, and obtain the speed and heading control actions of the unmanned ship through forward propagation reasoning;

[0137] 4.3 The unmanned vessel performs output actions in the marine environment, tracks the reference path, and moves to a new state;

[0138] 4.4 Input the newly acquired state information into the policy network to generate new control actions for the unmanned ship;

[0139] 4.5 Repeat steps 4.3 and 4.4 until the unmanned vessel completes the path tracking task for the reference path.

[0140] In this embodiment, the unmanned ship path tracking is as follows Figure 1 As shown, the unmanned ship is in an ocean environment with ocean currents and winds. Track the reference path with a small error and reach the target point. The time-varying ocean environment model we constructed by integrating multiple environmental factors is as follows: Figure 2 As shown, arrows indicate the direction of ocean currents and wind fields, the background color is the intensity of the ocean current and wind field environment, and the environmental factors at different locations and times change dynamically.

[0141] In addition, the deep reinforcement learning process diagram is as follows Figure 3 As shown in the figure, the environment is the ocean environment where the unmanned ship moves. The intelligent agent is an unmanned ship, and the interaction between the intelligent agent and the environment is realized through the Markov decision process. Righteous Figure 3 The states, actions, and rewards in , by interacting with the environment and training to maximize the reward.

[0142] Figure 4 The figure shows the model optimization solution process, that is, the flowchart of the improved TD3 algorithm training process. The agent interacts with the ocean environment through the Markov decision process, and stores the collected experience samples in the experience replay buffer; in the network parameter update phase, small batch samples are sampled from the experience replay buffer for network update, the actor policy network learns excellent control strategies, and the critic value network guides the update of the strategy. The learning process of the strategy is accelerated and stabilized through mechanisms such as delayed policy update and soft update.

[0143] Figure 5 Shown is a step flow chart of the unmanned ship path tracking control method based on deep reinforcement learning of the present invention.

[0144] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A path tracking control method for an unmanned ship based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Construct a time-varying marine environment model integrating multiple environmental elements; Step 2: Based on the ocean environment model in step 1, the unmanned ship path tracking control problem is modeled as a solution framework based on the Markov decision process; Step 3: Based on the solution framework of step 2, the improved Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is used to optimize the path tracking control of the unmanned ship; Step 4: Use the strategy neural network trained in step 3 to control the unmanned boat to perform the path tracking task.

2. The unmanned ship path tracking control method based on deep reinforcement learning according to claim 1 is characterized in that: The step 1 constructs a time-varying ocean environment model integrating multiple environmental elements, including: obtaining ocean current and wind field environmental data of a part of the South China Sea (113.5°E-115.5°E, 17.25°N to 19.25°N) from the National Marine Science Data Center; dividing the area into 120×120 grids using a grid method, and interpolating the ocean current and wind field environmental data into the grid environment using an inverse distance weighted interpolation method; setting a time window for environmental data changes to describe the dynamic changes of the ocean environment; When the USV is at position P, the environmental data at a certain time t is expressed as: Where I P (t) is the environmental information received by the USV in real time at position P and time t through sensors such as velocity profiler and anemometer, and represent the lateral and longitudinal velocities of the ocean current, respectively. and They represent the lateral and longitudinal speeds of the wind respectively. The environmental data I(t) of the non-grid point location P can be obtained from the surrounding four adjacent grid points P i Environmental data I at (i=1,...,4) i (t) is obtained by distance weighting and is calculated by the following formula:

3. The unmanned ship path tracking control method based on deep reinforcement learning according to claim 1 is characterized in that: The step 2 models the unmanned ship path tracking control problem as a solution framework based on the Markov decision process, which specifically includes the state space S, action space A, reward function R, state transition relationship P, and discount factor γ of the Markov decision process: Among them, the state space S represents the set of state information s perceived by the unmanned ship, including motion parameter information s m , environmental information e , LOS guidance information LOS , the state s at the tth control moment t It is expressed as: s t ={s m ,s e ,s LOS } Where: motion parameter information s m ={(x, y), U, ψ}, where (x, y) is the current position of the unmanned ship, U and ψ are the current speed and heading angle of the unmanned ship respectively; as described in step 1, the environmental data, environmental information s e =I P (t);LOS guidance information s LOS =(ψ d ,x e ,y e ), where ψ d is the ideal heading angle of the unmanned ship, x e and e They are longitudinal tracking error and lateral tracking error respectively. The LOS guidance method is used to calculate the expected heading angle and tracking error to improve the tracking accuracy. The expected heading angle can be expressed as: In the formula, ψ k is the slope of the tracking path, Δ is the forward distance of the unmanned ship, and β is the sideslip angle; For the unmanned ship at position (x, y), the parameterized path P is tracked k (t), the tracking error can be expressed as: Among them, the action space A represents the set of actions a of the unmanned ship, and the action a at the tth control moment is t It is defined as the output speed and heading angle of the unmanned ship. The heading angle is controlled by controlling the direction of the rudder, and the speed of the unmanned ship is adjusted by controlling the thrust of the propeller, as shown in the following formula: Where U(t) represents the speed of the unmanned ship, Indicates the heading angle of the unmanned ship; Among them, the reward function R defines the unmanned ship in different states s t and action a t The reward signal r obtained under t , which is used to guide the learning and decision-making process of the control strategy, is defined as follows: Where K1 and K2 are the proportional parameters assigned to different error penalty terms, t step is the total control time; in addition, when the unmanned ship successfully completes the tracking task, it will receive a huge reward incentive. However, if the unmanned ship fails to track within the specified number of steps or deviates from the tracking area, it will receive a huge penalty. Among them, the state transition relationship is defined as the probability of updating from the current environmental state to the next state after the unmanned ship makes an action decision, as follows: P(S t+1 |S t ×[A] t )=ρ Where ρ is the environmental transfer rate, which is 1 in the unmanned ship path tracking environment; Among them, the discount factor is defined as γ, which is a value between 0 and 1, indicating the importance of future rewards; the larger the discount factor γ, the more attention is paid to the impact of long-term rewards on the control strategy.

4. According to the unmanned ship path tracking control method based on deep reinforcement learning in claim 1, it is characterized in that the step 3 uses the improved TD3 algorithm to optimize and solve the unmanned ship path tracking control, and the specific steps include: 4.1 Initialize the ocean environment, initialize the network parameters of the actor strategy neural network and critic value neural network of the improved TD3 algorithm, and set the experience replay buffer. 4.2 The unmanned ship agent interacts with the marine environment. The interaction process is the same as step 2. The state information of the unmanned ship is input into the actor strategy network. The output action of the actor strategy network is calculated through forward propagation. After the unmanned ship executes the action, the state is transferred and the reward is calculated. During this interaction process, the experience data is collected and stored in the experience replay buffer. Repeat this step until the experience replay buffer is full. 4.3 The Critic value network uses the Q-value function to evaluate the performance of the actor policy network and guide its next stage update; the actor policy network determines to take better actions by changing the strategy π to obtain higher rewards; 4.4 Use two state value networks to solve the problem of overestimation of Q value function. The next state s t+1 The Q value can be estimated by two target value networks: Where Q′1 and Q′2 are the target Q values ​​of the two value networks, π′ is the strategy of the target strategy network, and θ μ′ are the parameters of the target policy network, and are the parameters of the two target value networks respectively; Select a smaller target Q value. According to the Bellman equation, the target value of the value network is calculated as: 4.5 According to the priority experience sampling mechanism, N experience samples are sampled from the experience replay buffer to update the network, the state set S and action set A are input into the value neural network, and the parameters of the neural network are updated through back propagation; the loss function of the value network is constructed by the following formula to guide the learning and optimization of the strategy: 4.6 Update the parameters of the critic value network by minimizing the loss function and gradient descent: In the formula is the gradient of the loss function, is the gradient of the Q value, and α1 is the learning rate of the value network; 4.7 Use the policy delay update mechanism to improve the stability of the policy network update. The policy network is updated at a lower frequency than the value network. Through gradient ascent, the policy network updates the parameter θ in the direction of increasing the Q value. μ : In the formula, is the objective function J(θ μ ), is the strategy π(s t |θ μ ), α2 is the learning rate of the policy network; 4.8 Use soft update mechanism to adjust the parameters of target network and θ μ′ Update the current network so that it gradually updates the target network at a smaller update rate, which is expressed by the following formula: i μ′ ←th μ +(1−e)θ μ′ Where ε is the soft update rate; 4.9 Repeat steps 4.2-4.8 for several training times to finally obtain a policy network with excellent control strategy.

5. The unmanned ship path tracking control method based on deep reinforcement learning according to claim 1 is characterized in that: The step 4 uses the trained strategy neural network to control the unmanned ship to perform the path tracking task, and the specific steps include: 5.1 Initialize the ocean environment and load the trained strategy neural network parameters; 5.2 Input the initial environment state of the unmanned ship into the policy network trained in step 3, and obtain the speed and heading control actions of the unmanned ship through forward propagation reasoning; 5.3 The unmanned vessel performs output actions in the marine environment, tracks the reference path, and moves to a new state; 5.4 Input the newly acquired state information into the policy network to generate new control actions for the unmanned ship; 5.5 Repeat steps 5.3 and 5.4 until the unmanned vessel completes the path tracking task for the reference path.

Images