Unmanned ship target tracking decision control method based on deep reinforcement learning

Abstract

The invention provides an unmanned ship target tracking decision control method based on deep reinforcement learning. The method comprises the following steps: establishing an unmanned ship motion model to obtain a current motion state of an unmanned ship; establishing a target tracking task reward function; constructing a deep reinforcement learning network model based on a soft actor-commentator algorithm, and performing initialization; acquiring a current position and a predicted position of the target ship; obtaining an optimal control strategy of the unmanned ship and an updated motion state of the unmanned ship by using a deep reinforcement learning network model; randomly selecting a control strategy to obtain an updated motion state of the target ship, storing the updated motion state of the unmanned ship and the updated motion state of the target ship as training samples in an experience pool, and repeating the above process when the target ship is not tracked; and when the target ship is tracked, randomly sampling from the experience pool to update the parameters of the deep reinforcement learning network model. The intelligent decision-making capability and tracking performance of the unmanned ship are improved, and the application development of the unmanned ship in the civil and military fields is promoted.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned boat control, and in particular relates to an unmanned boat target tracking decision control method based on deep reinforcement learning. Background Art

[0002] Unmanned boats are highly automated and intelligent marine operation platforms, widely used in environmental monitoring, maritime patrol, rescue operations and other fields. With the advancement of unmanned driving technology, the autonomous operation capabilities of unmanned boats in complex marine environments are gradually enhanced. However, target tracking, as the core function of autonomous control of unmanned boats, faces many challenges, including environmental complexity, real-time requirements and inherent system delays.

[0003] The commonly used method is to obtain the navigation state information of the target through the perception system or collaborative intelligent system, establish the state transfer equation and observation equation, use the extended Kalman filter algorithm to predict the target state, and calculate the expected heading in combination with the pure tracking method. According to the distance between the unmanned boat and the target, the expected speed is calculated through the PID controller, and finally the unmanned boat can track the target stably at the expected heading and speed.

[0004] There are two main problems with traditional decision-making planning methods that limit the efficiency of unmanned vehicles in target tracking tasks.

[0005] First, these methods usually ignore the maneuverability of the unmanned boat. The underactuated characteristics, time delays, and complex motion characteristics of the unmanned boat are often not fully considered, resulting in deviations between decision planning and actual control execution. In other words, the planned optimal path or control strategy is often difficult to implement in actual operation, making it impossible for the unmanned boat to adjust its course or speed as planned, affecting the accuracy and efficiency of tracking.

[0006] Secondly, existing path planning methods rarely consider the uncertainty of target motion. The position of the target changes at any time, and traditional methods rely on the current state for geometric path planning, ignoring the possible future motion trajectory of the target. This method causes the decision result to always lag behind the actual movement of the target. When the unmanned boat adjusts its course according to the planned path, it often cannot keep up with the changes in the target, resulting in slow response and poor tracking effect. Especially in high-speed or complex environments, this lag will significantly reduce the tracking ability of the unmanned boat, making it difficult to quickly approach the moving target. Summary of the invention

[0007] In response to the problems existing in the prior art, the present invention provides an unmanned boat target tracking decision-making control method based on deep reinforcement learning, which improves the intelligent decision-making ability and tracking performance of the unmanned boat and promotes its application development in civil and military fields.

[0008] The technical solution of the present invention is as follows:

[0009] The unmanned boat target tracking decision control method based on deep reinforcement learning is characterized by comprising the following steps:

[0010] Establish the unmanned boat motion model to obtain the current motion state of the unmanned boat;

[0011] Establish a target tracking task reward function to guide the unmanned boat to quickly approach the target;

[0012] Build a deep reinforcement learning network model based on the soft actor-critic algorithm and initialize it;

[0013] Obtain the current position of the target ship, obtain the historical trajectory of the target ship and fit the target ship motion trajectory to obtain the predicted position of the target ship; obtain the current environmental state; use the current environmental state, the current position and predicted position of the target ship, and the current motion state of the unmanned boat as input, and use the deep reinforcement learning network model to obtain the optimal control strategy of the unmanned boat and the updated motion state of the unmanned boat; randomly select the target ship control strategy to obtain the updated motion state of the target ship, store the updated motion state of the unmanned boat and the updated motion state of the target ship as training samples in the experience pool, and repeat the above process when the unmanned boat fails to catch up with the target ship;

[0014] When the unmanned boat catches up with the target ship, random sampling is performed from the experience pool to update the parameters of the deep reinforcement learning network model.

[0015] Preferably, in the unmanned boat target tracking decision control method based on deep reinforcement learning, in order to describe the target tracking motion control of the unmanned boat, the following is given:

[0016] A continuous, bounded, non-empty state space Each of these states Including the position coordinates x and y of the 3-DOF unmanned boat model in the fixed coordinate system, the heading angle φ, and the longitudinal velocity u, lateral velocity v and bow angular velocity r in the body coordinate system;

[0017] A continuous, bounded and state-dependent control action space Each control action u=[n,δ] T It is composed of propeller speed n and rudder angle δ;

[0018] Current status of unmanned boats and control input u usv =[n usv ,δ usv ] T ;

[0019] 3-DOF model of unmanned boat u usv It will generate thrust and torque;

[0020] Tracking the current state of the target and control input u tar =[n tar ,δ tar ] T and the target motion model

[0021] Preferably, in the unmanned boat target tracking decision control method based on deep reinforcement learning,

[0022] During the tracking process, the positions of the unmanned boat and the target ship are expressed as and The relative position relationship between the target ship and the unmanned boat at time t is defined as follows:

[0023]

[0024] The position information is represented by the Euclidean distance d∈<0,∞> between the target and the unmanned boat and the azimuth θ∈<-π,π> relative to the heading of the unmanned boat, and its expression is as follows:

[0025]

[0026] θ=arctan 2(Δy,Δx) (3)

[0027] The goal of the mission is to control the unmanned boat to approach the target ship and reach the predetermined distance threshold d. threshold .

[0028] Preferably, in the unmanned boat target tracking decision control method based on deep reinforcement learning,

[0029] The reward function based on the task objective is defined as follows:

[0030] r t =r dis +r time = k1(d t -d t+1 )-k2(5)

[0031] Where rt represents the feedback reward obtained by the agent from the environment after performing action at at time t; rdis represents the distance reward, which guides the unmanned boat to approach the target; rtime represents the time reward, which makes the agent aware of the time consumption in the tracking process; k1 and k2 are hyperparameters used to determine the weights of distance reward and time reward;

[0032] The goal of the DRL algorithm is to maximize the reward R, which is calculated as follows:

[0033]

[0034] Where T represents the total time spent in the tracking process; the final cumulative distance reward is determined by the initial distance d between the unmanned boat and the target. 0 and the final distance d T+1 And the time T taken is determined.

[0035] Preferably, in the unmanned boat target tracking decision control method based on deep reinforcement learning,

[0036] The optimal strategy π in SAC * It is expressed as:

[0037]

[0038] Among them, the reward function R(s t ,a t ) reflects the state s t Next, perform action a t The expected reward E can be expressed as:

[0039] R(s t ,a t )=E[R t |s t =s,a t =a] (8)

[0040] H is entropy, calculated as:

[0041]

[0042] The size of the entropy value reflects the strategy π(·|s t ) is the random degree of the reward. When the entropy is large, the unmanned boat will randomly select an action with similar value to execute. α is the temperature coefficient, which determines the proportion of entropy value to reward. Adjusting α can control the exploratory nature of the optimal strategy.

[0043] Soft state value function V soft and the soft action value function Q soft Respectively expressed as:

[0044]

[0045] γ t represents the discount factor used to evaluate the importance of future rewards, r(a t ,s t ) represents the reward obtained by executing the action in the state, ρ πrepresents the probability of the optimal strategy;

[0046] The Bellman equation form of the soft Q function is expressed as:

[0047]

[0048] s' represents the state at the next moment, and a' represents the action performed in the state at the next moment;

[0049] The update method of soft Q function and soft V function is:

[0050]

[0051] By continuously iterating equations (13) to (14), the output of the Q function will gradually converge to the optimal Q value. During the training process, the strategy π is updated in the exponential direction of the soft Q function:

[0052]

[0053] During the training process, two Q networks with the same structure are introduced for simultaneous training, and a smaller Q value is selected to update the strategy network in each iteration;

[0054] Set the experience pool D to store samples (s, a, s′, r) obtained by interacting with the environment, and randomly sample from it during training to increase sample independence and improve training efficiency. Set the policy network parameter to φ, the Q network parameter to ω, and the loss functions to:

[0055]

[0056] in, The calculation formula is:

[0057]

[0058] The parameters of the target Q network are The parameters are soft updated in each iteration: where the τ value is greater than 0 but much less than 1.

[0059] The present invention has the following beneficial effects:

[0060] Through the end-to-end decision-making and control approach, the inconsistency between the decision-making process and the control execution is reduced, thereby improving the efficiency and accuracy of tracking; at the same time, the real maneuverability model of the unmanned boat is modeled using a data-driven method, providing a learning and training basis for the training of the reinforcement learning algorithm. In this process, the uncertainty of the future behavior of the tracking target, the motion characteristics of the unmanned boat itself, and the environmental interference are considered in the environment in order to learn a more efficient tracking control strategy; this comprehensive consideration not only improves the adaptability of the system, but also enhances the autonomous decision-making ability of the unmanned boat in a complex environment.

[0061] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A variable diagram in an embodiment of the unmanned boat target tracking decision control method based on deep reinforcement learning provided by the present invention;

[0063] Figure 2 A schematic diagram of optimization targets in one embodiment of the unmanned boat target tracking decision control method based on deep reinforcement learning provided by the present invention;

[0064] Figure 3 An overall framework diagram of an embodiment of the unmanned boat target tracking decision control method based on deep reinforcement learning provided by the present invention;

[0065] Figure 4 A flowchart of an embodiment of the unmanned boat target tracking decision control method based on deep reinforcement learning provided by the present invention;

[0066] Figure 5 This is a simulation result diagram of an embodiment of the unmanned boat target tracking decision-making control method based on deep reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0068] Reinforcement learning is an algorithm that learns the best decision-making strategy by interacting with the environment. The unmanned boat adjusts its behavior based on the feedback from the environment to maximize the cumulative reward. The core of reinforcement learning is to set a reward mechanism so that the agent can continuously try and optimize its decision-making and eventually learn how to make the best choice in different environments.

[0069] Aiming at the target tracking problem of unmanned boats, this paper proposes an end-to-end control method based on reinforcement learning. Under this framework, a reward function is designed for the unmanned boat to encourage it to approach the moving target on the water surface quickly and accurately. Through continuous interaction with the environment, the unmanned boat not only needs to learn to track the target, but also needs to deal with the unpredictability of the target behavior, environmental interference such as wind and waves at sea, and the control limitations of the unmanned boat itself (such as time delay and underactuation). The reward mechanism of reinforcement learning provides feedback based on the distance between the unmanned boat and the target: if it is close to the target, the reward value increases; if it is far away from the target, the reward value decreases. This design guides the unmanned boat to gradually adjust the control strategy to achieve the goal of optimizing the tracking path. In addition, the end-to-end method can use neural networks to directly process the position information of the target and the motion state of the unmanned boat, and directly output control signals. This reduces complex planning steps, enables the unmanned boat to respond and adjust faster in actual operation, and significantly improves the tracking efficiency and response speed. At the same time, as the training progresses, the algorithm gradually adapts to the diversity of the environment and targets, enhancing the adaptability and robustness of the unmanned boat in a dynamic marine environment.

[0070] The present invention proposes an end-to-end control method based on reinforcement learning.

[0071] First, the historical motion trajectory points of the target ship are used to fit based on the cubic spline curve. Then the future position of the target is extrapolated using the trajectory, and the prediction results are integrated into the state observation. Therefore, the system not only relies on the current position of the target, but also considers the future motion trend of the target, which helps to reduce the uncertainty caused by the target motion. Combined with the current motion state of the unmanned boat, the deep learning model is used to directly infer the optimal control signal. According to the designed reward function, the unmanned boat can be guided to explore the learning control strategy without prior knowledge. This method enables the unmanned boat to adjust the heading and speed in real time in a dynamic and uncertain environment to approach the target faster and more accurately. Through this end-to-end control method, the difference between decision planning and control execution is significantly reduced. The unmanned boat can quickly respond to the motion changes of the target in a complex environment, overcome the influence of lag and uncertainty in traditional methods, and thus greatly improve the tracking efficiency and real-time performance.

[0072] like Figure 3 and Figure 4 As shown, the target tracking decision control method of the unmanned boat based on deep reinforcement learning of the present invention has the following specific steps:

[0073] The first step is to model the maneuverability of the unmanned boat through a data-driven approach for the nonlinear motion system of the unmanned boat, and establish an accurate unmanned boat motion model. This model can better reflect the motion characteristics of the unmanned boat under different environments and operating conditions, and provide a basis for subsequent control and optimization.

[0074] To describe the target tracking motion control of the unmanned boat, we give:

[0075] 1) A continuous, bounded, non-empty state space Each of these states It includes the position coordinates x and y of the 3-DOF unmanned boat model in the fixed coordinate system, the heading angle φ, and the longitudinal velocity u, lateral velocity v and bow angular velocity r in the body coordinate system, as shown in Figure 1 As shown;

[0076] 2) A continuous, bounded and state-dependent control action space Each control action u=[n,δ] T It is composed of propeller speed n and rudder angle δ;

[0077] 3) Current status of unmanned boats and control input u usv =[n usv ,δ usv ] T ;

[0078] 4) 3-DOF model of unmanned boat u usv It will generate thrust and torque;

[0079] 5) Track the current status of the target and control input u tar =[n tar ,δ tar ] T and the target motion model

[0080] calculate:

[0081] A real-time feasible control input u usv =[n usv ,δ usv ] T , so that the unmanned boat can approach the target in the shortest possible time.

[0082] In the present invention, it is assumed that in a non-cooperative scenario, the control strategy and dynamic model of the target ship are not available; the movement of the target ship is not affected by the tracking action of the unmanned boat. The unmanned boat can only perceive the current position of the target ship, which makes the problem a partially observable Markov decision process (POMDP). In addition, the speed of the unmanned boat is higher than that of the target ship, which ensures the feasibility of the target tracking task.

[0083] During the tracking process, the positions of the unmanned boat and the target ship are expressed as and The relative position relationship between the target ship and the unmanned boat at time t is defined as follows:

[0084]

[0085] In general, the position of the target can be obtained by the onboard sensor of the unmanned boat (such as marine radar). Specifically, the position information is represented by the Euclidean distance d∈<0,∞> between the target and the unmanned boat and the azimuth angle θ∈<-π,π> relative to the heading of the unmanned boat, and its expression is as follows:

[0086]

[0087] θ=arctan 2(Δy,Δx) (3)

[0088] The target tracking task aims to track the target accurately and quickly, minimizing errors and delays in the process. The goal of the task is to control the unmanned boat to approach the target ship and reach a predetermined distance threshold d. threshold During the tracking process, the relative distance between the unmanned boat and the target changes as follows: Figure 2 As shown, where d initia l represents the initial distance between the target and the unmanned boat. T1 and T2 represent the time required to complete the tracking under two different control strategies. In the same scenario, the goal of the tracking task is to reach the predetermined distance d in a shorter time T2. threshold .

[0089] The second step is to design a reward function for the target tracking task to guide the agent to learn the optimal decision-making control strategy. At the same time, the target tracking task of the unmanned boat is constructed as a partially observable Markov decision process to provide support for the decision-making control model training of the agent.

[0090] Reinforcement learning algorithm for target tracking control problem:

[0091] This method uses the current position of the target, the predicted future position, and the state information of the unmanned boat as observation inputs, designs a deep reinforcement learning algorithm, and learns the optimal control strategy through continuous experiments and environmental feedback. This end-to-end reinforcement learning algorithm directly maps the observed state to the control signal and integrates the traditional planning and control modules. This integration avoids the potential limitations and time lags caused by designing these modules separately. This reinforcement learning process is similar to the way human drivers accumulate experience - by observing and predicting the movement of other vehicles, drivers actively adjust their behavior, thereby achieving more efficient target tracking and the ability to cope with complex environments.

[0092] Specifically, the observed state at time t is defined as:

[0093]

[0094] in and represents the predicted target ship position after N time steps. The prediction of target motion is achieved by fitting a cubic spline curve. This end-to-end solution enables the unmanned boat to generate control actions using its own motion information, the current position of the target, and the predicted target position.

[0095] The target tracking decision control strategy is based on the Soft Actor-Critic (SAC) algorithm, which is a model-free reinforcement learning framework based on the actor-critic architecture. The SAC algorithm improves the Deep Deterministic Policy Gradient (DDPG) by introducing the concept of maximizing entropy, enhancing the exploration ability and stability of the model. The framework of the proposed method is as follows Figure 3 The interactive environment includes: unmanned boats, target ships, and a disturbed marine environment.

[0096] In order to encourage the agent to explore itself and learn effective control strategies without prior knowledge, this paper designs a reward function based on task objectives, which is defined as follows:

[0097] r t =r dis +r time =k1(d t -d t+1 )-k2(5)

[0098] Where rt represents the feedback reward obtained by the agent from the environment after performing action at at time t. rdis represents the distance reward, which guides the unmanned boat to approach the target. rtime represents the time reward, which makes the agent aware of the time consumption during the tracking process. k1 and k2 are hyperparameters used to determine the weights of distance reward and time reward. The goal of the DRL algorithm is to maximize the reward R, which is calculated as follows:

[0099]

[0100] Where T represents the total time spent in the tracking process. The final cumulative distance reward is determined by the initial distance d between the unmanned boat and the target. 0 and the final distance d T+1 And the time T. In order to prevent the task from failing due to unnecessary exploration during training, a constant upper limit is set for T.

[0101] By analyzing the overall results of the tracking task, we can clearly understand the reward value. In the early stages of training, the agent does not yet have the ability to complete the task within the specified time. According to formula (6), the reward value depends on the initial relative distance d between the unmanned boat and the target. 0 And the relative distance d at the end state T+1 , where k2 is a constant. Once the agent has learned an effective control strategy, it can operate within the distance range set by the target navigator within the specified time. At this time, d T+1 Therefore, the reward value is determined by the initial distance d 0 and task duration T. This reward function helps to improve the control strategy from the perspective of tracking efficiency.

[0102] The reward function designed by the present invention is highly flexible because it does not require in-depth understanding of the specific details of the task. The reward signal is provided directly based on the agent behavior and the environment state, which reduces the rule constraints and minimizes the deviation caused by artificially defined rules. At the same time, it allows a wider exploration space, enhances the decision-making ability of the agent, and fully utilizes the advantages of the DRL algorithm.

[0103] The optimal policy π in SAC * It is expressed as:

[0104]

[0105] Among them, the reward function R(s t ,a t ) reflects the state s t Next, perform action a t The expected reward E can be expressed as:

[0106] R(s t ,a t )=E[R t |s t =s,a t =a] (8)

[0107] The specific definition of the reward function R in the target tracking process is defined in formula (6). H is the entropy, which is calculated as:

[0108]

[0109] The size of the entropy value reflects the strategy π(·|s t ) is the randomness of the action. When the entropy is large, the unmanned boat will randomly select an action with similar value to execute. α is the temperature coefficient, which determines the proportion of entropy value to reward. Adjusting α can control the exploratory nature of the optimal strategy.

[0110] In the process of target tracking, the state value function V and the action value function Q respectively evaluate the expected value of long-term rewards under specific environmental conditions and the possible long-term rewards after executing different control actions. In the SAC algorithm, maximum entropy is also one of the optimization goals of the strategy. Therefore, the soft state value function V soft and the soft action value function Q soft Respectively expressed as: γ t 、r(a t ,s t ),ρ π 、a'、s'Please define

[0111]

[0112]

[0113] γ t represents the discount factor used to evaluate the importance of future rewards, r(a t ,s t ) represents the reward obtained by executing the action in the state, ρ π represents the probability of the optimal strategy. The Bellman equation form of the soft Q function is expressed as:

[0114]

[0115] s' represents the state at the next moment, and a' represents the action performed under the state at the next moment. The update method of soft Q function and soft V function is:

[0116]

[0117]

[0118] By continuously iterating equations (13) to (14), the output of the Q function will gradually converge to the optimal Q value. During the training process, the strategy π is updated in the exponential direction of the soft Q function:

[0119]

[0120] During the training process, in order to prevent the action value from being overestimated, two Q networks with the same structure are introduced for simultaneous training. A smaller Q value is selected to update the policy network in each iteration. The experience pool D is set to store samples (s, a, s′, r) obtained by interacting with the environment. During training, random sampling is performed to increase sample independence and improve training efficiency. Let the policy network parameter be φ, the Q network parameter be ω, and the loss functions be:

[0121]

[0122]

[0123] in, The calculation formula is:

[0124]

[0125] The parameters of the target Q network are The parameters are soft updated in each iteration: where the τ value is greater than 0 but much less than 1.

[0126] The third step is to analyze the historical motion trajectory of the target ship and use a polynomial curve to fit it to predict the future position of the target. This method can infer the motion trend of the target based on the existing trajectory data, increase human prior knowledge, provide a basis for the tracking decision of the unmanned boat, and reduce the uncertainty of the task.

[0127] Future position prediction of target ship:

[0128] In the scenario of non-cooperative target tracking, the target ship will produce different motion trajectories under different control strategies, which makes it challenging to learn an effective tracking strategy based only on the target's real-time position information. To address this problem, we incorporate the predicted information of the target's motion into the observation. This allows the agent to consider both the current and future positions of the target during self-exploration.

[0129] set up Represents the position of the target ship observed at time step k, forming a trajectory The future position of the target after n time steps is expressed as Target location is the result of short-term trajectory prediction. The present invention uses a cubic polynomial curve to fit the target ship's motion trajectory based on the nearest trajectory point for prediction. For example, the mathematical representation of its prediction is:

[0130]

[0131] Where N = k + n represents the predicted n time steps in the future, and the unknown parameters are a0, a1, a2, a3. In order to determine these unknown parameters, the matrix representation is used:

[0132] X=V*A(21)

[0133] in Table unknown coefficients, represents the position of the target at the past moment, and V represents the coefficient matrix about the time step. The specific elements are as follows:

[0134]

[0135] The unknown coefficients are obtained by A = V -1 *X. Similarly, we can estimate The relative position of the target ship and the unmanned boat is calculated as follows:

[0136]

[0137] According to formulas (2) and (3), we can get and And put it into observation status.

[0138] In the fourth step, based on the current position of the target and the predicted future position, combined with the current motion state of the unmanned boat, the deep neural network is used to directly output the control action. This action is used to update the state of the unmanned boat. At the same time, the target ship adopts a random strategy to control and update the motion state to generate training samples, which are stored in the experience pool for subsequent training.

[0139] The fifth step is to randomly sample training samples from the experience pool and continuously update the parameters of the target execution network and the target evaluation network through soft updates. This process enables the neural network to gradually optimize the control strategy and improve the tracking efficiency of the unmanned boat and its ability to cope with complex environments.

[0140] Through interaction with the environment, the unmanned boat can learn an efficient control strategy to quickly approach the target ship. The trained model can effectively track ships with different control strategies. Figure 5 As shown in the figure, they correspond to the navigation trajectories of the target ship tracked by the unmanned boat from different directions with different control strategies. During the tracking process, the unmanned boat adjusts its own course to adapt to the position change of the target ship. Under the condition that the future behavior of the target cannot be obtained, the trained model can control the unmanned boat to track the target ship at different directions and quickly approach the target.

[0141] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to specific details.

Claims

1. An unmanned boat target tracking decision-making control method based on deep reinforcement learning, characterized in that: The following steps are involved: Establish the unmanned boat motion model to obtain the current motion state of the unmanned boat; Establish a target tracking task reward function to guide the unmanned boat to quickly approach the target; Build a deep reinforcement learning network model based on the soft actor-critic algorithm and initialize it; Acquire the current position of the target ship, acquire the historical trajectory of the target ship and fit the target ship motion trajectory to obtain the predicted position of the target ship; acquire the current environmental state; Taking the current environmental state, the current position and predicted position of the target ship, and the current motion state of the unmanned boat as input, the deep reinforcement learning network model is used to obtain the optimal control strategy of the unmanned boat and the updated motion state of the unmanned boat; Randomly select a target ship control strategy to obtain an updated target ship motion state, store the updated unmanned boat motion state and the updated target ship motion state in an experience pool as training samples, and repeat the above process when the unmanned boat fails to catch up with the target ship; When the unmanned boat catches up with the target ship, random sampling is performed from the experience pool to update the parameters of the deep reinforcement learning network model.

2. The unmanned boat target tracking decision control method based on deep reinforcement learning as claimed in claim 1, characterized in that: To describe the target tracking motion control of the unmanned boat, we give: A continuous, bounded, non-empty state space Each of these states Including the position coordinates x and y of the 3-DOF unmanned boat model in the fixed coordinate system, the heading angle φ, and the longitudinal velocity u, lateral velocity v and bow angular velocity r in the body coordinate system; A continuous, bounded and state-dependent control action space Each control action u=[n,δ] T It is composed of propeller speed n and rudder angle δ; Current status of unmanned boats and control input u usv =[n usv ,δ usv ] T ; 3-DOF model of unmanned boat u usv It will generate thrust and torque; Tracking the current state of the target and control input u tar =[n tar ,δ tar ] T and the target motion model 3. The unmanned boat target tracking decision control method based on deep reinforcement learning as claimed in claim 2, characterized in that: During the tracking process, the positions of the unmanned boat and the target ship are expressed as and The relative position relationship between the target ship and the unmanned boat at time t is defined as follows: The position information is composed of the Euclidean distance d∈<0,∞> between the target and the unmanned boat and the azimuth relative to the heading of the unmanned boat. θ∈<-π,π>, and its expression is as follows: θ=arctan2(Δy,Δx) (3) The goal of the mission is to control the unmanned boat to approach the target ship and reach the predetermined distance threshold d. threshold .

4. The unmanned boat target tracking decision control method based on deep reinforcement learning as claimed in claim 3, characterized in that: The reward function based on the task objective is defined as follows: r t =r dis +r time =k1(d t -d t+1 )-k2(5) where r t Indicates that the agent performs action a at time t t The feedback reward obtained from the environment; r dis Indicates distance reward, guiding the unmanned boat to approach the target; r time represents the time reward, which makes the agent aware of the time consumption in the tracking process; k1 and k2 are hyperparameters used to determine the weights of distance reward and time reward; The goal of the DRL algorithm is to maximize the reward R, which is calculated as follows: Where T represents the total time spent in the tracking process; the final cumulative distance reward is determined by the initial distance d between the unmanned boat and the target. 0 and the final distance d T+1 And the time T taken is determined.

5. The unmanned boat target tracking decision control method based on deep reinforcement learning as claimed in claim 4, characterized in that: Optimal strategy π * It is expressed as: Among them, the reward function R(s t ,a t ) reflects the state s t Next, perform action a t The expected reward E can be expressed as: R(s t ,a t )=E[R t |s t =s,a t =a](8) H is entropy, calculated as: The size of the entropy value reflects the strategy π(·|s t ) is the random degree of the reward. When the entropy is large, the unmanned boat will randomly select an action with similar value to execute. α is the temperature coefficient, which determines the proportion of entropy value to reward. Adjusting α can control the exploratory nature of the optimal strategy. Soft state value function V soft and the soft action value function Q soft Respectively expressed as: γ t represents the discount factor used to evaluate the importance of future rewards, r(a t ,s t ) represents the reward obtained by executing the action in the state, ρ π represents the probability of the optimal strategy; The Bellman equation form of the soft Q function is expressed as: s' represents the state at the next moment, and a' represents the action performed in the state at the next moment; The update method of soft Q function and soft V function is: By continuously iterating equations (13) to (14), the output of the Q function will gradually converge to the optimal Q value; During training, the policy π is updated in the exponential direction of the soft Q function: During the training process, two Q networks with the same structure are introduced for simultaneous training, and a smaller Q value is selected to update the strategy network in each iteration; Setting up experience pool D Store samples (s, a, s′, r) obtained by interacting with the environment, and randomly sample from them during training to increase sample independence and improve training efficiency; Assume that the policy network parameter is φ, the Q network parameter is ω, and the loss functions are: in, The calculation formula is: The parameters of the target Q network are The parameters are soft updated in each iteration: where the τ value is greater than 0 but much less than 1.

Images