Path tracking method of unmanned surface vehicle based on deep reinforcement learning

Through a deep reinforcement learning method, the path tracking problem of surface unmanned boats is converted into Markov decision-making process, and the strategy model is constructed using flexible actor critic algorithm, which solves the stability of path tracking control of unmanned boats in complex marine environments. and accuracy issues, enabling adaptive and efficient path tracking.

CN115016496BActive Publication Date: 2024-11-22CHONGQING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210772926.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-30
Publication Date
2024-11-22
Estimated Expiration
2042-06-30

AI Technical Summary

Technical Problem

The existing surface unmanned boat path tracking control methods have stability and accuracy problems in complex marine environments, especially due to increased control difficulty and lack of adaptability due to environmental interference and nonlinear characteristics.

Method used

Using a deep reinforcement learning method, the path tracking problem is converted into Markov decision-making process, and the strategy model is constructed based on the flexible actor critic algorithm, using real-time pose and environmental interference information to generate state values, and output the optimal action value for the path. Track control to avoid modeling dynamics and environment.

Benefits of technology

It improves the stability and accuracy of unmanned boat path tracking control, has adaptive capabilities, can maintain stable tracking effect when environmental changes, reduce path tracking errors, and enhance anti-interference ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115016496B_ABST
    Figure CN115016496B_ABST
Patent Text Reader

Abstract

The present invention specifically relates to a surface unmanned boat path tracking method based on deep reinforcement learning, including: converting the path tracking problem of the target unmanned boat into a Markov decision process, setting the corresponding state space, action space and reward function; obtaining the planned path of the target unmanned boat; calculating the reference heading angle according to the planned path of the target unmanned boat combined with real-time posture information; then generating the current state value of the target unmanned boat based on the reference heading angle, real-time posture information and environmental interference information of the target unmanned boat; inputting the current state value of the target unmanned boat into a trained strategy model, and outputting the optimal action value; the strategy model is constructed based on a flexible actor critic algorithm; sending the optimal action value to the target unmanned boat for execution; until the tracking control of the planned path is completed. The present invention does not require environmental and unmanned boat motion modeling and has adaptive capabilities, thereby further improving the stability and accuracy of the unmanned boat path tracking control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned boat path tracking control, and in particular to a surface unmanned boat path tracking method based on deep reinforcement learning. Background Art

[0002] In recent years, with the development of science and technology, the research on vehicles at home and abroad has gradually developed in the direction of unmanned and intelligent. A large number of intelligent unmanned platforms have emerged. As one of them, unmanned boats have attracted much attention from the shipbuilding and shipping industries. Unmanned surface vehicles (USVs) are small and medium-sized ships that can independently complete navigation and operations according to predetermined procedures without manual operation. Compared with traditional surface ships, they have the characteristics of small size, high maneuverability, low cost, and high intelligence level. At the same time, unmanned boats can be installed with different functional modules according to actual mission requirements, and then realize various functions such as cruising, mapping, and monitoring, meet the requirements of various tasks, and can replace manned ships to complete tasks with high complexity, high risk factor, and high difficulty such as deep-sea resource detection, battlefield reconnaissance, and ship rescue, reducing personnel losses and mission costs.

[0003] Unmanned boats are highly intelligent unmanned systems that integrate environmental perception technology, planning and decision-making technology, and motion control technology. Through their own sensors, they can perceive the external environment and their own status in real time, plan paths and track paths in a dynamically changing and complex external environment, and perform predetermined tasks. Functions such as path planning, environmental perception, and path tracking are the current research focus of unmanned boats. Among them, path tracking is the most basic and important capability of unmanned boats, which guarantees the realization of other functions. The better the motion control performance, the better the overall performance of the unmanned boat.

[0004] To this end, the Chinese patent with publication number CN113625725A discloses "A method for controlling the path tracking of a surface unmanned boat", which includes: setting the expected tracking path, discretizing the expected tracking path to form an expected path dot matrix; judging the distance between the unmanned boat and the starting point of the expected path dot matrix; determining the position of the reference point according to the design parameters and the position of the unmanned boat, obtaining the curvature at the reference point according to the path curvature information, and then obtaining the acceleration according to the guidance algorithm formula: calculating the change in heading angle within the Δt time caused by the acceleration; calculating the expected heading angle according to the change in heading angle and the current heading angle of the unmanned boat; calculating the rudder angle output value of the unmanned boat at the current moment based on the heading deviation information obtained from the expected heading and the actual heading of the unmanned boat, and sending it to the underlying actuator to achieve path tracking.

[0005] The path tracking control method for the surface unmanned boat in the above-mentioned existing scheme does not rely on the precise mathematical model of the unmanned boat and can track any type of path. The applicant found that there are unpredictable environmental disturbances such as wind, waves, and currents in the actual marine environment. These factors cause the unmanned boat control system to have high uncertainty, time-varying and nonlinearity, which makes the motion problem of the unmanned boat complicated and difficult. However, the above-mentioned existing methods do not have adaptive capabilities. When the surface environmental parameters change, not only a complex parameter setting process is required, but it is also difficult to actually ensure the stability of the path tracking control effect, resulting in poor stability of the path tracking control of the unmanned boat. At the same time, the prior art also proposes an anti-disturbance control method based on optimal control and feedback linearization. This type of algorithm can obtain better control accuracy by establishing an accurate model of the environment and the unmanned boat dynamics, but also due to the complexity of the unmanned boat navigation environment, it is difficult to accurately model the environment and the unmanned boat, which leads to poor accuracy of the path tracking control of the unmanned boat. Therefore, how to design a method that can improve the stability and accuracy of the path tracking control of the unmanned boat is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a surface unmanned boat path tracking method based on deep reinforcement learning, which does not require environment and unmanned boat motion modeling and has adaptive capabilities, thereby further improving the stability and accuracy of the unmanned boat path tracking control.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The path tracking method of the surface unmanned boat based on deep reinforcement learning includes the following steps:

[0009] S1: Convert the path tracking problem of the target unmanned boat into a Markov decision process and set the corresponding state space, action space and reward function;

[0010] S2: Obtain the planned path of the target unmanned boat;

[0011] S3: Calculate the reference heading angle according to the planned path of the target unmanned boat combined with the real-time posture information; then generate the current state value of the target unmanned boat based on the reference heading angle of the target unmanned boat, the real-time posture information and the environmental interference information;

[0012] S4: Input the current state value of the target unmanned boat into the trained strategy model and output the optimal action value;

[0013] The policy model is built based on the flexible actor-critic algorithm;

[0014] During training, the reward value returned after the target unmanned boat executes the optimal action value is calculated based on the state space, action space and reward function to update the network parameters of the strategy model;

[0015] S5: Send the optimal action value to the target unmanned boat for execution;

[0016] S6: Repeat steps S3 to S5 until the tracking control of the planned path is completed.

[0017] Preferably, in step S1, the Markov decision process is represented by the following formula:

[0018] 1) The state space is represented as:

[0019]

[0020] Where: represents the difference between the current heading angle of the target unmanned boat and the reference heading angle; d represents the path tracking error; β represents the drift angle; δn d Indicates the rudder angle command size; It represents the difference between the current heading error of the target unmanned boat and the heading error at the previous moment; Δd represents the difference between the current path tracking error and the path tracking error at the previous moment;

[0021] 2) The action space is represented as:

[0022]

[0023] Where: δn d Indicates the rudder angle command;

[0024] 3) The reward function is expressed as:

[0025]

[0026]

[0027]

[0028]

[0029] Where: represents the reward associated with the heading angle; r d represents the reward associated with the path tracking error; represents the reward related to control stability; k1, k2, k3 represent the constant term coefficients set; σ δ It represents the standard deviation of the rudder angle command values ​​​​for 20 times within the preset time period; w1, w2, w3 represent the weight coefficients of each part of the reward item set.

[0030] Preferably, in step S2, the planned path includes a straight line planned path and a curved line planned path;

[0031] 1) The straight line planning path is expressed as:

[0032] P s = <p0,...,p i , p i+1 , ...p T >, p i ∈R 2 ;

[0033] Where: T represents the number of path points of the straight line planning path; p i =(x pi ,y pi ), 1<=i<=T represents the position coordinates of the i-th path point, and R represents an arbitrary real number;

[0034] The curve planning path is expressed as:

[0035] P c :(xx c ) 2 +(yy c ) 2 =R c ;

[0036] Where: p c =(x c ,y c ) represents the coordinates of the center of the curve planning path; R c Indicates the path radius of the curve planning path.

[0037] Preferably, define p t =(x t ,y t ) is the position coordinate of the target unmanned boat at time t;

[0038] 1) For a straight line planning path, the azimuth and path tracking error are calculated using the following formula:

[0039]

[0040]

[0041] Where: p Indicates the azimuth of the target unmanned boat under the straight line planning path; represents the path tracking error of the target unmanned boat under the straight line planning path; p i =(x pi ,y pi ) represents the position coordinates of the i-th path point; p i+1 =(xpi+1 ,y pi+1 ) represents the position coordinates of the i+1th path point;

[0042] 2) For the curved planning path, the azimuth and path tracking error are calculated using the following formula:

[0043] χ c =arctan(y t -y c , x t -x c );

[0044]

[0045]

[0046] Where: c Indicates the azimuth of the target unmanned boat under the curve planning path; represents the path tracking error of the target unmanned boat under the curve planning path; p c =(x c ,y c ) represents the coordinates of the center of the curve planning path; R c represents the path radius; Indicates p c to the vector of p.

[0047] Preferably, in step S3, a reference heading angle is calculated according to the planned path of the target unmanned boat in combination with the real-time posture information by using a vector field algorithm.

[0048] Preferably, the vector field algorithm calculates the reference heading angle using the following formula:

[0049] 1) For straight line planning path:

[0050]

[0051] Where: d represents the reference heading angle under the straight line planning path; p represents the azimuth; χ Ms represents the maximum approach angle, which is set to (0, π / 2]; k s The control parameter representing the convergence speed of the vector field. The larger the value, the faster the heading angle changes. s represents the tracking error of the target unmanned boat to the straight path;

[0052] 2) For curve planning path:

[0053]

[0054] Where: drepresents the reference heading angle under the curve planning path; ρ d Indicates the path direction of the curve planning path, 1 is clockwise and -1 is counterclockwise; Mc represents the maximum approach angle, which is set to (0, π / 2]; k c The control parameter representing the convergence speed of the vector field. The larger the value, the faster the heading angle changes. c represents the azimuth of the target unmanned boat under the curve planning path; d c Represents the tracking error of the target unmanned boat to the curved path.

[0055] Preferably, in step S3, the real-time posture information includes heading angle and drift angle; and the environmental interference information includes wind speed and wind direction.

[0056] Preferably, in step S4, the strategy model is trained by the following steps:

[0057] S401: Initialize the network parameters of the strategy model and the starting position of the target unmanned boat;

[0058] S402: Obtain the current path point information and real-time posture information of the planned path of the target unmanned boat and calculate the current reference heading angle, and then generate the current state value s of the target unmanned boat based on the current reference heading angle, real-time posture information and environmental interference information of the target unmanned boat. t ;

[0059] S403: Current state value s of the target unmanned boat t Input to the strategy model and output the optimal action value a t ; Then control the target unmanned boat to execute the optimal action value a t , and returns the corresponding reward value r t+1 ;

[0060] S404: Obtain the current path point information and real-time posture information of the planned path of the target unmanned boat again and calculate the current reference heading angle, and then generate the execution action value a based on the current reference heading angle, real-time posture information and environmental interference information of the target unmanned boat. t The state value after s t+1 ; Then store and update the experience replay pool D←D∪(s t , a t , r t+1 ,s t+1 );

[0061] S405: Repeat steps S302 to S304 until the final path point is reached or the maximum number of steps in a round is reached;

[0062] S406: Extract M samples from the experience replay pool D to form batch data for training and updating the network parameters of the strategy model.

[0063] Preferably, in step S406, the network parameters to be updated in the strategy model include network parameters of the Q network and the π network and temperature parameters;

[0064] The objective function of the Q network is expressed as:

[0065]

[0066] Where: φ represents the network parameters of the Q network; represents the expectation of all strategies; Q φ represents the action-state value function; γ represents the discount factor; V φ represents the state value function;

[0067] The objective function of the π network is expressed as:

[0068]

[0069] Where: θ represents the network parameters of the π network; D KL represents KL divergence (an indicator of similarity); π θ represents the policy network; Q φ represents the action-state value function; γ represents the discount factor; V φ represents the state value function;

[0070] The objective function of the temperature parameter is expressed as:

[0071]

[0072] Where: α represents the temperature parameter; represents the expectation; H0 represents the initial strategy entropy threshold.

[0073] Preferably, the network parameter update process of the policy model is as follows:

[0074] Update the network parameters of the Q network:

[0075] Update the network parameters of the Pi network:

[0076] Update temperature parameters:

[0077] Update the network parameters of the target Q network:

[0078] Where: Q represents the learning rate of the Q network; Denotes the network parameter θ of the Q network i The gradient of π represents the learning rate of the policy network; represents the gradient of the network parameter φ of the policy network; λ α represents the learning rate of the temperature parameter; represents the gradient of the temperature parameter α; θ i represents the network parameters of the Q network; Represents the network parameters of the target Q network.

[0079] The surface unmanned boat path tracking method based on deep reinforcement learning in the present invention has the following beneficial effects:

[0080] The present invention converts the path tracking problem of the target unmanned boat into a Markov decision process, and then generates a state value through the reference heading angle, real-time posture information and environmental interference information of the target unmanned boat, and inputs the trained strategy model to output the optimal action value, and finally sends the optimal action value to the target unmanned boat for execution to complete the tracking control of the planned path. On the one hand, the strategy model of the present invention is constructed based on the flexible actor-critic algorithm, and through Markov decision process training and parameter update, it is possible to apply the deep reinforcement learning algorithm to the motion control of the unmanned boat, and this control method based on deep reinforcement learning does not need to perform artificial explicit modeling of the dynamics and environmental interference of the unmanned boat, and through the state value generated by the reference heading angle, real-time posture information and environmental interference information, it can autonomously implicitly model the environmental interference and learn the optimal control strategy in the process of interaction with the environment, thereby improving the accuracy of the path tracking control of the unmanned boat. On the other hand, the present invention converts the path tracking problem of the target unmanned boat into a Markov decision process with adaptive capabilities. When the water surface environment parameters change, no complicated parameter setting process is required, and the stability of the path tracking control effect can be guaranteed. In addition, by reasonably setting the state space, action space and reward function of the Markov decision process, the path tracking error of the strategy model can be reduced, and the heading control stability, adjustment speed and anti-interference ability can be improved, thereby further improving the stability of the path tracking control of the unmanned boat.

[0081] At the same time, the present invention indirectly controls the unmanned boat through the reference heading angle of the unmanned boat. Compared with the existing method of directly using the current distance from the unmanned boat to the path point as the reference for the unmanned boat heading control, the problem of changes in the absolute distance representation range caused by different map scales can be avoided, thereby further improving the accuracy of the unmanned boat path tracking control. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0083] Figure 1 This is a logical block diagram of the path tracking method for unmanned surface vehicles based on deep reinforcement learning;

[0084] Figure 2 Schematic diagram of the straight line path tracking problem and the curve path tracking problem;

[0085] Figure 3 This is a framework diagram of the unmanned boat path tracking algorithm based on the flexible actor-critic algorithm;

[0086] Figure 4 Examples of the VF algorithm in a straight line tracking path and a curve tracking path;

[0087] Figure 5 Reward function curve for path tracking with multiple constraints;

[0088] Figure 6 This is a schematic diagram of the plane motion of the WAM-V unmanned boat;

[0089] Figure 7 This is the architecture diagram of the ROS software package for reinforcement learning based on the VRX environment;

[0090] Figure 8 The curves of average reward and average round length during training;

[0091] Fig. 9 Tracking results for straight path scenario;

[0092] Fig.10 It is the tracking error and heading angle error variation curve of the straight path scene;

[0093] Fig.11 Experimental results of path tracking for the path search scenario;

[0094] Fig.12 Tracking error changes for path search scenarios;

[0095] Fig.13 To search for the change in heading angle error of the path scenario;

[0096] Fig.14 The experimental results of path tracking in curved path scene are shown;

[0097] Fig.15 Tracking error and heading angle error changes for curved path scenarios;

[0098] Fig.16 The curves of average reward and average round length during algorithm training under different state space designs;

[0099] Fig.17 are the average and RMS values ​​of the tracking error under different average wind speed conditions;

[0100] Fig.18are the average and root mean square values ​​of heading angle error under different average wind speed conditions;

[0101] Fig.19 Path tracking experimental results for path search scenarios under different wind speed conditions. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0103] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0104] The following is a further detailed description through specific implementation methods:

[0105] Example:

[0106] This embodiment discloses a path tracking method for an unmanned surface vehicle based on deep reinforcement learning.

[0107] like Figure 1 As shown, the path tracking method of the surface unmanned boat based on deep reinforcement learning includes the following steps:

[0108] S1: Convert the path tracking problem of the target unmanned boat into a Markov decision process and set the corresponding state space, action space and reward function;

[0109] S2: Obtain the planned path of the target unmanned boat;

[0110] S3: Calculate the reference heading angle according to the planned path of the target unmanned boat combined with the real-time posture information; then generate the current state value of the target unmanned boat based on the reference heading angle of the target unmanned boat, the real-time posture information and the environmental interference information;

[0111] In this embodiment, the real-time posture information includes the heading angle and the drift angle; the environmental interference information includes the wind speed and the wind direction. The real-time posture information is obtained through the GPS and IMU installed on the target unmanned boat; the environmental interference information is obtained through the wind speed sensor installed on the target unmanned boat.

[0112] S4: Input the current state value of the target unmanned boat into the trained strategy model and output the optimal action value;

[0113] The policy model is built based on the flexible actor-critic algorithm;

[0114] During training, the reward value returned after the target unmanned boat executes the optimal action value is calculated based on the state space, action space and reward function to update the network parameters of the strategy model;

[0115] S5: Send the optimal action value to the target unmanned boat (actuator) for execution;

[0116] S6: Repeat steps S3 to S5 until the tracking control of the planned path is completed.

[0117] The present invention converts the path tracking problem of the target unmanned boat into a Markov decision process, and then generates a state value through the reference heading angle, real-time posture information and environmental interference information of the target unmanned boat, and inputs the trained strategy model to output the optimal action value, and finally sends the optimal action value to the target unmanned boat for execution to complete the tracking control of the planned path. On the one hand, the strategy model of the present invention is constructed based on the flexible actor-critic algorithm, and through Markov decision process training and parameter update, it is possible to apply the deep reinforcement learning algorithm to the motion control of the unmanned boat, and this control method based on deep reinforcement learning does not need to perform artificial explicit modeling of the dynamics and environmental interference of the unmanned boat, and through the state value generated by the reference heading angle, real-time posture information and environmental interference information, it can autonomously implicitly model the environmental interference and learn the optimal control strategy in the process of interaction with the environment, thereby improving the accuracy of the path tracking control of the unmanned boat. On the other hand, the present invention converts the path tracking problem of the target unmanned boat into a Markov decision process with adaptive capabilities. When the water surface environment parameters change, no complicated parameter setting process is required, and the stability of the path tracking control effect can be guaranteed. In addition, by reasonably setting the state space, action space and reward function of the Markov decision process, the path tracking error of the strategy model can be reduced, and the heading control stability, adjustment speed and anti-interference ability can be improved, thereby further improving the stability of the path tracking control of the unmanned boat.

[0118] At the same time, the present invention indirectly controls the unmanned boat through the reference heading angle of the unmanned boat. Compared with the existing method of directly using the current distance from the unmanned boat to the path point as the reference for the unmanned boat heading control, the problem of changes in the absolute distance representation range caused by different map scales can be avoided, thereby further improving the accuracy of the unmanned boat path tracking control.

[0119] In the specific implementation process, Figure 2 As shown, the planned path includes a straight line planned path and a curve planned path;

[0120] 1) The straight line planning path is expressed as:

[0121] P s =<p0, ..., p i , p i+1 , ...p T >, p i ∈R 2 ;

[0122] Where: T represents the number of path points of the straight line planning path; p i =(x pi ,y pi ), 1<=i<=T represents the position coordinates of the i-th path point; R represents an arbitrary real number;

[0123] The curve planning path is expressed as:

[0124] P c :(xx c ) 2 +(yy c ) 2 =R c ;

[0125] Where: p c =(x c ,y c ) represents the coordinates of the center of the curve planning path; R c Indicates the path radius of the curve planning path.

[0126] Define p t =(x t ,y t ) is the position coordinate of the target unmanned boat at time t;

[0127] 1) For a straight line planning path, the azimuth and path tracking error are calculated using the following formula:

[0128]

[0129]

[0130] Where: p Indicates the azimuth of the target unmanned boat under the straight line planning path; represents the path tracking error of the target unmanned boat under the straight line planning path; p i =(x pi ,y pi ) represents the position coordinates of the i-th path point; p i+1 =(x pi+1 ,y pi+1 ) represents the position coordinates of the i+1th path point;

[0131] 2) For the curved planning path, the azimuth and path tracking error are calculated using the following formula:

[0132] χ c =arctan(y t -y c , x t -x c );

[0133]

[0134]

[0135] Where: cIndicates the azimuth of the target unmanned boat under the curve planning path; represents the path tracking error of the target unmanned boat under the curve planning path; p c =(x c ,y c ) represents the coordinates of the center of the curve planning path; R c represents the path radius; Indicates p c to the vector of p.

[0136] The present invention takes into account the different effects of straight-line planning paths and curved planning paths on the path tracking control of unmanned boats, and designs corresponding parameter calculation schemes, so that the path tracking problem of the target unmanned boat can be better converted into an adaptive Markov decision process, so that the state space, action space and reward function of the Markov decision process can be set more reasonably, thereby reducing the path tracking error of the strategy model, and improving the heading control stability, adjustment speed and anti-interference ability, thereby further improving the stability of the path tracking control of the unmanned boat.

[0137] In the specific implementation process, Figure 3 As shown, the policy model is trained through the following steps:

[0138] S401: Initialize the network parameters of the strategy model and the starting position of the target unmanned boat;

[0139] In this embodiment, the random seed size, the parameters of the guidance algorithm, the maximum training step size and the maximum round step size are set, and the target unmanned boat path tracking scene simulation environment is initialized; the Q network parameters θ1, θ2 are randomly initialized; the π network parameters φ; the target network parameters are initialized Initialize experience replay pool

[0140] S402: Obtain the current path point information and real-time posture information of the planned path of the target unmanned boat and calculate the current reference heading angle, and then generate the current state value s of the target unmanned boat based on the current reference heading angle, real-time posture information and environmental interference information of the target unmanned boat. t ;

[0141] S403: Current state value s of the target unmanned boat t Input to the policy model (policy network) and output the optimal action value a t ; Then control the target unmanned boat to execute the optimal action value a t , and returns the corresponding reward value r t+1 ;

[0142] S404: Obtain the current path point information and real-time posture information of the planned path of the target unmanned boat again and calculate the current reference heading angle, and then generate the execution action value a based on the current reference heading angle, real-time posture information and environmental interference information of the target unmanned boat. t The state value after s t+1 ; Then store and update the experience replay pool D←D∪(s t , a t , r t+1 ,s t+1 );

[0143] S405: Repeat steps S302 to S304 until the final path point is reached or the maximum number of steps in a round is reached;

[0144] S406: Extract M samples from the experience replay pool D to form batch data for training and updating the network parameters of the strategy model.

[0145] The strategy model of the present invention is constructed based on the flexible actor-critic algorithm, and the above-mentioned training and parameter update are performed through the Markov decision process, so that the deep reinforcement learning algorithm can be applied to the motion control of the unmanned boat. This control method based on deep reinforcement learning does not require artificial explicit modeling of the dynamics and environmental interference of the unmanned boat, and by referring to the state values ​​generated by the heading angle, real-time posture information and environmental interference information, it can autonomously implicitly model the environmental interference and learn the optimal control strategy during the interaction with the environment, thereby improving the accuracy of the unmanned boat path tracking control.

[0146] During the specific implementation process, when the unmanned boat is tracking the path, the role of the guidance algorithm is to calculate the reference heading angle of the unmanned boat at the next moment based on the current position of the unmanned boat, and indirectly control the output of the unmanned boat propeller to keep the unmanned boat in the predetermined heading or converge to the desired path. Compared with directly using the distance from the current unmanned boat to the path point as the reference for the heading control of the unmanned boat, using the reference heading angle as the reference for heading control can avoid the problem of changes in the absolute distance representation range caused by different map scales. The vector field (VF) algorithm is a commonly used heading guidance algorithm in unmanned boats and drones. Compared with the line of sight guidance algorithm, the reference heading angle given by the VF algorithm is smoother, and the reference trajectory is shorter than the LOS algorithm. At the same time, the VF algorithm requires fewer parameters to be manually set than the LOS algorithm, and has lower computational complexity.

[0147] Therefore, the present invention adopts the VF algorithm as the heading guidance algorithm for the path tracking task. The reference heading angle is calculated by the vector field algorithm according to the planned path of the target unmanned boat combined with the real-time posture information.

[0148] like Figure 4As shown in the figure, the VF algorithm follows two principles when calculating the reference heading: first, when the unmanned boat is far away from the following path, the reference heading angle should guide the unmanned boat to return to the path as quickly as possible with the maximum approach angle; second, when the unmanned boat is close to the following path, the reference heading angle should align the movement direction of the unmanned boat with the direction of the straight path or the tangent direction of the curved path.

[0149] The vector field algorithm calculates the reference heading angle using the following formula:

[0150] 1) For straight line planning path:

[0151]

[0152] Where: d represents the reference heading angle under the straight line planning path; p represents the azimuth; χ Ms represents the maximum approach angle, which is set to (0, π / 2]; k s The control parameter representing the convergence speed of the vector field. The larger the value, the faster the heading angle changes. s represents the tracking error of the target unmanned boat to the straight path;

[0153] 2) For curve planning path:

[0154]

[0155] Where: d represents the reference heading angle under the curve planning path; ρ d Indicates the path direction of the curve planning path, 1 is clockwise and -1 is counterclockwise; Mc represents the maximum approach angle, which is set to (0, π / 2]; k c The control parameter representing the convergence speed of the vector field. The larger the value, the faster the heading angle changes. c represents the azimuth of the target unmanned boat under the curve planning path; d c Represents the tracking error of the target unmanned boat to the curved path.

[0156] The present invention calculates the reference heading angle by combining the planned path and real-time posture information of the target unmanned boat with the vector field algorithm, and can calculate a reference heading angle with a smoother reference trajectory and a shorter reference trajectory, and requires fewer parameters to be manually set, and has lower calculation complexity, thereby improving the calculation accuracy and efficiency of the reference heading angle. At the same time, the present invention indirectly controls the unmanned boat through the reference heading angle of the unmanned boat. Compared with the existing method of directly using the distance from the current unmanned boat to the path point as the reference for the unmanned boat's heading control, it can avoid the problem of absolute distance representation range changes caused by different map scales, thereby further improving the accuracy of the unmanned boat's path tracking control.

[0157] In the specific implementation process, the flexible actor-critic algorithm uses a sample experience replay pool to store the sampled data of each round. Each time it is updated, the flexible actor-critic algorithm randomly extracts batches of sampled data from the experience replay pool D and uses stochastic gradient descent to update the network parameters. At the same time, the flexible actor-critic algorithm adds the target network delayed update mechanism of the DQN algorithm to enhance the stability of the training process. In addition, during the training process of the flexible actor-critic algorithm, the temperature parameter α value is very sensitive to changes in the reward size. If α is set to a fixed value, it is easy to cause training instability and difficulty in algorithm convergence due to inappropriate reward function value size.

[0158] Therefore, the network parameters that need to be updated in the strategy model include the network parameters of the Q network and the π network as well as the temperature parameters;

[0159] The objective function of the Q network is expressed as:

[0160]

[0161] Where: φ represents the network parameters of the Q network; represents the expectation of all strategies; Q φ represents the action-state value function; γ represents the discount factor; V φ represents the state value function;

[0162] The objective function of the π network is expressed as:

[0163]

[0164] Where: θ represents the network parameters of the π network; D KL represents KL divergence (an indicator of similarity); π θ represents the policy network; Q φ represents the action-state value function; γ represents the discount factor; V φ represents the state value function;

[0165] The objective function of the temperature parameter is expressed as:

[0166]

[0167] Where: α represents the temperature parameter; represents the expectation; H0 represents the initial strategy entropy threshold.

[0168] The network parameter update process of the policy model is as follows:

[0169] Update the network parameters of the Q network:

[0170] Update the network parameters of the Pi network:

[0171] Update temperature parameters:

[0172] Update the network parameters of the target Q network:

[0173] Where: Q represents the learning rate of the Q network; Denotes the network parameter θ of the Q network i The gradient of π represents the learning rate of the policy network; represents the gradient of the network parameter φ of the policy network; λ α represents the learning rate of the temperature parameter; represents the gradient of the temperature parameter α; θ i represents the network parameters of the Q network; Represents the network parameters of the target Q network.

[0174] In the specific implementation process, in the design of path tracking control algorithm based on reinforcement learning, modeling the problem as a Markov decision process is a very important step. The selection of state space, action space and reward function has a great influence on the performance and convergence speed of the strategy model.

[0175] The Markov decision process is represented by the following formula:

[0176] 1) State space design

[0177] Because the guidance algorithm gives the reference heading angle for the unmanned boat to track the path, the unmanned boat heading angle can be adjusted according to the reference heading angle to indirectly achieve the path tracking control effect. Therefore, we add the difference between the current unmanned boat heading angle and the output reference heading angle of the VF algorithm into the state space, which is expressed as Allow the strategy model to accurately perceive the control effect of the current heading angle of the unmanned boat during the control process. Considering that the goal of the path tracking problem of the unmanned boat is to minimize the tracking error of the unmanned boat to the tracking path, in addition to the indirect control based on a given reference heading angle, the path tracking error d is also added to the state space. Because there are complex environmental disturbances in the water environment, and the unmanned boat itself is underdriven and has low friction on the water surface, the unmanned boat is prone to lateral drift during the actual tracking process, affecting the effect of the actual heading angle control. Therefore, the drift angle β of the current unmanned boat and the rudder angle command size δn are added to the state space. d, so that the strategy model can sense the current drift degree of the unmanned boat, so as to perform better control. In addition, in order to enable the strategy model to achieve a more stable control effect, in addition to only selecting the current state, we also added the state value comparison between the current moment and the previous moment in the state space, including the difference between the current heading error and the heading error of the previous moment. The difference between the current tracking error and the previous tracking error is Δd = d t -d t-1 , so that the state space can better reflect whether the unmanned boat is moving in the direction where the tracking error and heading angle error are reduced.

[0178] The state space is represented as:

[0179]

[0180] Where: represents the difference between the current heading angle of the target unmanned boat and the reference heading angle; d represents the path tracking error; β represents the drift angle; δn d Indicates the rudder angle command size; It represents the difference between the current heading error of the target unmanned boat and the heading error at the previous moment; Δd represents the difference between the current path tracking error and the path tracking error at the previous moment;

[0181] 2) Action space design

[0182] Because we are studying the path tracking problem of the unmanned boat at a fixed cruising speed, the action space is set to the rudder angle command size of the unmanned boat, that is, δn d The value range and maximum change speed are set according to the requirements of the unmanned boat maneuvering response model in the prior art.

[0183] The action space is represented as:

[0184]

[0185] Where: δn d Indicates the rudder angle command;

[0186] 3) Reward function design

[0187] In order to better complete the tracking task, we designed a multi-constrained reward function based on the relevant experience of actual unmanned boat control. The reward function is designed into three parts: the first is the reward related to the heading angle If the heading angle of the unmanned boat is closer to the reference heading angle given by the VF algorithm, the reward value is higher, which encourages the algorithm to adjust the heading angle of the unmanned boat to the reference heading angle as soon as possible; the second is the reward rd related to the current tracking error of the unmanned boat. The smaller the current lateral tracking of the unmanned boat, the higher the reward obtained, which encourages the algorithm to find a strategy that can minimize the tracking error while adjusting the heading angle of the unmanned boat to the reference heading angle, so that it can enter the target tracking path faster; the last is the reward related to control stability The slower the rudder angle command value changes, the higher the reward value, which encourages the algorithm to find a more stable and smoother unmanned boat motion control strategy, making the control strategy more in line with the needs of most actual mission requirements.

[0188] The reward function is expressed as:

[0189]

[0190]

[0191]

[0192]

[0193] Where: represents the reward associated with the heading angle; r d represents the reward associated with the path tracking error; represents the reward related to control stability; k1, k2, k3 represent the constant term coefficients set; σ δ It represents the standard deviation of the rudder angle command values ​​​​for 20 times within the preset time period; w1, w2, w3 represent the weight coefficients of each part of the reward item set.

[0194] In actual training, k1, k2, and k33 are set to 6.0, 0.2, and 1.05 respectively. The reward function change curve drawn at this time is Figure 5 As shown in the figure. From the trend of the reward curve change in the figure, we can see that the reward value range of the heading error is [-1, 1], reaching the maximum value of 1 when the heading error is 0, and decreasing as the absolute value of the heading error increases; the reward value range of the lateral error is [0, 1], which is the maximum when the tracking error is 0, and also decreases as the absolute value of the tracking error increases; the reward value range of the rudder angle command is [0, 1], and the greater the change in adjacent rudder angle commands, the lower the reward value. The change trend of the reward function of each part meets our expectations. During the training process, the values ​​of w1, w2, and w3 are 0.5, 0.4, and 0.1, respectively.

[0195] The present invention converts the path tracking problem of the target unmanned boat into an adaptive Markov decision process, so that when the water surface environment parameters change, no complicated parameter setting process is required, and the stability of the path tracking control effect can be guaranteed; and by reasonably setting the state space, action space and reward function of the Markov decision process including rewards related to the heading angle, rewards related to the path tracking error and rewards related to the control stability, the path tracking error of the strategy model can be reduced, and the heading control stability, adjustment speed and anti-interference ability can be improved, so that the stability of the path tracking control of the unmanned boat can be further improved.

[0196] In order to better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.

[0197] This experiment first designed the unmanned boat reinforcement learning simulation environment.

[0198] 1) Water simulation environment

[0199] The VRX simulation environment is a virtual simulation environment based on ROS and Gazebo developed by RoboNation, the Office of Naval Research (ONR) and the Naval Postgraduate School (NPS) for the Robotx competition. It extends the original functions of Gazebo by adding additional plug-ins, and realizes functions such as water surface environment model, ship dynamics model, sensor simulation and general marine object model. The VRX environment has sufficient fidelity for the real water environment. At the same time, the Gazebo software itself can achieve simulation speeds that are many times faster than the real world environment, which is suitable for a series of water task simulations, allowing developers to quickly develop new solution prototypes and smoothly transition the algorithm from the simulation environment to the real water environment after certain tuning and optimization. The VRX environment is written in C++ and Python under the framework of ROS. The API and control scripts provided are also the standard communication format of ROS. C++ or Python can be selected on its basis for flexible secondary development, which is convenient for developing machine learning algorithms in combination with deep learning frameworks such as Tensorflow and Pytorch. Therefore, this experiment chooses VRX as the surface simulation environment used in the experiment.

[0200] 2) Kinematics and dynamics model of unmanned boat

[0201] This experiment selected a 16-foot wave adaptive modular unmanned boat (WAM-V) platform as the surface unmanned boat experimental platform for algorithm development and testing. Figure 6As shown in the figure, the WAM-V unmanned boat is a catamaran that uses two inflatable pontoons as the hull. The hull is equipped with two electric thrusters. The thruster vector angle is fixed and the steering operation is performed by differential control. This experiment uses a simplified three-degree-of-freedom dynamic model to model the kinematics and dynamics of the WAM-V unmanned boat in a simulation environment, and briefly introduces its control response model.

[0202] Since the horizontal plane motion of the unmanned boat is the main focus of the path tracking problem, this experiment only focuses on the three degrees of freedom of the surface unmanned boat: sway, pitch, and pitch. A simplified three-degree-of-freedom unmanned boat dynamic model is used, and the properties of the unmanned boat hull are marked according to the agreed rules in the prior art. i and i is the due north and due east direction in the inertial coordinate system, while u, v and V are the sway, surge and total velocity relative to the USV's own body coordinate system. ψ represents the bow angle of the ship, χ and β represent the heading angle and motion drift angle, and β = χ-ψ. port and T stbd Represent the force of the left and right thrusters respectively.

[0203] Therefore, the three-degree-of-freedom kinematic model of the unmanned boat can be expressed by the following formula, where v and η are the velocity vector and position vector of the unmanned boat, and R(η) is the rotation matrix of the unmanned boat's appendage coordinate system relative to the inertial coordinate system.

[0204]

[0205] v=(u,v,r) T ;

[0206] η=(x i ,y i , ψ) T ;

[0207]

[0208] The three-degree-of-freedom dynamic model of the unmanned boat can be described as:

[0209]

[0210] Where M is the mass inertia matrix of the unmanned boat,

[0211]

[0212] C(v) is the Coriolis centripetal force matrix,

[0213]

[0214] D(v) is the damping coefficient matrix,

[0215]

[0216] f is the propulsion force and torque. In the differential motion mode of the unmanned boat, f can be expressed as:

[0217]

[0218] Where B is the width of the ship.

[0219] 3) Unmanned boat maneuvering response model

[0220] The unmanned boat maneuvering response model gives the relationship between the input (rudder angle, speed) and the bow angle and speed change rate of the unmanned boat. The propeller model of the unmanned boat is implemented with reference to the linear maneuvering response model and parameter identification results in the prior art, so that the thrust of the propeller can be directly calculated from the revolution (s) per minute (RPM) instruction of the propeller to achieve interaction with the simulation environment. The propeller RPM instruction value range is set to [-1, 1], where 1 corresponds to the maximum speed of the propeller, -1 corresponds to the minimum speed, and the rate of change of the RPM instruction per unit second is 1. δn port and δn stbd are the RPM instructions for the left and right thrusters of the unmanned boat, δn max = is the maximum difference between the left and right propulsion RPMs, which is 2. Because the WAM-V unmanned boat adopts differential motion mode, the rudder angle command δn d and speed command δn m It can be expressed as:

[0221] δn d =(δn port -δn stbd ) / (2δn max );

[0222] δn m =(δn port +δn stbd ) / (2δn max );

[0223] It can be seen from the above formula that the rudder angle command δn d and speed command δn m The value range is [-1, 1].

[0224] 4) Packaging of simulation environment and unmanned boat model

[0225] The original VRX simulation environment realizes a water simulation environment with sufficient fidelity, but it is not suitable to be used directly as a simulation environment for reinforcement learning algorithm training. Therefore, this experiment designed a ROS software package to encapsulate the original water simulation environment and unmanned boat simulation model, implement the Open AIGYM interface, and build an unmanned boat water environment suitable for the development of reinforcement learning algorithms for reinforcement learning training. The generation method of the unmanned boat path tracking task scenario is also given.

[0226] like Figure 7 As shown in the figure, in order to reduce the coupling between codes and improve the code reuse rate, we abstract the entire software package framework into three classes: Task Environment RobotEnvironment and GazeboEnvironment. There is a hierarchical inheritance relationship between the three classes.

[0227] Among them, the GazeboEnvironment class directly inherits from the Gym Environment, is responsible for uniformly defining the interface functions required by the GYM environment, and managing the communication between the reinforcement learning algorithm and the VRX environment; the RobotEnvironment inherits from the GazeboEnvironment, is responsible for defining the manipulation method of the unmanned boat used in the experiment, defining the type of sensors selected by the unmanned boat and the method of receiving sensor data, and providing the operation interface to the reinforcement learning algorithm; the TaskEnvironment inherits from the Robot Environment, is responsible for setting the requirements of the current task, and processing the state collection and reward feedback of the current learning task. The software package decouples the communication, unmanned boat simulation model and the specific task environment, making the software package not only suitable for the reinforcement learning training of the path tracking task of the WAM-V unmanned boat water environment, but also can easily replace the simulation models of different types of unmanned boats and switch different unmanned boat water sports control tasks. The specific role of each part will be explained in detail below.

[0228] Gazebo Environment: Mainly used to load simulation environment settings, start VRX simulation process, and use ROS to realize the communication between reinforcement learning algorithm and VRX simulation environment. It realizes the seed function of random seed setting in GYM environment, reset simulation reset function, execute training step step function and close simulation close function. During the execution process, it will call the subclass function to obtain the environment observation state value and execute action, and publish the cumulative reward function value of each round and other information.

[0229] Robot Environment: Implements all ROS functions required for the UAV to control and perceive the surrounding environment, and checks the UAV functions before starting the UAV control each time (checks whether the ROStopic and service required for control and perception are started normally and can be controlled, and checks whether the sensor installation position and the UAV's own posture position are correct). And loads the UAV simulation model with a specific configuration into the VRX environment.

[0230] Task Environment is responsible for loading the task scene map into the VRX environment and providing all the context information for specific tasks of intelligent learning, including: initializing the environment variables at the beginning of each round, initializing the position of the unmanned boat, letting the unmanned boat perform the action selected by the algorithm, obtaining the environmental observation value obtained by the unmanned boat when performing the selected action, calculating the reward value of the environmental feedback at each step, and checking whether the agent has reached the end state of the round. In this experiment, only the scene settings of the path tracking task and the autonomous obstacle avoidance task of the surface unmanned boat are given.

[0231] Specific experimental process:

[0232] 1. Hyperparameter settings

[0233] In view of the characteristics of the path tracking task, two hidden layers are set for the Q network, π network and target network. The number of units in each hidden layer is 256. The hidden layer activation function selects the ReLU function. The gradient descent uses the Adam optimizer, the learning rate is 0.0003, the batch size is set to 256, and a normalization layer is added before the input layer to unify the format of the input data, so that the network converges better. During the training process, the specific network parameter settings such as the reward function discount factor γ are shown in Table 1.

[0234] Table 1

[0235]

[0236] 2. Comparison of algorithms and evaluation indicators

[0237] In order to demonstrate the effectiveness of the unmanned boat path tracking algorithm based on the flexible actor-critic algorithm proposed in this invention, this experiment sets up the traditional PID control algorithm, the DDPG-based and PPO-based unmanned boat path tracking algorithms as comparison algorithms for unmanned boat path tracking.

[0238] The specific settings of the comparison algorithm are as follows:

[0239] 1) PID: PID is a commonly used traditional model-free control algorithm. According to the settings in the prior art, after the VF algorithm outputs the reference heading angle, PID is used to calculate the control amount of the heading angle according to the error between the current heading angle and the reference heading angle, and the rudder angle control command is calculated according to this error to perform path tracking control. The specific control parameters are set as follows: kp = 3, ki = 0.01, kd = 3.

[0240] (2) DDPG: DDPG is an offline reinforcement learning algorithm that combines the Actor-Critic framework with gradient descent, adopts a deterministic strategy, and can handle continuous action spaces. In the comparative experiment, the network learning rate is set to 0.0001, the action network and the value network share network parameters, the exploration noise variable has a mean of 0, a variance of 0.25, and the soft target update coefficient is set to 0.005.

[0241] (2) PPO: The PPO algorithm is an online learning algorithm. The PPO algorithm is also based on the Actor-Critic framework and uses a random strategy to handle continuous control problems. The PPO algorithm uses importance sampling technology to improve the problem of low data utilization in traditional online learning algorithms, and uses the advantage function to clip the gap between the new and old strategies to make the strategy update process more stable. During the training of the PPO algorithm, the learning rate is set to 0.003, the action network clipping parameter is 0.2, the policy gradient clipping maximum value is 0.5, the entropy coefficient and the value network entropy coefficient are set to 0.0 and 0.5 when calculating the loss function, the GAE parameter is 0.95, and other network hyperparameters such as the discount factor and the hidden layer network structure are the same as those in Table 4.1.

[0242] In the path tracking experiment of the unmanned boat, we selected the mean error (Meanerror) and root mean square error (RMSE) of the tracking error and heading angle error as the final evaluation indicators for evaluating the control effect of the path tracking algorithm.

[0243] 3. Analysis of path tracking experimental results

[0244] 3.1 Algorithm training process setup and analysis

[0245] For several path tracking algorithms based on reinforcement learning, in order to make the trained controller model adapt to a variety of different path tracking environments, we use existing experimental ideas to simulate various path corner changes encountered in the actual unmanned boat navigation environment. The algorithm uses a straight path with a random heading angle and a length of 60m to perform track tracking training in each round. The initial position of the unmanned boat is randomly set within 20m of the starting point of the path, and the initial heading angle of the unmanned boat is random. The speed of the unmanned boat is set to a cruising speed of 1.2m / s, and the controller control frequency is 10Hz. The parameters of the ocean current interference in the training environment are set to gain=0.1, period=5, direction_x=1.0, direction_y=0.0, angle=0.4, scale=1.5, simulating the micro-wave scene in the real environment. During the algorithm training process, we set the maximum training time step of the algorithm to 2.4×106 steps, the maximum step length per round is 1200, and the sampling time interval is 0.1s. The algorithm is trained according to the process in the present invention. After the training, we analyze the training results as follows.

[0246] Figure 8These are the average reward change curves and the average round length change curves of the three unmanned boat path tracking training processes based on reinforcement learning. The three algorithms are all based on the same training scenario and the same training settings to obtain the results. From the figure, we can get: 1) The training process model of the path tracking control algorithm based on the flexible actor critic proposed in the present invention converges faster, the average round length decreases faster, and the final result is more stable (it has converged at about 1×106), and the highest average reward is obtained. Compared with the other two algorithms, it has faster training speed and more stable results, and the tracking control adjustment speed is fast; 2) In the early stage of training (about 6.5×105 steps before) of the PPO path tracking algorithm, the growth rate of the average reward is also greater than that of the DDPG-based path tracking algorithm. The average reward growth rate of the two algorithms based on random strategies is greater than that of the deterministic strategy algorithm in the early stage, indicating that the random strategy encourages more exploration of the environment, and can find a better unmanned boat path tracking control strategy faster in the early stage of training; 3) The reward of the path tracking control algorithm based on PPO changes greatly with time, and is the most unstable. The same average round length change curve also has a large change range. Such changes show that PPO, as an online learning algorithm, lacks a replay experience pool, and the collected data is discarded after updating, which makes the training process more unstable than the other two offline algorithms; 4) The changing trends of the average round length curves of the three algorithms are basically opposite to the changing trends of the average return reward. As the time step increases, the average return of the algorithm increases while the average round length decreases, indicating that the three algorithms have learned strategies to complete the task faster and meet the requirements of the path tracking task, which also shows that the reward function we designed is reasonable.

[0247] according to Figure 8 As can be seen from the results in , we use the model parameters after training convergence (about 2.25×106 time steps) as the model parameters used for path tracking results comparison of the three algorithms. Considering the actual driving environment of the unmanned boat, three different simulation scenarios are set up below: straight path tracking scenario, search path tracking scenario and curve path tracking scenario to compare and show the actual performance of the algorithm. In all three scenarios, the sea wave interference with the same parameters as the training environment is set.

[0248] 3.2. Linear path tracking experimental results

[0249] We designed a straight line path as the planning path (path points are (0,0), (0,80), and the length is 80m) for the unmanned boat to track, and set the starting point of the unmanned boat to (-15,0). Fig. 9 The comparison between the actual path and the planned path generated by different algorithms in this scenario is shown. Fig.10The tracking error change curve and heading angle error change curve of the unmanned boat during the path tracking process are given respectively. It can be seen from the figure that the three unmanned boat path tracking control algorithms based on reinforcement learning can achieve better path tracking effects than the traditional PID control algorithm in an environment with certain sea wave interference. At the same time, by observing the changes in the heading angle error of the unmanned boat during the tracking process of the PID algorithm in the presence of sea wave interference, it can be found that it can never be adjusted to steady-state control and there is oscillation. Table 2 further compares the path control effects of several algorithms on unmanned boats under straight paths.

[0250] Table 2

[0251]

[0252] It can be concluded from Table 2 that although the PID-based path tracking control has the smallest root mean square error of the heading angle error, its performance in tracking error is very poor, while the path tracking algorithm based on deep reinforcement learning can obtain more balanced results in tracking error and heading angle tracking error, both of which are controlled in a smaller range. The reason for this phenomenon is that the PID control algorithm we implemented is for a single-input and single-output system, and the selection of the rudder angle control command is only related to the current heading angle error, so the algorithm aggressively chooses the action that can reduce the heading angle error the fastest, without considering how to move the unmanned boat to the target path faster. At the same time, due to the existence of wave interference, the PID-based control algorithm does not model the interference during control, resulting in its inability to maintain stable control and oscillation. Although the other three methods based on deep reinforcement learning are also model-free control, they can simultaneously focus on multiple optimization targets such as heading angle error and path tracking error, and can also receive additional state information such as the current drift angle of the unmanned boat, and can simultaneously optimize multiple targets to produce better control effects; and during the training process, the environment interference can be implicitly modeled into the algorithm model, and it can show more anti-interference ability in the face of environmental interference and make more robust control. The experimental results show the effectiveness of deep reinforcement learning in unmanned boat control.

[0253] We further analyzed the experimental results. It can be seen from the figure that the path tracking control algorithm proposed in the present invention performs best among the three path tracking control algorithms based on deep reinforcement learning, with the smallest average tracking error and heading angle error. The path tracking control algorithms based on DDPG and PPO will have more overshoot in scenarios with relatively large path angles. In this case, both algorithms choose faster control actions to reduce the tracking error, and the output rudder angle control command changes greatly, resulting in overshoot. Among them, the overshoot phenomenon caused by the path tracking control algorithm based on PPO is the most serious, and the actual tracking performance is also the worst among the three algorithms. The reason for this phenomenon may be that the online learning reinforcement learning algorithm is more unstable than the offline reinforcement learning algorithm, and the training is more likely to fall into the local optimum. When the tracking error is small, it is easier to choose the action to reduce the tracking error as soon as possible. The path tracking algorithm based on DDPG is different from the other two algorithms based on randomness strategies. This algorithm is based on a deterministic strategy, and its ability to explore the environment is slightly weaker than the other two algorithms. Compared with the algorithm proposed in the present invention, no better strategy for balancing the tracking error and heading angle error has been found.

[0254] 3.3. Experimental results of search path tracking

[0255] The PID algorithm implemented in this experiment has shown poor control performance in the straight path scene tracking experiment, and it is also unable to complete the search path tracking and curve path tracking tasks in the experiment. Therefore, we will no longer compare the PID algorithm with the reinforcement learning-based path tracking control algorithm in the following search path scenarios and curve path scenarios.

[0256] The second set of experiments Fig.11 As shown in the figure, according to the actual unmanned boat mission scenario, a search path with a length of about 380m and composed of multiple continuous straight line paths (the coordinates of the path points are (0,0), (0,50), (15,50), (15,0), (30,0), (30,50), (45,50), (45,0), (60,0), (60,20)) is designed to carry out the tracking experiment. The starting position of the unmanned boat is (-5,-10).

[0257] Fig.12 and Fig.13The tracking error changes and heading angle error changes of several algorithms during the tracking process are shown. From the figure, we can see that in the actual tracking process of the unmanned boat path tracking algorithm proposed in the present invention, the tracking error and heading angle error are relatively small, the actual tracking path generated is also the smoothest, and it has excellent control performance. The path tracking algorithm based on PPO performs slightly worse, and the unmanned boat path tracking control algorithm based on DDPG performs the worst. Compared with the straight path tracking scene, which is very similar to the straight path scene used in the algorithm training process, there are more motion control situations that have not been encountered or have been encountered very little during the training process in the search path, which is a certain test of the stability and generalization of the control algorithm. From the experimental result figure, we can see that the path tracking algorithm based on DDPG performs the worst in the search path tracking control, with serious overshoot when turning, and even unable to adjust to steady state in many path sections, and the control effect is unstable. The control performance of the path tracking algorithm based on PPO and the path tracking algorithm proposed in the present invention is much better, but the control performance of the path tracking algorithm based on PPO is very unstable when dealing with continuous corners, and there is a phenomenon of excessive turning amplitude. At the same time, combined with the average error and root mean square error of the heading angle tracking error in Table 3, the average heading angle error and root mean square error of the path tracking algorithm based on PPO are the smallest, but the tracking error is very different from the algorithm result proposed in the present invention. Like the straight path, the path tracking algorithm based on PPO is more focused on optimizing the heading angle tracking error, and has not found a strategy that can better balance multiple optimization objectives like the path tracking algorithm proposed in the present invention.

[0258] Table 3

[0259]

[0260] 3.4 Curve Path Tracking Experimental Results

[0261] Fig.14 The test results of different path tracking algorithms in the curved path tracking scenario are shown. Fig.15 The curves of the heading angle error and tracking error during the path tracking process are given. The curved path scene sets the path center to (0,0), the radius to 30m, the path starting point to (38,38), and the path tracking direction to clockwise.

[0262] Although the three algorithms all use straight paths for training during the training process, and the VF guidance algorithm has slight differences in the calculation methods of straight paths and curved paths, the three algorithms can complete the path tracking test of the curved path well, and all obtain a relatively stable control result. As can be seen from the figure, the path tracking control algorithm proposed in the present invention also achieves the best control performance among the three algorithms on the curved path, with the smallest average tracking distance error and heading angle tracking error, the tracking steady-state error after stabilization can be around 0.38m, and the heading angle steady-state error can be around 0.1rad, which can better meet the actual curved path tracking task requirements. The tracking control algorithm based on PPO has the worst control performance, with the most dramatic changes in distance tracking error and angle tracking error, and the largest error after stabilization, which once again shows that the stability of the online algorithm PPO is slightly weaker than the other two algorithms. At the same time, although the three algorithms show good generalization in the curved path tracking scenario, there are still certain steady-state errors, and there is room for further improving the control performance of the algorithm.

[0263] In summary, through the comparison of experimental results in three scenarios, it can be seen that the actual control performance of the unmanned boat path tracking algorithm proposed in the present invention is relatively robust. It can quickly make appropriate adjustments in the face of path changes, has a short adjustment time, stable control effect, and has excellent control performance.

[0264] 3.5 Analysis of state space and reward function ablation experiment results

[0265] In order to demonstrate the effectiveness of the Markov decision process design in the unmanned boat path tracking algorithm proposed in this invention, this experiment conducted additional ablation experiments on the design of the state space and reward function. In the ablation experiment of the state space, the drift angle, the comparison value of the current and previous state, and the current rudder angle control command value in the state space were deleted to obtain three other state space designs. The same reward function design was used and the training was carried out according to the original training method. The maximum time step per round was 900, and the converged model was obtained after training for 1.2×106 time steps. Fig.16These are the curves of the changes in reward return functions and the average length of the rounds for several state space designs and the original state space. Among them, No_Drift represents the state space design after deleting the drift angle, No_Steer represents the state space design after deleting the rudder angle control command value in the state space, and No_Past represents the state space design after deleting the comparison value of the current state and the previous state. As can be seen from the figure, after removing the drift angle in the state space, the average length of the round gradually increases from about 350 to about 900 with the increase of training time, which is close to the time step limit of each round, and the average reward value is also much lower than other models, indicating that the algorithm cannot learn an effective path tracking control strategy at this time. This phenomenon shows that the drift angle is directly related to the control effect of the unmanned boat. After removing the drift angle, the design of the state space does not satisfy the Markov property, resulting in the algorithm being unable to train to an effective result. After removing the current rudder angle control command value from the state space, the average reward obtained by the algorithm during the training process is reduced, and the average round time also increases. After removing the comparison value of the state value at the current moment and the previous moment, the algorithm's ability to perceive whether it is controlling in the direction of reducing the error decreases, resulting in a decrease in the control stability of the algorithm and a decrease in the average return reward. Through the above comparison, it can be seen that the unmanned boat path tracking control algorithm model obtained by the state space training designed by the present invention can obtain the highest average reward benefit, the lowest average round length, and the most stable control effect, which illustrates the design effectiveness of the state space of the present invention.

[0266] In the ablation experiment of reward function design, we deleted the heading angle related rewards, path tracking error related rewards and control stability related rewards respectively, that is, set w1, w2, w3 to 0 respectively, and then trained according to the original training method. Finally, the algorithm model after training convergence was tested in the search path tracking scene respectively. The average and root mean square value results of the path and heading angle and tracking error, and the standard deviation of the rudder angle value change are shown in Table 4. Among them, No_Heading represents the reward function design after deleting the heading angle related rewards, No_Cross represents the reward function design after deleting the path tracking error related rewards, and No_Steer represents the reward function design after deleting the control stability related rewards. It can be seen from the table that after removing the heading angle related rewards, the algorithm tracking error is the smallest, and the standard deviation of the rudder angle command change is also the smallest, but the heading angle tracking performance is poor. Further observation of its actual tracking trajectory shows that the algorithm has been moving in a circle around a certain position near the starting point at this time, and the rudder angle command remains basically unchanged at the maximum value, and it is impossible to complete the actual path tracking task. The other two algorithm models trained after removing some rewards can complete the path tracking task and reach the end of the path. This shows that after the guidance algorithm is used to transform the path tracking problem, the heading angle provides the most basic guidance information for completing the path tracking task. The lack of rewards related to the heading angle will cause the algorithm to be unable to train a successful strategy. It can also be seen from the figure that after deleting the rewards related to the tracking error, although the heading angle error is reduced compared with the original algorithm, the tracking error increases more, causing the unmanned boat to converge to the path more slowly. Comparing the algorithm model that deletes the rewards related to control stability and the original algorithm, because the rudder angle control changes more drastically, the model has a tracking error and a heading angle error greater than the original algorithm, and the control stability is reduced. According to the above analysis, it can be seen that the reward function design proposed by the present invention can guide the unmanned boat training to obtain a control strategy that keeps both the tracking error and the heading angle error at a good level and the rudder angle control is also relatively stable.

[0267] Table 4

[0268]

[0269] 3.6 Analysis of experimental results of sea breeze interference scenario

[0270] In the real driving environment of unmanned boats, there is not only the interference of waves, but also the interference of sea breeze. Therefore, the path tracking control algorithm of unmanned boats needs to enhance its performance in the face of sea breeze interference. To this end, this experiment improves the state space on the basis of the original algorithm, and adds the variable representation related to sea breeze interference to the state space. In the actual environment, the wind direction at a location is fixed for a long period of time. But even if the wind direction of the sea breeze remains unchanged for a period of time, the performance of the unmanned boat under the interference of sea breeze will be different depending on its position relative to the sea breeze. Therefore, on the basis of the original state space, we added the current wind speed v w and the wind direction angle χ relative to the current heading angle of the unmanned boat w Indicates that the state space at this time can be expressed as follows:

[0271]

[0272] In the meteorological wind force scale, wind speed is divided into levels 0-12 according to the speed.

[0273] We take into account the physical property limitations of the unmanned boat used. In an environment with too strong winds, the unmanned boat is not suitable for sailing. Therefore, we only consider the situation when the wind speed is [0,8]m / s, which corresponds to the windless to strong wind levels in the meteorological wind force level table. During the algorithm training process in the simulation environment, according to the description of the setting of sea breeze interference in the prior art, the sea breeze interference parameters of the training environment are set to mean=5, var_gain=2, var_time=5, update_time=1, direction=270, and the training method in item x and the hyperparameters in section 4.3.1 are used for training to obtain the converged model parameters. Then, in the search path tracking scenario, the average wind speed is set to 0-8 and the wind direction is set to -45° to test the algorithm model obtained by training convergence. The test scenario covers the windless to strong wind scenarios defined by meteorology.

[0274] Fig.19 The actual tracking trajectory of the unmanned boat at wind speeds of 2, 4, 6, and 8 is given. The tracking error and heading angle error are compared with the actual tracking trajectory results of the unmanned boat. Fig.17 and Fig.18As shown in the figure. It can be seen from the results in the figure that after the unmanned boat path tracking control algorithm proposed by the present invention introduces the variable representation of the sea breeze into the state space representation, within the range of the average wind speed of 0m / s-6m / s, as the average wind speed increases, the tracking error and the heading angle error do not change significantly, showing a relatively excellent anti-interference ability. However, after the average wind speed exceeds 6m / s, the control performance of the algorithm has dropped significantly, and the tracking error and the heading angle error have increased to varying degrees. It can also be seen from the actual trajectory of the actual unmanned boat path tracking that when the wind speed is less than or equal to 6m / s, the unmanned boat basically does not have a large deviation, and basically moves forward on the planned path, and only some deviations occur at the turning point of the path facing the wind speed. However, when the average wind speed increases to 8m / s, the tracking trajectory of the unmanned boat has a large deviation in the straight forward part, and the deviation phenomenon at the corner is also more obvious. Considering the configuration of the unmanned boat propeller in the simulation environment, when the wind speed is 6m / s, the resistance caused by the sea breeze interference is approximately equal to the maximum propulsion force set by the unmanned boat propeller. When the wind speed is higher, the unmanned boat is limited by the maximum thrust of the propeller and can no longer resist greater resistance to maintain stability, so the tracking effect decreases. Theoretically, if the unmanned boat model that can provide greater thrust is replaced, a better anti-interference effect can be achieved. In summary, within the limit of the maximum thrust of the unmanned boat propeller, the unmanned boat path tracking control algorithm proposed in the present invention can enable the unmanned boat to have a stable path tracking control effect under the condition of sea breeze interference, and has good anti-interference ability.

[0275] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. Path tracking method of surface unmanned boat based on deep reinforcement learning, It is characterized in that The following steps are involved: S1: Convert the path tracking problem of the target unmanned boat into a Markov decision process and set the corresponding state space, action space and reward function; S2: Obtain the planned path of the target unmanned boat; In step S2, the planned path includes a straight line planned path and a curved line planned path; 1) The straight line planning path is expressed as: P s =<p 0 ,...,p i ,p i+1 ,...p T >,p i ∈R 2 ; Where: T represents the number of path points of the straight line planning path; p i =(x pi ,y pi ), 1<=i<=T represents the position coordinates of the i-th path point, and R represents an arbitrary real number; The curve planning path is expressed as: P c :(x-x c ) 2 +(y-y c ) 2 =R c ; Where: p c =(x c ,y c ) represents the coordinates of the center of the curve planning path; R c Indicates the path radius of the curve planning path; Define p t =(x t ,y t ) is the position coordinate of the target unmanned boat at time t; 1) For a straight line planning path, the azimuth and path tracking error are calculated using the following formula: Where: p Indicates the azimuth of the target unmanned boat under the straight line planning path; represents the path tracking error of the target unmanned boat under the straight line planning path; p i =(x pi ,y pi ) represents the position coordinates of the i-th path point; p i+1 =(x pi+1 ,y pi+1 ) represents the position coordinates of the i+1th path point; 2) For the curved planning path, the azimuth and path tracking error are calculated using the following formula: χ c =arctan(and t -and c ,x t -x c ); Where: c Indicates the azimuth of the target unmanned boat under the curve planning path; represents the path tracking error of the target unmanned boat under the curve planning path; p c =(x c ,y c ) represents the coordinates of the center of the curve planning path; R c represents the path radius; Indicates p c to the vector of p; S3: Calculate the reference heading angle according to the planned path of the target unmanned boat combined with the real-time posture information; then generate the current state value of the target unmanned boat based on the reference heading angle of the target unmanned boat, the real-time posture information and the environmental interference information; In step S3, the reference heading angle is calculated by the vector field algorithm according to the planned path of the target unmanned boat combined with the real-time posture information; the vector field algorithm calculates the reference heading angle by the following formula: 1) For straight line planning path: Where: d represents the reference heading angle under the straight line planning path; p represents the azimuth; χ Ms represents the maximum approach angle, set to (0,π / 2]; k s The control parameter representing the convergence speed of the vector field. The larger the value, the faster the heading angle changes. s represents the tracking error of the target unmanned boat to the straight path; 2) For curve planning path: Where: d represents the reference heading angle under the curve planning path; ρ d Indicates the path direction of the curve planning path, 1 is clockwise and -1 is counterclockwise; Mc represents the maximum approach angle, set to (0,π / 2]; k c The control parameter representing the convergence speed of the vector field. The larger the value, the faster the heading angle changes. c represents the azimuth of the target unmanned boat under the curve planning path; d c represents the tracking error of the target unmanned boat to the curved path; S4: Input the current state value of the target unmanned boat into the trained strategy model and output the optimal action value; The policy model is built based on the flexible actor-critic algorithm; During training, the reward value returned after the target unmanned boat executes the optimal action value is calculated based on the state space, action space and reward function to update the network parameters of the strategy model; In step S4, the policy model is trained by the following steps: S401: Initialize the network parameters of the strategy model and the starting position of the target unmanned boat; S402: Obtain the current path point information and real-time posture information of the planned path of the target unmanned boat and calculate the current reference heading angle, and then generate the current state value s of the target unmanned boat based on the current reference heading angle, real-time posture information and environmental interference information of the target unmanned boat. t ; S403: Current state value s of the target unmanned boat t Input to the strategy model and output the optimal action value a t ; Then control the target unmanned boat to execute the optimal action value a t , and returns the corresponding reward value r t+1 ; S404: Obtain the current path point information and real-time posture information of the planned path of the target unmanned boat again and calculate the current reference heading angle, and then generate the execution action value a based on the current reference heading angle, real-time posture information and environmental interference information of the target unmanned boat. t The state value after t+1 ; Then store and update the experience replay pool D←D∪(s t ,a t ,r t+1 ,s t+1 ); S405: Repeat steps S302 to S304 until the final path point is reached or the maximum number of steps in a round is reached; S406: extracting M samples from the experience replay pool D to form batch data for training and updating the network parameters of the strategy model; In step S406, the network parameters that need to be updated in the strategy model include the network parameters of the Q network and the π network and the temperature parameters; The objective function of the Q network is expressed as: Where; φ represents the network parameters of the Q network; represents the expectation of all strategies; Q φ represents the action-state value function; γ represents the discount factor; V φ represents the state value function; The objective function of the π network is expressed as: Where: θ represents the network parameters of the π network; D KL represents KL divergence (an indicator of similarity); π θ represents the policy network; Q φ represents the action-state value function; γ represents the discount factor; V φ represents the state value function; The objective function of the temperature parameter is expressed as: Where: α represents the temperature parameter; Expresses expectation; H 0 represents the initial policy entropy threshold; S5: Send the optimal action value to the target unmanned boat for execution; S6: Repeat steps S3 to S5 until the tracking control of the planned path is completed.

2. The method for tracking the path of an unmanned surface vehicle based on deep reinforcement learning as claimed in claim 1, Features: In step S1, the Markov decision process is expressed by the following formula: 1) The state space is represented as: Where: represents the difference between the current heading angle of the target unmanned boat and the reference heading angle; d represents the path tracking error; β represents the drift angle; δn d Indicates the rudder angle command size; It represents the difference between the current heading error of the target unmanned boat and the heading error at the previous moment; Δd represents the difference between the current path tracking error and the path tracking error at the previous moment; 2) The action space is represented as: Where: δn d Indicates the rudder angle command; 3) The reward function is expressed as: Where: represents the reward associated with the heading angle; r d represents the reward associated with the path tracking error; represents the reward related to control stability; k 1 ,k 2 ,k 3 Represents the constant term coefficient of the setting; σ δ represents the standard deviation of the 20 rudder angle command values ​​within the preset period; w 1 ,w 2 ,w 3 Indicates the weight coefficient of each reward item set.

3. The path tracking method for an unmanned surface vehicle based on deep reinforcement learning as claimed in claim 1, Features: In step S3, the real-time posture information includes the heading angle and the drift angle; the environmental interference information includes the wind speed and the wind direction.

4. The method for tracking the path of an unmanned surface vehicle based on deep reinforcement learning as claimed in claim 1, Features: The network parameter update process of the policy model is as follows: Update the network parameters of the Q network: Update the network parameters of the Pi network: Update temperature parameters: Update the network parameters of the target Q network: Where: Q represents the learning rate of the Q network; Denotes the network parameter θ of the Q network i The gradient of π represents the learning rate of the policy network; represents the gradient of the network parameter φ of the policy network; λ α represents the learning rate of the temperature parameter; represents the gradient of the temperature parameter α; θ i represents the network parameters of the Q network; Represents the network parameters of the target Q network.