A risk-adaptive navigation algorithm for unmanned boats based on distributed reinforcement learning

Abstract

The present invention provides an unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning, comprising the following steps: constructing an unmanned boat model and establishing a dynamic ocean current environment model; using distributed reinforcement learning to solve the problem of autonomous navigation of unmanned boats with perception uncertainty in a dynamic environment, distributed reinforcement learning can retain more information compared to reinforcement learning, can better learn strategies under uncertain conditions, and has a good performance improvement for sparse reward tasks; in view of environmental uncertainty, an implicit quantile network with adaptive risk tendency is further proposed to enhance the ability of risk assessment and risk-aware decision-making, reduce the occurrence rate of collisions, and ensure navigation safety. The present invention provides an unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning, which adopts a distributed reinforcement learning method to solve the sensor-based navigation problem of unmanned boats in uncertain environments.

Description

Technical Field

[0001] The present invention relates to the field of autonomous navigation of unmanned boats, and in particular to a risk-adaptive navigation algorithm for unmanned boats based on distributed reinforcement learning. Background Art

[0002] Unmanned boats are highly intelligent unmanned systems that integrate environmental perception, planning and decision-making, and motion control technologies. They have shown great potential and development prospects in many marine missions, such as port reconnaissance, maritime search, and island cruising. Safe and efficient autonomous navigation is an important technical prerequisite for performing missions. Modern navigation algorithms based on deep reinforcement learning show good efficiency and performance. However, there is currently little research on sensor-based navigation under the influence of unknown ocean currents and obstacles. Complex environments and perception uncertainties pose major challenges to completing safe and efficient navigation tasks.

[0003] USVs are very sensitive to flow fields, and it is very challenging to navigate safely and efficiently under the influence of flow fields. However, in previous studies, few people paid attention to the local path planning problem of USVs in unknown ocean current environments. Most studies did not consider ocean currents or were based on globally known ocean currents.

[0004] When reinforcement learning algorithms are applied to safety-critical robots such as unmanned boats and drones, they often perform tasks under uncertainty perception and dynamic environments. Traditional reinforcement learning methods are mostly global risk-neutral or risk-conservative strategies. How can robots enhance risk assessment and risk awareness and make decisions?

[0005] Therefore, it is necessary to provide an unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning to solve the above technical problems. Summary of the invention

[0006] The present invention provides an unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning, which solves the problem of how to enable robots to enhance risk assessment and risk awareness and make decisions, as traditional reinforcement learning methods are mostly global risk neutral or risk conservative strategies.

[0007] In order to solve the above technical problems, the present invention provides an unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning, comprising the following steps:

[0008] S1. Build an unmanned boat model and establish a dynamic ocean current environment model;

[0009] S2. Use distributed reinforcement learning to solve the problem of autonomous navigation of unmanned boats with perception uncertainty in dynamic environments. Distributed reinforcement learning can retain more information than reinforcement learning, can better learn strategies under uncertain conditions, and has a good performance improvement for sparse reward tasks.

[0010] S3. In view of environmental uncertainty, an implicit quantile network with adaptive risk tendency is further proposed to enhance the ability of risk assessment and risk-aware decision-making, reduce the occurrence rate of collision, and ensure navigation safety.

[0011] Preferably, the S1 comprises the following steps:

[0012] S11. Construct the kinematic model of the unmanned boat:

[0013] In the actual navigation process of the unmanned boat, since the unmanned boat moves along the sea level, its heave, pitch and roll movements can be ignored, and its motion problem is simplified from six degrees of freedom to three degrees of freedom, namely forward, drift and bow swing. The three-degree-of-freedom motion focuses on the changes in the bow angle θ and position (x, y). The three-degree-of-freedom motion equation is as follows:

[0014]

[0015] S12, simulate the ocean environment, requiring the robot to navigate under fluid disturbance and reach the target without colliding with any obstacles. The Rankine vortex model is used to simulate the ocean flow field, where it is assumed that there is a rigid body rotating in the vortex core with a radius of r0, and Γ is the circulation intensity of the vortex:

[0016]

[0017] The total velocity vector calculation method of the unmanned boat is:

[0018]

[0019] Preferably, the angular velocity of the vortex center in S12 is The linear velocity v at the edge of the vortex center edge =Ωr0, the flow velocity at a specific location is approximately calculated by superimposing the effect of nearby vortices.

[0020] Preferably, V(t) in S12 is the total speed of the unmanned boat, V C (X(t)) is the ocean current velocity at the position of the unmanned boat at time t, α is the proportional coefficient that controls the influence of the ocean current on the unmanned boat, V S (t) is the turning speed of the unmanned boat.

[0021] Preferably, S2 comprises the following steps:

[0022] S21. Define the state space:

[0023] The status information includes the position information X(t) of the unmanned boat, the steering speed V S (t), flow field information V at the current position of the unmanned boat C(X), obstacle information observed by the unmanned boat radar O LiDAR , target location information O goal

[0024]

[0025] S22. Define action space:

[0026] If the propulsion force and rudder angle of the unmanned boat dynamics parameters are used as algorithm input, it will increase the difficulty of model training. For this reason, the thrust of the USV is fixed.

[0027] The rudder angle is defined as a set of fine-grained discrete motion spaces containing 13 elements as follows:

[0028] A=[a1,a2,…,a7,…a 12 ,a 13 ]

[0029] Its value range is from -60° to 60°, with an increment of 10°. This design is more in line with the actual operation of USV;

[0030] S23. Design reward function:

[0031] After fully considering the various situations that the unmanned boat will encounter during the entire process, the following reward function is designed:

[0032] R=R collision +R distance +R time +R current +R goal .

[0033] Preferably, in S21, the sensor-based unmanned boat navigation task is defined as a partially observable Markov decision process (POMDP): POMDP can be defined as a tuple in and Representing the state, action and observation space, the UAV interacts with the environment in discrete time steps. At each time step t, it receives observations from the environment And according to its policy function π t (a t ∣o t )exist Execute an action on the t Transition to Generate a reward r t =R(s t ,a t ) and a new observation After policy π, the discounted sum of future rewards is expressed as a random variable is the discount factor, and the goal of standard RL is to make Z π The expected value of is maximized, that is, the behavior value function

[0034] Preferably, the S23 comprises the following steps:

[0035] S231, Collision Reward R collision Used to encourage the agent to avoid obstacles:

[0036]

[0037] Among them, distance obstacle_t is the shortest distance between the agent and the surrounding obstacles, γ1 and r collision is a constant, when distance obstacle_t When it is less than γ1, the agent will receive a negative reward -r collision ;

[0038] S232, time step reward R distance is the posture-related reward item under the time step, which alleviates the sparsity of the reward of the navigation task and encourages the unmanned boat to move towards the target point:

[0039]

[0040]

[0041] Indicates that the action selection of the unmanned boat at each time step is rewarded. If the angle of the current target point relative to the unmanned boat is less than 90°, it means that the unmanned boat is constantly approaching the target point, and a positive reward is given. On the contrary, if the angle of the target point relative to the unmanned boat is greater than 90°, it means that the unmanned boat is constantly moving away from the target point, and a negative penalty is given;

[0042] S233, Time Reward R time Used to encourage the agent to reach the target point in the shortest possible time:

[0043] R time =-λ3*R t

[0044] Among them, r t is the time taken by the unmanned boat from the start of the navigation mission to time t, and λ3 is a normal quantity;

[0045] S234, Ocean Current Reward R current Used to encourage movement along the direction of ocean currents, giving the agent the ability to take advantage of ocean currents:

[0046] Rcurrent=λ4*Similarity

[0047]

[0048] Similarity is used to reflect the similarity between the ocean current vector at the position of the unmanned boat at the previous moment and the ocean current vector at the current moment, and λ4 is a constant;

[0049] S235, Target Reward R goal Used to encourage the agent to shorten the distance to the goal point to get closer to the destination:

[0050]

[0051] where distance goal_t is the distance between the agent and the destination at time t, γ0,r goal ,andλ goal is a constant. When the agent approaches the destination, the reward value R goal is positive, otherwise it is negative. goal_t When it is less than γ0, it is considered that the agent has reached the destination and thus obtained the maximum reward r goal .

[0052] Preferably, S3 comprises the following steps:

[0053] S31. In order to dynamically respond to risk trends, we proposed an implicit quantile network (RTA-IQN) algorithm with risk trend adaptation based on the value distribution reinforcement learning algorithm IQN. RTA-IQN can adapt its overall risk trend by responding to changes in inherent uncertainty: when RTV increases and the current CVaR is large, the agent will choose a trajectory with a small CvaR to reduce uncertainty;

[0054] The structure of RTA-IQN mainly consists of three key parts: (1) IQN (2) Intrinsic uncertainty estimation (3) Uncertainty prediction based on EWAF:

[0055] S32, Implicit Quantile Network IQN:

[0056] In value-based reinforcement learning, Q(s,a) is defined as the expected reward value of a specific action in a specific state:

[0057] Q(s,a)=E π [G t ∣s t =s,a t =a]

[0058] For distributed reinforcement learning, the expected value Q(s,a) is replaced by the expected distribution Z(s,a):

[0059]

[0060] in Indicates uniform distribution, the state s' and action a' of the next time step are The IQN structure is mainly a neural network, with the input being the state x and a sample τ~U[0,1], and the output being the τ quantile of the value function distribution corresponding to different discrete actions

[0061] S33. Risk-sensitive strategies and risk measurement:

[0062] Define the value function related to risk propensity:

[0063]

[0064] If the β(·):[0,1]→[0,1] function is a unit mapping, then the Q function is the same as defined before, which is the expectation of the value function distribution. If the function is a convex function (or it is below the unit mapping on the image), then it is equivalent to adding a larger weight to the worse situation, which produces a risk-averse risk preference. If the function is a concave function (or it is above the unit mapping on the image), then it is equivalent to adding a larger weight to the better situation, which produces a risk-seeking risk preference. The greedy strategy under this value function can be defined as:

[0065]

[0066] In actual calculation, the strategy is obtained by sampling:

[0067]

[0068] Use gradient descent to optimize the following loss function:

[0069]

[0070] Among them, ρ(·) function is the loss function of component regression Q, and the TD error is:

[0071]

[0072] The first two items of the TD error are the targets, and the last item is the neural network to be optimized, that is, only the gradient of the last item is propagated;

[0073] S34: Changes in risk-sensitive strategies:

[0074] Considering the consistent risk measure - conditional value at risk CVaR as a distortion function, it is very convenient to generate risk trends. Specifically, by modifying Apply CVaR to IQN, where α is the CVaR value. When α is close to zero, we can get a risk-averse strategy, and when it is close to 1, we return to risk neutrality.

[0075] S35. Estimation of internal uncertainty:

[0076] The disturbances to which the agent is subjected, such as changes in ocean currents, changes in the number and location of obstacles, and noise, are called internal uncertainty. As the main source of risk, we can use a relatively dispersed return distribution to indicate the level of internal uncertainty. The upper tail conditional variance of the expected distribution is used to attenuate the exploration strategy. Therefore, the low tail conditional variance is used as the basis for estimating the intrinsic uncertainty of risk trend adaptation, which can be equivalently expressed by the right truncated variance (RTV):

[0077]

[0078] where τ i For the quantile level, the RTV is calculated based on the median rather than the mean;

[0079] S36, Exponentially Weighted Average Forecast (EWAF):

[0080] In our framework, in order to express CVaR as a function of RTV and allow risk trends to be projected in the choice of CVaR, we use EWAF to model CVaR.

[0081] Specifically, consider a class distribution with two classes By letting α = C1, CVaR is constrained to the range (0, 1) and can be adjusted by changing the classification weights: For each time step t, CVaR is calculated by the feedback indicator f = RTV t -RTV t-1 , update w with step size η i :w1=w1-ηf,w2=w2+ηf to adjust,

[0082] In order to prevent CVaR from approaching zero, when i=1,2, i Add ∑ to the denominator of and the numerator of C1 i exp(w i ) / b, the range of CVaR is For example, when b = 9, α∈(0.1,1);

[0083] S37. Build a simulation environment and perform model training:

[0084] The size of the simulation environment is set to 100m × 100m. The idea of ​​course-based training is adopted. According to the schedule shown in Table 1, the difficulty level in the course-based environment is gradually increased.

[0085] Compared with the related art, the unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning provided by the present invention has the following beneficial effects:

[0086] The present invention provides an unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning. The distributed reinforcement learning method is used to solve the sensor-based navigation problem of the unmanned boat in an uncertain environment. No prior information on ocean currents and obstacles is required. The unmanned boat, as an intelligent agent, can cope with environmental risks in the process of interaction with the environment and learn safe navigation strategies.

[0087] Compared with the global risk-neutral strategy or risk-conservative strategy, the algorithm RTA-IQN of the present invention enables the intelligent agent to have risk self-adaptation capability, which can change with different tasks or real-time feedback from the environment, and balance safety and exploration.

[0088] Compared with the traditional deep reinforcement learning method DQN and the baseline distributed reinforcement learning method IQN, it can be seen that the algorithm framework adopted in this article has improved the average reward value, safety and task return rate, and can complete the unmanned boat autonomous navigation task more safely and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A schematic structural diagram of a preferred embodiment of a risk-adaptive navigation algorithm for an unmanned boat based on distributed reinforcement learning provided by the present invention;

[0090] Figure 2 It is the structure diagram of RTA-IQN;

[0091] Figure 3 Examples of environments at different stages of the course. DETAILED DESCRIPTION

[0092] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0093] Please refer to Figure 1 , Figure 2 and Figure 3 ,in, Figure 1 A schematic structural diagram of a preferred embodiment of a risk-adaptive navigation algorithm for an unmanned boat based on distributed reinforcement learning provided by the present invention; Figure 2 It is the structure diagram of RTA-IQN; Figure 3 The following are examples of environments at different stages of the course. A risk-adaptive navigation algorithm for unmanned boats based on distributed reinforcement learning includes the following steps:

[0094] S1. Build an unmanned boat model and establish a dynamic ocean current environment model;

[0095] S2. Use distributed reinforcement learning to solve the problem of autonomous navigation of unmanned boats with perception uncertainty in dynamic environments. Distributed reinforcement learning can retain more information than reinforcement learning, can better learn strategies under uncertain conditions, and has a good performance improvement for sparse reward tasks.

[0096] S3. In view of environmental uncertainty, an implicit quantile network with adaptive risk tendency is further proposed to enhance the ability of risk assessment and risk-aware decision-making, reduce the occurrence rate of collision, and ensure navigation safety.

[0097] The S1 comprises the following steps:

[0098] S11. Construct the kinematic model of the unmanned boat:

[0099] In the actual navigation process of the unmanned boat, since the unmanned boat moves along the sea level, its heave, pitch and roll movements can be ignored, and its motion problem is simplified from six degrees of freedom to three degrees of freedom, namely forward, drift and bow swing. The three-degree-of-freedom motion focuses on the changes in the bow angle θ and position (x, y). The three-degree-of-freedom motion equation is as follows:

[0100]

[0101] S12, simulate the ocean environment, requiring the robot to navigate under fluid disturbance and reach the target without colliding with any obstacles. The Rankine vortex model is used to simulate the ocean flow field, where it is assumed that there is a rigid body rotating in the vortex core with a radius of r0, and Γ is the circulation intensity of the vortex:

[0102]

[0103] The total velocity vector calculation method of the unmanned boat is:

[0104]

[0105] The angular velocity of the vortex center in S12 The linear velocity v at the edge of the vortex center edge =Ωr0, the flow velocity at a specific location is approximately calculated by superimposing the effect of nearby vortices.

[0106] V(t) in S12 is the total speed of the unmanned boat, V C (X(t)) is the ocean current velocity at the position of the unmanned boat at time t, α is the proportional coefficient that controls the influence of the ocean current on the unmanned boat, V S (t) is the turning speed of the unmanned boat.

[0107] The S2 comprises the following steps:

[0108] S21. Define the state space:

[0109] The status information includes the position information X(t) of the unmanned boat, the steering speed V S (t), flow field information V at the current position of the unmanned boat C (X), obstacle information observed by the unmanned boat radar O LiDAR , target location information O goal

[0110]

[0111] S22. Define action space:

[0112] If the propulsion force and rudder angle of the unmanned boat dynamics parameters are used as algorithm input, it will increase the difficulty of model training. For this reason, the thrust of the USV is fixed.

[0113] The rudder angle is defined as a set of fine-grained discrete motion spaces containing 13 elements as follows:

[0114] A=[a1,a2,…,a7,…a 12 ,a 13 ]

[0115] Its value range is from -60° to 60°, with an increment of 10°. This design is more in line with the actual operation of USV;

[0116] S23. Design reward function:

[0117] After fully considering the various situations that the unmanned boat will encounter during the entire process, the following reward function is designed:

[0118] R=R collision +R distance +R time +R current +R goal .

[0119] In S21, the sensor-based unmanned boat navigation task is defined as a partially observable Markov decision process (POMDP): POMDP can be defined as a tuple in and O represent the state, action, and observation space. The UAV interacts with the environment in discrete time steps. At each time step t, it receives observations from the environment. And according to its policy function π t (a t ∣o t )exist Execute an action on the t Transition to Generate a reward r t =R(s t ,a t ) and a new observation After policy π, the discounted sum of future rewards is expressed as a random variable γ∈(0,1) is the discount factor. The goal of standard RL is to make Z π The expected value of is maximized, that is, the behavior value function

[0120] The S23 comprises the following steps:

[0121] S231, Collision Reward R collision Used to encourage the agent to avoid obstacles:

[0122]

[0123] Among them, distance obstacle_t is the shortest distance between the agent and the surrounding obstacles, γ1 and r collision is a constant, when distance obstacle_t When it is less than γ1, the agent will receive a negative reward -r collision ;

[0124] S232, time step reward R distance is the posture-related reward item under the time step, which alleviates the sparsity of the reward of the navigation task and encourages the unmanned boat to move towards the target point:

[0125]

[0126]

[0127] Indicates that the action selection of the unmanned boat at each time step is rewarded. If the angle of the current target point relative to the unmanned boat is less than 90°, it means that the unmanned boat is constantly approaching the target point, and a positive reward is given. On the contrary, if the angle of the target point relative to the unmanned boat is greater than 90°, it means that the unmanned boat is constantly moving away from the target point, and a negative penalty is given;

[0128] S233, Time Reward R time Used to encourage the agent to reach the target point in the shortest possible time:

[0129] R time =-λ3*r t

[0130] Among them, r tis the time taken by the unmanned boat from the start of the navigation mission to time t, and λ3 is a normal quantity;

[0131] S234, Ocean Current Reward R current Used to encourage movement along the direction of ocean currents, giving the agent the ability to take advantage of ocean currents:

[0132] R current =λ4*Similarity

[0133]

[0134] Similarity is used to reflect the similarity between the ocean current vector at the position of the unmanned boat at the previous moment and the ocean current vector at the current moment, and λ4 is a constant;

[0135] S235, Target Reward R goal Used to encourage the agent to shorten the distance to the goal point to get closer to the destination:

[0136]

[0137] where distance goal_t is the distance between the agent and the destination at time t, γ0,r goal ,andλ goal is a constant. When the agent approaches the destination, the reward value R goal is positive, otherwise it is negative. goal_t When it is less than γ0, it is considered that the agent has reached the destination and thus obtained the maximum reward r goal .

[0138] The S3 comprises the following steps:

[0139] S31. In order to dynamically respond to risk trends, we proposed an implicit quantile network (RTA-IQN) algorithm with risk trend adaptation based on the value distribution reinforcement learning algorithm IQN. RTA-IQN can adapt its overall risk trend by responding to changes in inherent uncertainty: when RTV increases and the current CVaR is large, the agent will choose a trajectory with a small CvaR to reduce uncertainty;

[0140] The structure of RTA-IQN mainly consists of three key parts: (1) IQN (2) Intrinsic uncertainty estimation (3) Uncertainty prediction based on EWAF:

[0141] S32, Implicit Quantile Network IQN:

[0142] In value-based reinforcement learning, Q(s,a) is defined as the expected reward value of a specific action in a specific state:

[0143] Q(s,a)=E π [G t ∣s t =s,a t =a]

[0144] For distributed reinforcement learning, the expected value Q(s,a) is replaced by the expected distribution Z(s,a):

[0145]

[0146] in Indicates uniform distribution, the state s' and action a' of the next time step are The IQN structure is mainly a neural network, with the input being the state x and a sample τ~U[0,1], and the output being the τ quantile of the value function distribution corresponding to different discrete actions

[0147] S33. Risk-sensitive strategies and risk measurement:

[0148] Define the value function related to risk propensity:

[0149]

[0150] If the β(·):[0,1]→[0,1] function is a unit mapping, then the Q function is the same as defined before, which is the expectation of the value function distribution. If the function is a convex function (or it is below the unit mapping on the image), then it is equivalent to adding a larger weight to the worse situation, which produces a risk-averse risk preference. If the function is a concave function (or it is above the unit mapping on the image), then it is equivalent to adding a larger weight to the better situation, which produces a risk-seeking risk preference. The greedy strategy under this value function can be defined as:

[0151]

[0152] In actual calculation, the strategy is obtained by sampling:

[0153]

[0154] Use gradient descent to optimize the following loss function:

[0155]

[0156] Among them, ρ(·) function is the loss function of component regression Q, and the TD error is:

[0157]

[0158] The first two items of the TD error are the targets, and the last item is the neural network to be optimized, that is, only the gradient of the last item is propagated;

[0159] S34: Changes in risk-sensitive strategies:

[0160] Considering the consistent risk measure - conditional value at risk CVaR as a distortion function, it is very convenient to generate risk trends. Specifically, by modifying Apply CVaR to IQN, where α is the CVaR value. When α is close to zero, we can get a risk-averse strategy, and when it is close to 1, we return to risk neutrality.

[0161] S35. Estimation of internal uncertainty:

[0162] The disturbances to which the agent is subjected, such as changes in ocean currents, changes in the number and location of obstacles, and noise, are called internal uncertainty. As the main source of risk, we can use a relatively dispersed return distribution to indicate the level of internal uncertainty. The upper tail conditional variance of the expected distribution is used to attenuate the exploration strategy. Therefore, the low tail conditional variance is used as the basis for estimating the intrinsic uncertainty of risk trend adaptation, which can be equivalently expressed by the right truncated variance (RTV):

[0163]

[0164] where τ i For the quantile level, the RTV is calculated based on the median rather than the mean;

[0165] S36, Exponentially Weighted Average Forecast (EWAF):

[0166] In our framework, in order to express CVaR as a function of RTV and allow risk trends to be projected in the choice of CVaR, we use EWAF to model CVaR.

[0167] Specifically, consider a class distribution with two classes By letting α = C1, CVaR is constrained to the range (0, 1) and can be adjusted by changing the classification weights: For each time step t, CVaR is calculated by the feedback indicator f = RTV t -RTV t-1 , update w with step size η i :w1=w1-ηf,w2=w2+nf to adjust,

[0168] In order to prevent CVaR from approaching zero, when i=1,2, i Add ∑ to the denominator of and the numerator of C1 iexp(w i ) / b, the range of CVaR is For example, when b = 9, α∈(0.1,1);

[0169] S37. Build a simulation environment and perform model training:

[0170] The size of the simulation environment is set to 100m × 100m. The idea of ​​course-based training is adopted. According to the schedule shown in Table 1, the difficulty level in the course-based environment is gradually increased.

[0171] When initializing the training segment, a vortex model with random position, rotation direction and intensity, as well as circular obstacles with random position and size will be generated in the training environment. The sampling range of the linear velocity of the vortex core edge is [8, 20] m / s, and the sampling range of the obstacle radius is [2, 8] m. The starting and target positions are randomly generated, and the distance between them is not less than a given threshold. The initial position and speed of the unmanned boat are randomly generated. When the unmanned boat collides, arrives at the navigation location, or has been trained for more than 1000 steps, a training cycle ends.

[0172] Table 1. Course-based training environment parameters

[0173]

[0174] The agent is modeled as a fully connected network with 3 hidden layers, each with 512 neurons. Except for the output layer, each fully connected layer is followed by a ReLU activation function. Adam is the optimizer, lr is the learning rate, and in each update step of D episodes, B batch samples of size E are extracted from the experience replay pool. The specific hyperparameter values ​​are shown in Table 2:

[0175] Table 2 Hyperparameter settings

[0176]

[0177] Compared with the traditional reinforcement learning method DQN and the baseline distributed algorithm IQN, the RTA-IQN method of the present invention has the following comparison results in terms of average reward per round, navigation success rate, collision rate, and navigation time:

[0178] Table 3 RTA-IQN comparison experimental results

[0179]

[0180]

[0181] Compared with the related art, the unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning provided by the present invention has the following beneficial effects:

[0182] The present invention provides an unmanned boat risk adaptive navigation algorithm based on distributed reinforcement learning. The distributed reinforcement learning method is used to solve the sensor-based navigation problem of the unmanned boat in an uncertain environment. No prior information on ocean currents and obstacles is required. The unmanned boat, as an intelligent agent, can cope with environmental risks in the process of interaction with the environment and learn safe navigation strategies.

[0183] Compared with the global risk-neutral strategy or risk-conservative strategy, the algorithm RTA-IQN of the present invention enables the intelligent agent to have risk self-adaptation capability, which can change with different tasks or real-time feedback from the environment, and balance safety and exploration.

[0184] Compared with the traditional deep reinforcement learning method DQN and the baseline distributed reinforcement learning method IQN, it can be seen that the algorithm framework adopted in this article has improved the average reward value, safety and task return rate, and can complete the unmanned boat autonomous navigation task more safely and efficiently.

[0185] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A risk-adaptive navigation algorithm for unmanned boats based on distributed reinforcement learning, characterized in that: The following steps are involved: S1. Build an unmanned boat model and establish a dynamic ocean current environment model; S2. Use distributed reinforcement learning to solve the problem of autonomous navigation of unmanned boats with perception uncertainty in dynamic environments. Distributed reinforcement learning can retain more information than reinforcement learning, can better learn strategies under uncertain conditions, and has a good performance improvement for sparse reward tasks. S3. In view of environmental uncertainty, an implicit quantile network with adaptive risk tendency is further proposed to enhance the ability of risk assessment and risk-aware decision-making, reduce the collision rate, and ensure navigation safety; The S1 comprises the following steps: S11. Construct the kinematic model of the unmanned boat: In the actual navigation process of the unmanned boat, since the unmanned boat moves along the sea level, its heave, pitch and roll movements can be ignored, and its motion problem is simplified from six degrees of freedom to three degrees of freedom, namely forward, drift and bow swing. The three-degree-of-freedom motion focuses on the changes in the bow angle θ and position (x, y). The three-degree-of-freedom motion equation is as follows: S12, simulate the ocean environment, requiring the robot to navigate under fluid disturbance and reach the target without colliding with any obstacles. The Rankine vortex model is used to simulate the ocean flow field, where it is assumed that there is a rigid body rotating in the vortex core with a radius of r0, and Γ is the circulation intensity of the vortex: v r ＝0， The total velocity vector calculation method of the unmanned boat is: The angular velocity of the vortex center in S12 The linear velocity v at the edge of the vortex center edge =Ωr0, the velocity at a specific location is approximated by superimposing the effect of nearby vortices; V(t) in S12 is the total speed of the unmanned boat, V C (X(t)) is the ocean current velocity at the position of the unmanned boat at time t, α is the proportional coefficient that controls the influence of the ocean current on the unmanned boat, V S (t) is the turning speed of the unmanned boat; The S2 comprises the following steps: S21. Define the state space: The status information includes the position information X(t) of the unmanned boat, the steering speed V S (t), flow field information V at the current position of the unmanned boat C (X), obstacle information observed by the unmanned boat radar O LiDAR , target location information O goal S22. Define action space: If the propulsion force and rudder angle of the unmanned boat dynamics parameters are used as algorithm input, it will increase the difficulty of model training. For this reason, the thrust of the USV is fixed. The rudder angle is defined as a set of fine-grained discrete motion spaces containing 13 elements as follows: <h2 style=";text-align:left;direction:ltr">A=[a1,a2,…,a7,…a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> ,a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> ] Its value range is from -60° to 60°, with an increment of 10°. This design is more in line with the actual operation of USV; S23. Design reward function: After fully considering the various situations that the unmanned boat will encounter during the entire process, the following reward function is designed: R=R collision +R distance +R time +R current +R goal ; In S21, the sensor-based unmanned boat navigation task is defined as a partially observable Markov decision process (POMDP): POMDP can be defined as a tuple in and Representing the state, action and observation space, the UAV interacts with the environment in discrete time steps. At each time step t, it receives observations from the environment And according to its policy function π t (a t ∣o t )exist Execute an action on the t Transition to Generate a reward r t =R(s t ,a t ) and a new observation After policy π, the discounted sum of future rewards is expressed as a random variable γ∈(0,1) is the discount factor. The goal of standard RL is to make Z π The expected value of is maximized, that is, the behavior value function The S23 comprises the following steps: S231, Collision Reward R collision Used to encourage the agent to avoid obstacles: Among them, distance obstacle_t is the shortest distance between the agent and the surrounding obstacles, γ1 and r collision is a constant, when distance obstacle_t When it is less than γ1, the agent will receive a negative reward -r collision ; S232, time step reward R distance is the posture-related reward item under the time step, which alleviates the sparsity of the reward of the navigation task and encourages the unmanned boat to move towards the target point: Indicates that the action selection of the unmanned boat at each time step is rewarded. If the angle of the current target point relative to the unmanned boat is less than 90°, it means that the unmanned boat is constantly approaching the target point, and a positive reward is given. On the contrary, if the angle of the target point relative to the unmanned boat is greater than 90°, it means that the unmanned boat is constantly moving away from the target point, and a negative penalty is given; S233, Time Reward R time Used to encourage the agent to reach the target point in the shortest possible time: R time =-λ3*r t Among them, r t is the time taken by the unmanned boat from the start of the navigation mission to time t, and λ3 is a normal quantity; S234, Ocean Current Reward R current Used to encourage movement along the direction of ocean currents, giving the agent the ability to take advantage of ocean currents: R current =λ4*Similarity Similarity is used to reflect the similarity between the ocean current vector at the position of the unmanned boat at the previous moment and the ocean current vector at the current moment, and λ4 is a constant; S235, Target Reward R goal Used to encourage the agent to shorten the distance to the goal point to get closer to the destination: where distance goal_t is the distance between the agent and the destination at time t, γ0,r goal ,andλ goal is a constant. When the agent approaches the destination, the reward value R goal is positive, otherwise it is negative. goal_t When it is less than γ0, it is considered that the agent has reached the destination and thus obtained the maximum reward r goal ; The S3 comprises the following steps: S31. In order to dynamically respond to risk trends, we proposed an implicit quantile network RTA-IQN algorithm with risk trend adaptation based on the value distribution reinforcement learning algorithm IQN. RTA-IQN can adapt its overall risk trend by responding to changes in inherent uncertainty: when RTV increases and the current CVaR is large, the agent will choose a trajectory with a small CvaR to reduce uncertainty. The structure of RTA-IQN mainly consists of three key parts: (1) IQN (2) Intrinsic uncertainty estimation (3) Uncertainty prediction based on EWAF: S32, Implicit Quantile Network IQN: In value-based reinforcement learning, Q(s,a) is defined as the expected reward value of a specific action in a specific state: Q(s,a)=E π [G t ∣s t =s,a t =a] For distributed reinforcement learning, the expected value Q(s,a) is replaced by the expected distribution Z(s,a): in Indicates uniform distribution, the state s' and action a' of the next time step are a′~π(·|s') is selected. The IQN structure is mainly a neural network, with the input being the state x and a sample τ~U[0,1], and the output being the τ quantile of the value function distribution corresponding to different discrete actions S33. Risk-sensitive strategies and risk measurement: Define the value function related to risk propensity: If the β(·):[0,1]→[0,1] function is a unit mapping, then the Q function is the same as defined before, which is the expectation of the value function distribution. If the function is a convex function (or it is below the unit mapping on the image), then it is equivalent to adding a larger weight to the worse situation, which produces a risk-averse risk preference. If the function is a concave function (or it is above the unit mapping on the image), then it is equivalent to adding a larger weight to the better situation, which produces a risk-seeking risk preference. The greedy strategy under this value function can be defined as: In actual calculation, the strategy is obtained by sampling: Use gradient descent to optimize the following loss function: Among them, ρ(·) function is the loss function of component regression Q, and the TD error is: The first two items of the TD error are the targets, and the last item is the neural network to be optimized, that is, only the gradient of the last item is propagated; S34: Changes in risk-sensitive strategies: Considering the consistent risk measure - conditional value at risk CVaR as a distortion function, it is very convenient to generate risk trends. Specifically, by modifying Apply CVaR to IQN, where α is the CVaR value. When α is close to zero, we can get a risk-averse strategy, and when it is close to 1, it returns to risk neutrality. S35. Estimation of internal uncertainty: The disturbances to which the agent is subjected, such as changes in ocean currents, changes in the number and location of obstacles, and noise, are called internal uncertainty. As the main source of risk, we can use a relatively dispersed return distribution to indicate the level of internal uncertainty. The upper tail conditional variance of the expected distribution is used to attenuate the exploration strategy. Therefore, the low tail conditional variance is used as the basis for estimating the intrinsic uncertainty of risk trend adaptation, which can be equivalently expressed by the right truncated variance (RTV): where τ i For the quantile level, the RTV is calculated based on the median rather than the mean; S36, Exponentially Weighted Average Forecast (EWAF): In our framework, in order to express CVaR as a function of RTV and allow risk trends to be projected in the choice of CVaR, we use EWAF to model CVaR. Specifically, consider a class distribution C with two classes i =exp(w i ) / ∑ i exp(w i ), By letting α = C1, CVaR is constrained to the range (0, 1) and can be adjusted by changing the classification weights: For each time step t, CVaR is calculated by the feedback indicator f = RTV t -RTV t-1 , update w with step size η i :w1=w1-ηf,w2=w2+ηf to adjust, In order to prevent CVaR from approaching zero, when i=1,2, i Add ∑ to the denominator of and the numerator of C1 i exp(w i ) / b, the range of CVaR is For example, when b = 9, α∈(0.1,1); S37. Build a simulation environment and perform model training: The size of the simulation environment is set to 100m × 100m. The idea of ​​course-based training is adopted. According to the schedule shown in Table 1, the difficulty level in the course-based environment is gradually increased.

Images