Path planning method for unmanned surface vehicle based on psychological expectation reinforcement learning

Abstract

The invention relates to the technical field of machine learning, in particular to an unmanned surface vehicle path planning method based on psychological expectation reinforcement learning. Comprising a multi-head D3QN module, an RND3QN module, a psychological expectation strategy module, a risk avoiding strategy module and a priority experience playback pool module, the multi-head D3QN module is used for respectively processing different types of external reward streams, and each reward stream independently uses different discount factors to carry out value iteration. An external reward flow is effectively processed through a multi-head D3QN module, a RND3QN module evaluates novelty and generates an integrated reward signal, a psychological expectation strategy module balances exploration and utilization, a risk avoiding strategy module filters dangerous actions, and a priority experience playback pool module optimizes training samples, so that the decision-making quality and efficiency of an intelligent agent are comprehensively improved; and a more reasonable decision can be made in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and particularly to a path planning method for an unmanned surface vehicle based on psychological expectation reinforcement learning. Background Art

[0002] Reinforcement learning methods often operate by maximizing the expected return of a certain policy, which often goes smoothly in some scenarios where the rewards are intuitive and dense. This is because the trial-and-error cost is low, and it is relatively easy to randomly select the correct action (for example, when using strategies such as epsilon-greedy), and feedback from external rewards can be received in a timely manner. In fact, this is quite practical in many scenarios. In engineering, many external rewards can be designed based on experience to assist the learning of the agent, and the agent also needs such an intuitive method to complete low-cost and efficient learning.

[0003] However, in some special scenarios, which imply danger, a wrong move will cause the agent to "die", clearing all the rewards it has accumulated and returning it to the initial state, thus making all previous efforts wasted; on the other hand, the rewards in them are very sparse, lacking artificial prior knowledge to arrange enough external rewards to guide the agent's learning. The feedback from external rewards is very lagging and difficult to discover. The agent even needs to sacrifice short-term benefits to obtain long-term rewards, which easily makes it fall into local optimal solutions. Although difficult, for today's reinforcement learning models (such as DQN, PPO), in the case of sufficient exploration, it can still be completed. The special temporal difference (TD-error) mechanism can control the degree of the agent's foresight through the discount factor γ, so as to help it predict long-term rewards, making it theoretically possible to establish the cognition of "suffering first and then enjoying sweetness" in the agent's mind.

[0004] But that is in the ideal situation of sufficient exploration. In the actual learning process, the cognitive ability of the agent will affect its exploration. Insufficient cognitive ability will hinder its exploration. Especially in the context of model-free online reinforcement learning, the agent especially relies on the experience obtained from exploration to update the model. If the exploration is not sufficient, there will be a lack of appropriate experience to update the value network. At this time, it becomes impossible to establish the correct cognition in the agent's mind, which is particularly obvious in dangerous environments.

[0005] In the prior art, the agent's reckless exploration in dangerous scenarios is likely to incur danger and thus all previous efforts are wasted. It falls into local optimality before establishing a long-term perspective, further preventing it from exploring long-term possibilities, and is not decisive enough in making decisions in familiar scenarios, delaying the exploration progress.

[0006] Based on this, the present invention provides a path planning method for an unmanned surface vehicle based on psychological expectation reinforcement learning to solve the above-mentioned technical problems. Summary of the Invention

[0007] The object of the present invention is to provide a path planning method for an unmanned surface vehicle based on psychological expectation reinforcement learning. The present invention effectively processes external reward streams through a multi-head D3QN module, the RND3QN module evaluates novelty and generates an integrated reward signal, the psychological expectation strategy module balances exploration and exploitation, the risk avoidance strategy module filters dangerous actions, and the prioritized experience replay pool module optimizes training samples, comprehensively improving the decision-making quality and efficiency of the intelligent agent, enabling it to make more reasonable decisions in complex environments.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention provides a path planning method for an unmanned surface vehicle based on psychological expectation reinforcement learning, including the following steps:

[0010] S1: Construct an experimental environment and a state space;

[0011] S2: Use the reinforcement learning system ERND to conduct simulation experiments in a rasterized ocean maze;

[0012] S3: According to the experimental results, use the reinforcement learning system ERND to plan a path for the unmanned surface vehicle;

[0013] The reinforcement learning system ERND includes a multi-head D3QN module, an RND3QN module, a psychological expectation strategy module, a risk avoidance strategy module, and a prioritized experience replay pool module, where:

[0014] The multi-head D3QN module: is used to process different types of external reward streams respectively, and each reward stream independently performs value iteration using different discount factors;

[0015] The RND3QN module: is used to integrate random network distillation (RND) and D3QN, and uses a separate D3QN network to fit the internal reward to capture long-distance novelty;

[0016] The psychological expectation strategy module: is used to preferentially select actions with high familiarity and high rewards or actions with low familiarity and high expectations according to the dynamically set psychological expectation threshold by the user and by calculating the Q value;

[0017] The risk avoidance strategy module: is used to separate the Q value of the dangerous stream in the ε-greedy strategy and mask actions below the preset danger threshold;

[0018] The prioritized experience replay pool module: is used for the storage and sampling of experiences, and samples according to the importance of experiences through importance sampling.

[0019] The multi-head D3QN module includes a reward stream separation unit, a discount factor setting unit, a value iteration calculation unit, and a reward stream integration unit, where:

[0020] The reward stream separation unit is responsible for separating different types of external reward streams, including the reward when reaching the end point, the penalty generated by the distance from the end point, and the death penalty when hitting an obstacle.

[0021] The discount factor setting unit is used to set an independent discount factor for each separated reward stream.

[0022] The value iteration calculation unit is used to perform value iteration calculations for each reward stream respectively according to the set discount factor.

[0023] After completing the value iteration calculations for each reward stream, the reward stream integration unit will superimpose multiple reward streams according to specific rules to obtain a comprehensive reward value for the action decision-making of the agent.

[0024] The RND3QN module includes an RND network unit, an internal reward calculation unit, an independent D3QN network unit, a reward integration unit, and a value iteration unit, where:

[0025] The RND network unit contains a randomly initialized fixed-parameter network and a learnable prediction network. By minimizing the MSE error between the output of the prediction network and the fixed-parameter network, the novelty of the state-action pair is measured.

[0026] The internal reward calculation unit is used to calculate the internal reward corresponding to each state-action pair according to the novelty measurement result output by the RND network unit.

[0027] The independent D3QN network unit is used to fit the internal reward, and learn and predict the internal reward through the mechanism of the deep Q network to capture long-distance novelty.

[0028] The reward integration unit is used to integrate the internal reward learned by the independent D3QN network unit with the external reward to form a comprehensive reward signal.

[0029] The value iteration unit is used to construct a value iteration process for the internal reward of each state, solve the problem that the basic RND cannot predict the potential novelty in the future, continuously update the state-action value estimation, and optimize the strategy of the agent.

[0030] The psychological expectation strategy module includes a psychological threshold setting unit, a Q-value calculation unit, a familiarity evaluation unit, an action priority determination unit, and an action selection and execution unit, where:

[0031] The psychological threshold setting unit is responsible for receiving user input and flexibly setting a dynamically changing psychological expectation threshold based on factors such as user needs, task characteristics, training rounds, agent exploration progress, and task difficulty coefficient;

[0032] The Q-value calculation unit is used to calculate the Q-value based on the state of the agent and the actions taken, using a specific algorithm to quantify the expected rewards of different actions in the current state;

[0033] The familiarity assessment unit is used to evaluate the familiarity of each action by analyzing the relevant data of the RND3QN module output results, and determine the frequency of occurrence of the action in past training and the agent's mastery of it;

[0034] The action priority determination unit is used to determine the action type that should be currently selected based on the calculated Q value, the assessed familiarity, and the set psychological expectation threshold, in accordance with the new action priority rule of "familiar high reward > unfamiliar high reward ≈ unfamiliar low reward > familiar low reward";

[0035] The action selection and execution unit is used to select the final action to be executed from high-familiarity and high-reward actions or low-familiarity and high-expectation actions based on the results of the action priority determination unit, and convey the instructions to the intelligent agent execution system.

[0036] The specific formula for calculating the joint Q value in the action priority determination unit is:

[0037] Q joint (s,a)=Q θ (s,a)·F w (s,a)+E·(1-F w (s,a))

[0038] Where Q θ (s,a) is the original Q value; F w (s,a) is familiarity; E is the psychological expectation threshold; when F w (s,a)→1:Q joint ≈Q θ (s,a), prioritize the use of known high-reward actions; when F w (s,a)→0:Q joint ≈E, exploration driven by psychological expectations;

[0039] Determine the priority rule type:

[0040] ifF w (s,a)>δandQ θ (s,a)≥E Familiar with high returns

[0041] ifF w(s,a) ≤ δ and Q θ (s,a) ≥ E Strange high return

[0042] if F w (s,a) ≤ δ and Q θ (s,a) < E Strange low return

[0043] if F w (s,a) > δ and Q θ (s,a) < E Familiar low return

[0044] Where δ is the familiarity threshold, 0.5.

[0045] The risk aversion strategy module includes a Q-value separation unit, a threshold setting unit, and an action masking unit, where:

[0046] The Q-value separation unit is used to accurately identify and separate the dangerous flow Q-value related to dangerous situations from the overall Q-value during the execution of the ε-greedy strategy;

[0047] The threshold setting unit is used to set a reasonable danger threshold according to the task scenario and risk tolerance as the key boundary for judging whether an action is safe;

[0048] The action masking unit is used to compare the separated dangerous flow Q-value with the preset danger threshold, and mask the actions below the threshold to prevent the agent from selecting these dangerous actions.

[0049] The prioritized experience replay pool module includes an experience storage unit, an importance evaluation unit, a sampling probability calculation unit, an experience sampling unit, and a bias correction unit, where:

[0050] The experience storage unit is responsible for storing the experience data generated by the agent during the interaction with the environment into the experience replay pool in an orderly manner;

[0051] The importance evaluation unit is used to calculate the probability of each experience being sampled according to the importance degree of the experience given by the importance evaluation unit;

[0052] The sampling probability calculation unit is used to sample from the experience replay pool according to the probability obtained by the sampling probability calculation unit, and select a certain number of experiences for the learning and training of the agent;

[0053] The experience sampling unit is used to sample from the experience replay pool according to the probability obtained by the sampling probability calculation unit, and select a certain number of experiences for the learning and training of the agent;

[0054] The bias correction unit is used to correct the bias during the training process by calculating and assigning corresponding weights to the sampled experiences.

[0055] The probability of each experience being sampled is calculated, and the specific formula is:

[0056]

[0057] In the formula, p t is the probability of the t-th experience being sampled; R t+1 is the immediate reward obtained at the current time; γ is the discount factor, which measures the present value of future rewards; is the maximum Q-value prediction of all possible actions for the next state S t+1 by the target network; q θ (S t , A t ) is the Q-value estimate of the current network for the state-action pair (S t , A t ); ω is a hyperparameter that determines the shape of the distribution.

[0058] The deviation in the training process is corrected by calculating and assigning corresponding weights to the sampled experiences, and the specific formula is:

[0059] The specific formula for the weight is:

[0060]

[0061] The corrected loss function:

[0062]

[0063] In the formula, N is the total number of samples in the experience replay pool; P i is the sampling probability of the i-th sample; β is the deviation correction intensity parameter; Q(s t , a t ; θ) is the Q-value prediction of the current network for the state-action pair; Q ′ (s t+1 , a; θ) is the maximum Q-value prediction of the target network for the next state; w i is the importance weight.

[0064] The present invention also proposes a path planning method for a surface unmanned boat based on ERND, including the following steps:

[0065] S1: Construct an experimental environment and a state space;

[0066] S2: Use the reinforcement learning system ERND to conduct simulation experiments in a rasterized ocean maze;

[0067] S3: According to the experimental results, use the reinforcement learning system ERND to plan a path for the surface unmanned boat.

[0068] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0069] 1. The present invention effectively processes the external reward stream through the multi-head D3QN module, the RND3QN module evaluates novelty and generates an integrated reward signal, the psychological expectation strategy module balances exploration and exploitation, the risk avoidance strategy module filters dangerous actions, and the prioritized experience replay pool module optimizes training samples, comprehensively improving the decision-making quality and efficiency of the agent, enabling it to make more reasonable decisions in complex environments.

[0070] 2. By constructing a realistic experimental environment, reasonably encoding the state space, combining model training with dynamic parameter adjustment and real-time path planning with emergency response, the present invention enables the unmanned surface vehicle to still maintain a high path planning success rate and low decision-making latency in complex marine environments, effectively improving the environmental adaptability and task execution reliability of the unmanned surface vehicle in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a system diagram of the path planning method for an unmanned surface vehicle based on psychological expectation reinforcement learning of the present invention.

[0072] Figure 2 It is a pseudo-code diagram of the psychological expectation strategy in the path planning method for an unmanned surface vehicle based on psychological expectation reinforcement learning of the present invention.

[0073] Figure 3 It is a flowchart of a path planning method for an unmanned surface vehicle based on ERND of the present invention.

[0074] Explanation of the reference numerals in the drawings:

[0075] 100. Multi-head D3QN module; 101. Reward stream separation unit; 102. Discount factor setting unit; 103. Value iteration calculation unit; 104. Reward stream integration unit; 200. RND3QN module; 201. RND network unit; 202. Internal reward calculation unit; 203. Independent D3QN network unit; 204. Reward integration unit; 205. Value iteration unit; 300. Psychological expectation strategy module; 301. Psychological threshold setting unit; 302. Q-value calculation unit; 303. Familiarity evaluation unit; 304. Action priority determination unit; 305. Action selection and execution unit; 400. Risk avoidance strategy module; 401. Q-value separation unit; 402. Threshold setting unit; 403. Action masking unit. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0076] Next, in combination with the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0077] Embodiment 1:

[0078] As Figure 1 - Figure 2 shown, this embodiment provides a path planning method for a surface unmanned boat based on psychological expectation reinforcement learning. The reinforcement learning system ERND includes a multi-head D3QN module 100, an RND3QN module 200, a psychological expectation strategy module 300, a risk avoidance strategy module 400, and a prioritized experience replay pool module 500, where: The multi-head D3QN module 100: is used to process different types of external reward streams respectively, and each reward stream independently uses different discount factors for value iteration; The RND3QN module 200: is used to integrate Random Network Distillation (RND) and D3QN, and uses a separate D3QN network to fit the internal reward to capture long-distance novelty; The psychological expectation strategy module 300: is used to preferentially select high-familiarity and high-return actions or low-familiarity and high-expectation actions according to the dynamically set psychological expectation threshold by the user and by calculating the Q value; The risk avoidance strategy module 400: is used to separate the dangerous flow Q value in the ε-greedy strategy and mask the actions below the preset danger threshold; The prioritized experience replay pool module 500: is used for the storage and sampling of experiences, and samples according to the importance of the experiences by importance sampling.

[0079] Among them, it should be noted that the multi-head D3QN module 100 processes different types of external reward streams, the RND3QN module 200 evaluates state novelty and generates internal rewards, the psychological expectation strategy module 300 dynamically balances exploration and exploitation, the risk avoidance strategy module 400 filters dangerous actions in real time, and the prioritized experience replay pool module 500 optimizes the selection of training samples.

[0080] In this embodiment, it should also be noted that the RND3QN module 200 includes an RND network unit 201, an internal reward calculation unit 202, an independent D3QN network unit 203, a reward integration unit 204, and a value iteration unit 205, where: The RND network unit 201 includes a randomly initialized fixed-parameter network and a learnable prediction network. By minimizing the MSE error between the output of the prediction network and the fixed-parameter network, the novelty of the state-action pair is measured; the internal reward calculation unit 202 is used to calculate the internal reward corresponding to each state-action pair according to the novelty measurement result output by the RND network unit 201; the independent D3QN network unit 203 is used to fit the internal reward, and learn and predict the internal reward through the mechanism of the deep Q network to capture long-distance novelty; the reward integration unit 204 is used to integrate the internal reward learned by the independent D3QN network unit with the external reward to form a comprehensive reward signal; the value iteration unit 205 is used to construct a value iteration process for the internal reward of each state, solve the problem that the basic RND cannot predict the potential novelty in the future, continuously update the state-action value estimation, and optimize the strategy of the agent.

[0081] Among them, it should be noted that the RND network unit 201 generates a state novelty index, the internal reward calculation unit 202 quantifies it as an immediate intrinsic reward, the independent D3QN network unit 203 learns the temporal dependence of the reward through the Bellman equation, the reward integration unit 204 fuses the internal and external reward signals, and the value iteration unit 205 continuously optimizes the value estimation.

[0082] Furthermore, it should be noted that the MSE error between the prediction network f and is: Internal reward value iteration:

[0083] In this embodiment, it should also be noted that the psychological expectation strategy module 300 includes a psychological threshold setting unit 301, a Q-value calculation unit 302, a familiarity evaluation unit 303, an action priority determination unit 304, and an action selection and execution unit 305, where: The psychological threshold setting unit 301 is responsible for receiving user input and flexibly setting a dynamically changing psychological expectation threshold based on factors such as user requirements, task characteristics, number of training rounds, exploration progress of the agent, and task difficulty coefficient; The Q-value calculation unit 302 is used to calculate the Q-value using a specific algorithm according to the state of the agent and the actions taken, so as to quantify the expected return of different actions in the current state; The familiarity evaluation unit 303 is used to evaluate the familiarity of each action by analyzing the relevant data of the output result of the RND3QN module, and judge the frequency of occurrence of the action in past training and the degree of mastery of the agent; The action priority determination unit 304 is used to determine the type of action that should be preferentially selected currently according to the calculated Q-value, the evaluated familiarity, and the set psychological expectation threshold, according to the new action priority rule "familiar high return > unfamiliar high return ≈ unfamiliar low return > familiar low return"; The specific formula for jointly calculating the Q-value in the action priority determination unit 304 is:

[0084] Q joint (s,a) = Q θ (s,a)·F w (s,a) + E·(1 - F w (s,a))

[0085] In the formula, Q θ (s,a) is the original Q-value; F w (s,a) is the familiarity; E is the psychological expectation threshold; When F w (s,a) → 1 high familiarity: Q joint ≈ Q θ (s,a), preferentially utilize known high-return actions; When F w (s,a) → 0 low familiarity: Q joint ≈ E, drive exploration with psychological expectation;

[0086] Determine the type of priority rule:

[0087] if F w (s,a) > δ and Q<Q θ (s,a) ≥ E familiar high return

[0088] if F w (s,a) ≤ δ and Q θ (s,a) ≥ E unfamiliar high return

[0089] if F w (s,a) ≤ δ and Q θ(s,a) < E unfamiliar low return

[0090] ifF w (s,a) > δ and Q θ (s,a) < E familiar low return

[0091] In the formula, δ is the familiarity threshold, 0.5. The action selection and execution unit 305 is used to select the final executed action from high-familiarity high-return actions or low-familiarity high-expected actions based on the result of the action priority determination unit 304, and convey the instruction to the agent execution system.

[0092] Among them, it should be noted that the psychological threshold setting unit 301 dynamically adjusts the expected goal. The Q-value calculation unit 302 and the familiarity evaluation unit 303 respectively quantify the expected return of the action and the historical exploration degree. The action priority determination unit 304 screens the action types based on the combined Q-value formula and the priority rule: familiar high return > unfamiliar high return ≈ unfamiliar low return > familiar low return.

[0093] Furthermore, it should be noted that in the Q-value calculation unit 302, the action decision and value estimation in the update process are carried out through the following:

[0094]

[0095] In the formula, A t is the action actually taken by the agent at time step t; is the action that maximizes the Q-value in the current Q-network q θ for the state S t+1 ; R t+1 is the reward obtained by the agent at time step t + 1; q θ (S t , A t ) is the Q-value estimation of the current Q-network for the state-action pair (S t , A t ); L(θ) is the loss function;

[0096] The expected return Q(s t , a t ) is decomposed into the state value V(s t ) and the advantage function A(s t , a t ):

[0097]

[0098] In the formula, q θ (s,a) is the Q-value estimation of the current Q-network for the state-action pair (s,a), v η(f(s)) is the state value function, which is calculated by the function v with parameter η according to the feature representation f(s) of the state s. It represents the value of the agent in the state s itself without considering the specific action; is the advantage function; is the sum of the advantage function values of possible actions in state s.

[0099] In this embodiment, it should also be noted that the risk avoidance strategy module 400 includes a Q-value separation unit 401, a threshold setting unit 402, and an action masking unit 403, where: the Q-value separation unit 401 is used to accurately identify and separate the dangerous flow Q-value related to the dangerous situation from the overall Q-value during the execution of the ε-greedy strategy; the threshold setting unit 402 is used to set a reasonable danger threshold according to the task scenario and risk tolerance as the key boundary for judging whether an action is safe; the action masking unit 403 is used to compare the separated dangerous flow Q-value with the preset danger threshold and mask the actions below the threshold to prevent the agent from selecting these dangerous actions.

[0100] Among them, it should be noted that the Q-value separation unit 401 extracts the dangerous flow Q-value, the threshold setting unit 402 dynamically configures the safety boundary, and the action masking unit 403 forcibly filters high-risk actions according to the threshold.

[0101] In this embodiment, it should also be noted that the prioritized experience replay pool module 500 includes an experience storage unit 501, an importance evaluation unit 502, a sampling probability calculation unit 503, an experience sampling unit 504, and a deviation correction unit 505, where: the experience storage unit 501 is responsible for storing the experience data generated by the agent during the interaction with the environment into the experience replay pool in an orderly manner; the importance evaluation unit 502 is used to calculate the probability of each experience being sampled according to the importance degree of the experience given by the importance evaluation unit 501; the sampling probability calculation unit 503 is used to sample from the experience replay pool according to the probability obtained by the sampling probability calculation unit and select a certain number of experiences for the learning and training of the agent; the formula for calculating the probability of each experience being sampled is specifically:

[0102]

[0103] In the formula, p t is the probability of the t-th experience being sampled; R t+1 is the immediate reward obtained at the current time; γ is the discount factor, which measures the present value of future rewards; is the maximum Q-value prediction of the target network for the next state S t+1 for possible actions; q θ (S t ,A t) is the Q-value estimation of the current network for the state-action pair (S t , A t ); ω is a hyperparameter that determines the shape of the distribution. The experience sampling unit 504 is used to sample from the experience replay pool according to the probability calculated by the sampling probability calculation unit, and select a certain number of experiences for the learning and training of the agent; the bias correction unit 505 is used to correct the bias in the training process by calculating and assigning corresponding weights to the sampled experiences. The specific formula is:

[0104] The specific formula for the weight is:

[0105]

[0106] The corrected loss function:

[0107]

[0108] In the formula, N is the total number of samples in the experience replay pool; P i is the sampling probability of the i-th sample; β is the bias correction intensity parameter; Q(s t , a t ; θ) is the Q-value prediction of the current network for the state-action pair; Q ′ (s t+1 , a; θ) is the maximum Q-value prediction of the target network for the next state; w i is the importance weight.

[0109] Among them, it should be noted that the experience storage unit 501 accumulates interaction data, the importance evaluation unit 502 and the sampling probability calculation unit 503 dynamically allocate sampling priorities based on the TD error, the experience sampling unit 504 performs non-uniform sampling, and the bias correction unit 505 corrects the gradient update through the importance weight.

[0110] Example 2:

[0111] As Figure 3 shown, in this embodiment, a path planning method for an unmanned surface vehicle based on ERND specifically includes the following steps:

[0112] S1: Experimental environment construction and state encoding

[0113] S1.1: Construct a rasterized ocean environment map:

[0114] Design a rasterized maze of a certain size as the environment map, in which static obstacles composed of reefs, islands and land and dynamic obstacles composed of other agents are set to form a multi-agent environment;

[0115] S1.2: State space encoding:

[0116] After encoding the 5×5 area centered on the agent itself, combining its own coordinates and the coordinates of the end point, and connecting them as a tensor for state input, and each agent only has a field of view with a certain radius around itself, a partially observable Markov environment is formed;

[0117] S2: Model Training and Validation

[0118] S2.1: Training Configuration:

[0119] Multi-threaded asynchronous update: The action execution thread and the model training thread are synchronized through a circular buffer;

[0120] Success criterion: The task success rate is ≥90% for 10 consecutive times. The task success is defined as: reaching the end point and the number of collisions = 0;

[0121] S2.2: Dynamic Parameter Tuning:

[0122] The psychological expectation threshold linearly increases from the minimum task reward to 0.8max;

[0123] The exploration rate ε decays from 0.5 to 0.01;

[0124] S3: Real-time Path Planning

[0125] S3.1: Online Decision-making:

[0126] Action selection is performed every 100 ms;

[0127] When a sudden obstacle is detected, the emergency braking protocol of the risk avoidance strategy module 400 is triggered;

[0128] S3.2: Performance Metrics:

[0129] When the wave height ≥2m, the path planning success rate ≥85%;

[0130] The average decision-making delay ≤50ms.

[0131] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0132] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A path planning method for a surface unmanned boat based on psychologically expected reinforcement learning, characterized in that It includes the following steps: S1: Construct an experimental environment and a state space; S2: Use the reinforcement learning system ERND to conduct simulation experiments in a rasterized ocean maze; S3: According to the experimental results, use the reinforcement learning system ERND to plan a path for the surface unmanned boat; The reinforcement learning system ERND includes a multi-head D3QN module (100), an RND3QN module (200), a psychological expectation strategy module (300), a risk avoidance strategy module (400), and a prioritized experience replay pool module (500), where: The multi-head D3QN module (100): is used to process different types of external reward streams respectively, and each reward stream independently uses a different discount factor for value iteration; The RND3QN module (200): is used to integrate random network distillation RND and D3QN, and use a separate D3QN network to fit the internal reward to capture long-distance novelty; The psychological expectation strategy module (300): is used to preferentially select actions with high familiarity and high rewards or actions with low familiarity and high expectations according to the dynamically set psychological expectation threshold by the user and by calculating the Q value; The risk avoidance strategy module (400): is used to separate the Q value of the dangerous stream in the ε-greedy strategy and mask actions below the preset danger threshold; The prioritized experience replay pool module (500): is used for the storage and sampling of experiences, and sampling is carried out according to the importance of experiences by importance sampling.

2. The method for path planning of an unmanned surface vehicle based on psychologically expected reinforcement learning according to claim 1, wherein The multi-head D3QN module (100) includes a reward stream separation unit (101), a discount factor setting unit (102), a value iteration calculation unit (103), and a reward stream integration unit (104), where: The reward stream separation unit (101) is responsible for separating different types of external reward streams, including the reward when reaching the end point, the penalty generated by the distance from the end point, and the reward stream of the death penalty when hitting an obstacle; The discount factor setting unit (102) is used to set an independent discount factor for each separated reward stream; The value iteration calculation unit (103) is used to carry out value iteration calculations for each reward stream respectively according to the set discount factor; The reward stream integration unit (104) is used to stack multiple reward streams according to specific rules after completing the value iteration calculations of each reward stream, so as to obtain a comprehensive reward value for the action decision-making of the intelligent agent.

3. The path planning method for an unmanned surface vehicle based on psychological expectation reinforcement learning according to claim 1, characterized in that The RND3QN module (200) includes an RND network unit (201), an internal reward calculation unit (202), an independent D3QN network unit (203), a reward integration unit (204), and a value iteration unit (205), where: The RND network unit (201) contains a randomly initialized fixed-parameter network and a learnable prediction network, and measures the novelty of the state-action pair by minimizing the MSE error between the output of the prediction network and the fixed-parameter network; The internal reward calculation unit (202) is used to calculate the internal reward corresponding to each state-action pair according to the novelty measurement result output by the RND network unit (201); The independent D3QN network unit (203) is used to fit the internal reward, learn and predict the internal reward through the mechanism of the deep Q network to capture long-distance novelty; The reward integration unit (204) is used to integrate the internal reward learned by the independent D3QN network unit with the external reward to form a comprehensive reward signal; The value iteration unit (205) is used to construct a value iteration process for the internal reward of each state, solve the problem that the basic RND cannot predict the potential novelty in the future, continuously update the state-action value estimation, and optimize the agent's strategy.

4. The method for path planning of an unmanned surface vehicle based on psychological expectation reinforcement learning according to claim 3, wherein The psychological expectation strategy module (300) includes a psychological threshold setting unit (301), a Q value calculation unit (302), a familiarity evaluation unit (303), an action priority determination unit (304), and an action selection and execution unit (305), where: The psychological threshold setting unit (301) is responsible for receiving user input and flexibly setting a dynamically changing psychological expectation threshold based on factors such as user requirements, task characteristics, training rounds, the agent's exploration progress, and the task difficulty coefficient; The Q value calculation unit (302) is used to calculate the Q value according to the state of the agent and the actions taken, and use a specific algorithm to quantify the expected return of different actions in the current state; The familiarity evaluation unit (303) is used to evaluate the familiarity of each action by analyzing the relevant data of the output result of the RND3QN module, and judge the frequency of occurrence of the action in past training and the agent's mastery of it; The action priority determination unit (304) is used to determine the type of action to be preferentially selected currently according to the calculated Q value, the evaluated familiarity, and the set psychological expectation threshold, in accordance with the new action priority rule "high familiarity and high return > unfamiliar high return ≈ unfamiliar low return > familiar low return"; The action selection and execution unit (305) is used to select the final action to be executed from high familiarity and high return actions or low familiarity and high expected actions based on the result of the action priority determination unit, and convey the instruction to the agent execution system.

5. The method for path planning of an unmanned surface vehicle based on psychologically expected reinforcement learning according to claim 4, characterized in that The specific formula for jointly calculating the Q value in the action priority determination unit (304) is: Q joint (s,a) = Q θ (s,a)·F w (s,a) + E·(1 - F w (s,a)) where Q θ (s,a) is the original Q value; F w (s,a) is the familiarity; E is the psychological expectation threshold; when F w (s,a) → 1 (high familiarity): Q joint ≈ Q θ (s,a), giving priority to using known high-return actions; when F w (s,a) → 0 (low familiarity): Q joint ≈ E, driving exploration with psychological expectations; Determine the priority rule type: ifF w (s,a)>δ and Q θ (s,a)≥E familiar with high returns ifF w (s,a) ≤ δ and Q θ (s,a) ≥ E high return ifF w (s,a) ≤ δ and Q θ (s,a) < E strange low return ifF w (s,a)>δ and Q θ (s,a) < E Familiar with low returns [[ID= ​ ​ ​ 6. The method for path planning of an unmanned surface vehicle based on psychological expectation reinforcement learning according to claim 1, wherein ​ The experience storage unit (501) is responsible for storing the experience data generated by the agent during the interaction with the environment into the experience replay pool in an orderly manner; The importance evaluation unit (502) is used to calculate the sampling probability of each experience according to the importance degree of the experience given by the importance evaluation unit (501); The sampling probability calculation unit (503) is used to sample from the experience replay pool according to the probability obtained by the sampling probability calculation unit, and select a certain number of experiences for the learning and training of the agent; The experience sampling unit (504) is used to sample from the experience replay pool according to the probability obtained by the sampling probability calculation unit, and select a certain number of experiences for the learning and training of the agent; The deviation correction unit (505) is used to correct the deviation in the training process by calculating and assigning corresponding weights to the sampled experiences.

7. The method for path planning of an unmanned surface vehicle based on psychological expectation reinforcement learning according to claim 6, characterized in that, The calculation of the sampling probability of each experience, the specific formula is: where p t is the probability that the t-th experience is sampled; R t+1 is the immediate reward obtained at the current time; γ is the discount factor, which measures the present value of future rewards; is the maximum Q-value prediction of all possible actions of the target network for the next state S t+1 ; q θ (S t , A t ) is the Q-value estimate of the current network for the state-action pair (S t , A t ); ω is a hyperparameter that determines the shape of the distribution.

8. The method for path planning of an unmanned surface vehicle based on psychologically expected reinforcement learning according to claim 7, wherein The correction of the deviation in the training process by calculating and assigning corresponding weights to the sampled experiences, the specific formula is: The specific formula for the weight is: The corrected loss function: Where N is the total number of samples in the experience replay pool; P i is the sampling probability of the i-th sample; β is the bias correction intensity parameter; Q(s t ,a t ; θ) is the Q-value prediction of the current network for the state-action pair; Q ′ (s t+1 ,a; θ) is the maximum Q-value prediction of the target network for the next state; w i is the importance weight.

Images