Reinforced learning unmanned ship path control method for double-track regulation and control random network distillation

Abstract

The invention discloses a reinforcement learning unmanned ship path control method based on double-track regulation and control random network distillation. The method comprises the operation steps that an unmanned ship builds a path tracking simulation environment and a kinetic model; the unmanned ship builds a core algorithm flexible action evaluation algorithm framework; the unmanned ship deploys a priority experience playback pool based on quality and success guidance; an uncertainty perception and risk perception mechanism is introduced into the unmanned ship; the unmanned ship builds a success rate-based reward attenuation and cold start module, and the unmanned ship calculates a total reward and designs a reward softening mechanism to smooth the total reward; the unmanned ship imports hyper-parameters of all the modules, starts training circulation in a simulation environment, and dynamically adjusts exploration intensity and the like; according to the method, uncertainty and risk indexes are introduced, the exploration intensity of the intelligent agent is controlled, the intelligent agent is prevented from making dangerous actions, and the robustness is improved; a priority experience playback pool based on quality and success guidance is introduced, high-quality samples are better played back, and strategy convergence is accelerated.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of automatic path control of unmanned ships, and relates to a reinforcement learning unmanned ship path control method based on double-track regulation and control of random network distillation. BACKGROUND

[0002] With the increase of offshore development and marine traffic, unmanned ships are increasingly used in tasks such as environmental monitoring, target tracking and inspection cruising. Stable and reliable path control is the key to ensuring task completion. However, in complex sea conditions and narrow channels coexisting with multiple source disturbances such as wind, waves and currents, and shoreline and obstacle constraints, if the path control of the unmanned ship is not appropriate, it is easy to cause problems such as border crossing, collision or excessive path deviation.

[0003] Currently, two types of path control technologies are mainly used in engineering. One is the traditional control method based on the model, such as proportional-integral-derivative control, line-of-sight guidance combined with rudder angle control, and path constraint method based on model predictive control. This kind of method usually relies on a simplified dynamic model and fixed parameters, and works well in still water or weak disturbance environment, but in strong wind and wave, rudder lag, propeller saturation and other nonlinear working conditions, the control performance is obviously decreased, and it is difficult to timely consider multiple goals such as path tracking accuracy, obstacle avoidance constraint and safety speed limit, and it is easy to have problems such as large parameter adjustment workload and poor adaptability.

[0004] In recent years, the unmanned ship path control method based on deep reinforcement learning has appeared. By modeling the interaction between the unmanned ship and the simulation environment as a Markov decision process, the rudder angle and speed instructions are output by the policy network, and the external rewards related to the lateral deviation, heading deviation and collision penalty are constructed to guide policy learning. However, the existing reinforcement learning methods have the following shortcomings: first, single external reward weighted summation is mostly used, and the reward design relies on experience. In the scene where the path deviation is only explicitly fed back at the end point or a few key positions, the agent often needs a large number of trial and error to obtain effective reward signals, which is low in training efficiency and long in convergence time; second, uniform sampling or simple priority experience replay is mostly used in experience replay, which is difficult to highlight the key fragments that contribute more to task completion, and lacks fine distinction of different sample quality and representation; third, random noise is usually added to the action or strategy in the exploration mechanism, which assumes that different states and actions are equivalent in safety, cannot dynamically adjust the exploration intensity according to the uncertainty of value estimation, and lacks effective constraints on high-risk working conditions such as high-speed driving, approaching the shoreline or obstacles, which easily leads to frequent deviation from the track during the training process and even collision, overspeed and other unsafe behaviors.

[0005] Therefore, the existing unmanned ship path control technology is still difficult to simultaneously consider training efficiency, strategy convergence stability and navigation safety controllability under complex sea conditions and multiple constraint conditions, and an exploration efficiency and safety constraint path control method in a complex disturbance environment is urgently needed. SUMMARY

[0006] The purpose of the application is to provide a double-track regulation random network distillation reinforcement learning unmanned ship path control method which utilizes reinforcement learning and random network distillation for path control of unmanned ships, and provides uncertainty signals and danger signals for unmanned ships through uncertainty perception, so as to improve the adaptability and performance of the algorithm in complex environments.

[0007] Technical scheme: The double-track regulation random network distillation reinforcement learning unmanned ship path control method comprises the following steps: (1) An unmanned ship builds a path tracking simulation environment and a dynamics model; (2) The unmanned ship builds a flexible action evaluation algorithm framework for the core algorithm, integrates random network distillation and reinforcement learning calculation internal and external rewards, adjusts the balance of the two through a dynamic exploration mechanism, and trains the unmanned ship path control strategy; (3) The unmanned ship deploys a priority experience replay pool based on quality and success orientation to manage samples of different values and improve data utilization efficiency; (4) The unmanned ship introduces an uncertainty perception and risk perception mechanism to prevent the unmanned ship from mistakenly entering a dangerous area due to excessive exploration, affecting the training efficiency; (5) The unmanned ship builds a success rate-based reward decay and cold start module for internal reward control; (6) The unmanned ship calculates the total reward and designs a reward softening mechanism to smooth the total reward; (7) The unmanned ship imports the hyperparameters of all modules, starts the training cycle in the simulation environment, collects experience and evaluates risks, and dynamically adjusts the exploration intensity and optimizes the strategy.

[0008] Further, the operation steps of the unmanned ship for building the path tracking simulation training environment in step (1) are as follows: (11) The unmanned ship establishes a kinematics model of the unmanned ship, defines the state control and control input space of the unmanned ship, and ensures that the motion behavior can accurately reflect the response characteristics under different control inputs; (12) The unmanned ship builds a path tracking simulation training system based on the Gym library in Python, introduces environmental disturbance models such as wind, wave and current, and combines the control error model of the unmanned ship to accurately reflect the influence of complex environments on path tracking performance; (13) The unmanned ship will input the pulse width command of the remote control channel into the first-order lag model of steering and propulsion after standardization and the application of dead zone and amplitude limiting, and superimpose the quadratic resistance related to the square of the speed to integrate the angular velocity and the forward speed at a fixed time step to advance the pose, so that the "command-ship response" in the simulation is consistent with the caliber of the real ship; (14) The unmanned ship uniformly outputs the task indicators of path progress, lateral deviation, heading deviation, collision / boundary crossing, etc., and generates end / success event markers at the simulation end as the upstream input of reward and playback; (15) Zero-adaptive clipping is performed on the observation to ensure the numerical stability of the subsequent intrinsic reward and risk assessment.

[0009] Further, the operation steps of step (2) are as follows: (21) Build an SAC reinforcement learning framework to establish three update channels of policy, value, and temperature, and update the three update channels through the interaction of the agent and the simulation environment to obtain state, action, and reward; (22) Design a random network distillation module, which is completed by comparing the difference between the features output by the prediction network and the features output by the target network. The mean square error is used to measure the difference between the two features to provide rewards for novel states, wherein the intrinsic reward calculation formula is as follows: In the formula, represents the intrinsic reward, represents the state at time t, and represent the target network feature mapping and the prediction network feature mapping in the random network, respectively, represents the two-norm of the vector, and the superscript 2 outside the norm represents taking the square.

[0010] Further, the operation steps of step (3) are as follows: (31) Establish an experience replay pool based on the time difference error to improve the priority and importance sampling of high-value samples to increase the playback frequency of high-value samples; (32) On the basis of step (31), simultaneously introduce success segment weighting and quality division guidance two signals to build a double priority enhancement module to amplify the appearance frequency of samples with high completion and high quality in training, thereby improving data utilization efficiency, accelerating convergence and reducing training shock. The specific comprehensive priority calculation formula is as follows: In the formula, represents the enhanced priority of the experience sample , represents the basic priority of the sample , and respectively represent the weight coefficients of the adjustment success segment and the quality score influence degree, represents the success segment indicator function, which takes 1 when the sample comes from the success segment, and 0 otherwise, represents the quality score of the sample .

[0011] Further, the operation steps of step (4) are as follows: (41) The unmanned ship adopts feature residual variance estimation action value evaluation uncertainty score, and the specific calculation formula is as follows: In the formula, represents the uncertainty measure at time , represents the feature residual sequence in the latest time window, represents a normal number, which is used to prevent the denominator from being zero and smooth the uncertainty; (42) The unmanned ship calculates the risk score at the current time based on the current forward speed and the safety speed upper limit and the current heading angle speed and the safety angle speed upper limit, and the specific calculation formula is as follows: In the formula, represents the risk score at time , represents a sigmoid function that compresses the input to the [0, 1] interval, represents the current forward speed, represents the set safety forward speed upper limit, and represent the weight coefficients of the speed term and the angular velocity term, represents the current heading angle speed, represents the set safety heading angle speed upper limit; (43) The unmanned ship integrates the uncertainty score and the risk score to weight the internal reward and control exploration, and the specific formula is as follows: In the formula, represents the effective internal reward after weighting by the uncertainty and risk indicators, represents the uncertainty amplification coefficient, represents the uncertainty score, represents the risk suppression coefficient, represents the risk score, represents the original internal reward; (44) When the uncertainty or risk indicator exceeds the threshold, the exploration weight is reduced and a punitive early stop is triggered to prevent excessive exploration from causing unstable training.

[0012] Further, the operation steps of step (5) are as follows: (51) Set a threshold, cold start the internal reward at the beginning of training, and start the internal reward module after the training step exceeds the threshold, and give up the external task to collect a small amount of successful experience for the unmanned ship; (52) Collect the arrival success rate in the recent time window, set the weight β of the internal reward as a form that decreases with the increase of the arrival success rate, gradually reduce the weight of the internal reward, and transfer the learning focus to the external task target when the success rate is high, and the internal reward weight β is affected by the risk index in step (42). The calculation formula is as follows: In the formula, represents the internal reward weight at time , represents the risk suppression weight coefficient, represents the risk score, represents the initial weight, represents the arrival success rate in the recent time window, represents the exponential coefficient of the weight decay speed when the success rate is controlled.

[0013] Further, the operation steps of step (6) are as follows: (61) Soften the collected external reward to reduce its instantaneous fluctuation and make the training more stable. The softening calculation formula is as follows: In the formula, represents the softened external reward, represents the original external reward, and respectively represent the scale coefficients of positive and negative external rewards, which are used to limit the amplitude upper limit of the softened reward, represents the hyperbolic tangent function; (62) When the smoothed reward fluctuation still exceeds the threshold, temporarily suspend the learning rate to ensure stable update of the policy and value estimation; (63) Fuse the internal and external rewards after softening to obtain the total reward. The total reward calculation formula is as follows: In the formula, represents the total reward at time , represents the softened external reward, represents the internal reward weight at time , represents the internal reward weighted by uncertainty and risk index.

[0014] Further, the operation steps of step (7) are as follows: (71) The unmanned ship starts training, and the agent interacts with the environment according to the fixed step length to obtain the state, action, next state, total reward and termination flag; meanwhile, the quality score and risk flag are recorded, and the above information is written into the playback buffer with double priority enhancement; (72) The unmanned ship extracts a batch of samples from the playback according to the success segment weighting and the quality score, while retaining a small amount of uniform sampling as a bottom line, calculates the weighted coefficient for each sample, which is used for subsequent weighted update to offset the bias caused by priority sampling; (73) The unmanned ship constructs a learning target with the total reward of each sample and the estimated value of the next moment, and uses the total reward calculated in step (63) to shape the critic value estimation network in the maximum entropy policy learning in step (21); (74) The unmanned ship uses the state, action and learning target constructed in step (73) to perform weighted regression update on the double Q network in the maximum entropy policy learning in step (21); (75) The unmanned ship updates the policy in the current state distribution, so that it tends to produce actions with higher evaluation values and not too low entropy, and then adjusts the temperature and the current success rate according to the target entropy to achieve adaptive balance of exploration intensity; (76) The priority of the sampled sample is backfilled with the latest learning signal, the success segment weighting is automatically attenuated according to the predetermined step, the sampling ratio of the fresh sample inclination and the uniform bottom line is maintained, and the gating coefficient of the next stage is updated according to the latest arrival success rate after the round is completed, which takes effect synchronously with the sliding estimation of the external reward softening scale in step (6); (77) The unmanned ship continues to cycle the above steps until the path control task is completed.

[0015] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. Addressing the problems of insufficient control accuracy and poor adaptability to dynamic marine environments in existing unmanned vessel path tracking methods under complex sea conditions, this invention introduces an integrated design of uncertainty-aware stochastic network intrinsic reward, risk perception, and reward softening within a soft actor critic-type reinforcement learning framework (SAC). It no longer simply relies on the original sparse external reward but constructs a total reward signal consistent with navigation safety constraints, enabling the agent to obtain a convergent and stable control strategy even under strong disturbances such as waves and currents, significantly reducing yaw error and speed fluctuations; 2. This invention, through uncertainty assessment based on the variance of stochastic network feature residuals and joint adjustment of intrinsic reward based on speed and attitude risk scoring, compared to existing algorithms that only encourage exploration based on error magnitude, can distinguish between "valuable uncertain regions" and "high-risk regions." It actively strengthens exploration in areas where learning is not yet sufficient and automatically suppresses exploration when approaching instability or exceeding speed, thus balancing policy convergence speed and actual navigation safety, reducing the risk of yawing errors during sea trials. 3. The multi-factor weighted experience replay mechanism proposed in this invention comprehensively utilizes temporal difference (TD) error, successful segment marking, and sample quality score to determine sampling weights, and corrects the gradient through importance weights. Compared with traditional uniform sampling or priority experience replay based solely on a single TD error, it focuses more on experience data that "contributes to the task and is highly representative," enabling unmanned vessels to obtain control strategies with better performance and faster convergence under limited simulation data and a small amount of measured data, significantly reducing data acquisition and sea trial costs. 4. This invention designs an adaptive control function based on training schedule, success rate threshold, and cold start for the fusion of internal and external rewards. It can automatically adjust the weights of internal and external rewards as the training progresses. In the early stage of training, the internal reward module is frozen to allow the unmanned vessel to gain some successful experience first, preventing the unmanned vessel from over-exploring. When a certain number of steps are reached, the internal reward module is activated. When the success rate is low, the proportion of internal rewards is appropriately increased to enhance exploration. After the success rate continuously exceeds the set threshold, the internal rewards are gradually reduced, and the focus shifts to fine optimization based on external rewards. This mechanism can automatically balance exploration and utilization at different stages without frequent manual parameter adjustment. In practical engineering applications, it not only improves learning stability but also reduces the workload of tuning unmanned vessel control parameters. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall operation process of the present invention; Figure 2 This is a flowchart of the preferred experience playback process of the present invention; Figure 3 This is a flowchart of the uncertainty perception and risk-constrained reward construction process of the present invention; Figure 4 This is a flowchart of the external reward softening and internal and external reward fusion of the present invention; Figure 5 is a schematic diagram of the overall steps of the present application. DETAILED DESCRIPTION

[0017] The specific technical solutions of the present application will be further described in detail below in combination with specific examples.

[0018] As shown in the figure, the double-track regulation random network distillation reinforcement learning unmanned ship path control method of the present application uses reinforcement learning and random network distillation for path control of the unmanned ship, and uses uncertainty perception to provide uncertainty signals and danger signals for the unmanned ship to improve the adaptability and performance of the algorithm in complex environments.

[0019] The operation steps are as follows: Step 1, build a path tracking simulation environment and a dynamics model for the unmanned ship; specifically: Step 101, establish a kinematics model for the unmanned ship, define the state control and control input space of the unmanned ship, to ensure that its motion behavior can accurately reflect the response characteristics under different control inputs; Step 102, build a path tracking simulation training system based on the Gym library in Python, introduce environmental disturbance models such as wind, wave, and current into the unmanned ship, and combine the control error model of the unmanned ship to truly reflect the influence of complex environments on path tracking performance; Step 103, the unmanned ship will input the pulse width command of the remote control channel after standardization and application of dead zone and amplitude limiting to the first-order lag model of steering and propulsion, and superimpose the quadratic resistance related to the square of the speed, and then integrate the angular velocity and forward speed with fixed time steps to advance the pose, so that the "command-ship response" in the simulation is consistent with the actual ship caliber; Step 104, the unmanned ship uniformly outputs path progress, lateral deviation, heading deviation, collision / boundary crossing, and other task indicators, and generates end / success event markers at the simulation end as upstream inputs for rewards and replays; Step 105, zero-modulus clipping is performed on the observation to ensure the numerical stability of subsequent intrinsic rewards and risk assessments; Step 2, build a flexible action evaluation algorithm framework for the core algorithm, integrate random network distillation and reinforcement learning, and adjust the balance between the two through a dynamic exploration mechanism to train the unmanned ship path control strategy; specifically: Step 201, build a SAC reinforcement learning framework, establish three update channels for policy, value, and temperature, and update the three update channels through the interaction of the agent and the simulation environment to obtain state, action, and reward; Step 202: Design a random network distillation module, which is completed by comparing the differences between the features output by the prediction network and the features output by the target network. The mean squared error is used to measure the difference between these two features, thereby providing a reward for the novelty state. The specific formula for calculating the intrinsic reward is as follows: In the formula, Indicates internal reward, express The state at any given moment, and These represent the target network feature map and the prediction network feature map in a random network, respectively. The 2-norm of a vector is represented by the superscript 2 outside the norm, which indicates squaring. Step 3: Deploy unmanned surface vessels (USVs) based on a quality- and success-oriented priority experience replay pool to manage samples of varying value and improve data utilization efficiency; specifically: Step 301: Establish a priority experience replay pool based on temporal difference error to increase the priority and importance sampling of high-value samples, thereby increasing the replay frequency of high-value samples; Step 302: Building upon Step 301, a dual-priority enhancement module is constructed by simultaneously introducing two signals: weighted successful fragments and quality scores. This amplifies the frequency of high-completeness, high-quality samples during training, thereby improving data utilization efficiency, accelerating convergence, and reducing training oscillations. The specific comprehensive priority calculation formula is as follows: In the formula, Representing empirical samples Enhancement priority, Indicates sample Basic priority, and These represent the weighting coefficients indicating the degree of influence of the successful moderating segment and the quality score, respectively. This represents the success segment indicator function, which sets the value to 1 when the sample comes from a successful segment and 0 otherwise. Indicates sample Quality rating; Step 4: Introduce uncertainty and risk perception mechanisms to the unmanned surface vessel (USV) to prevent it from accidentally entering dangerous areas due to overexploration, thus affecting training efficiency; specifically: Step 401: The unmanned vessel uses characteristic residual variance estimation to assess the uncertainty of action value evaluation; Step 402: The unmanned surface vessel calculates its risk score for the current moment based on the current forward speed and the upper limit of the safe speed, and the current bow angular velocity and the upper limit of the safe angular velocity. The specific calculation formula is as follows: In the formula, Indicates time Risk score, This represents an sigmoid function that compresses the input to the interval [0,1]. Indicates the current forward velocity. This indicates the set safe forward speed limit. and This represents the weighting coefficients for the velocity and angular velocity terms. Indicates the current heading angular velocity. This indicates the set upper limit for safe heading angular velocity; Step 403: When the uncertainty or risk indicators exceed the set threshold, reduce the exploration weight and strategy entropy; Step 404: When the uncertainty or risk index exceeds the threshold, reduce the exploration weight and trigger a punitive early stop to prevent over-exploration from causing training instability. Step 5: The unmanned surface vessel (USV) is equipped with a success rate-based reward decay and cold start module for internal reward control; specifically: Step 501: Set a threshold value to implement a cold start for internal rewards in the early stage of training. After the training step exceeds the threshold value, the internal reward module is activated to give way to the completion of external tasks, allowing the unmanned ship to collect a small amount of successful experience first. Step 502: Collect the achievement success rate of the most recent time window, and set the weight β of the intrinsic reward to decrease as the achievement success rate increases, so as to gradually reduce the weight of the intrinsic reward. When the success rate is high, the learning focus will be shifted to the external task goal, and it will be affected by the risk indicator in step 402. The specific formula for calculating the intrinsic reward weight β is as follows: In the formula, Indicates time Internal reward weighting, This represents the risk suppression weighting coefficient. Indicates risk score, Indicates the initial weights. This indicates the arrival success rate within the most recent time window. The exponential coefficient representing the rate at which the weight decays as the success rate changes; Step 6: Calculate the total reward for the unmanned surface vessel (USV) and design a reward softening mechanism to smooth the total reward; specifically: Step 601: Soften the collected external rewards to reduce their instantaneous fluctuations and make the training more stable. The specific formula for softening is as follows: In the formula, This indicates the external reward after softening. Indicates the original external reward. and These represent the scaling coefficients for positive and negative external rewards, respectively, used to limit the upper limit of the magnitude of the softened reward. Represents the hyperbolic tangent function; Step 602: When the smoothed reward fluctuation still exceeds the threshold, temporarily suspend increasing the learning rate to ensure stable updates of the strategy and value estimation; Step 603: Merge the softened internal and external rewards to obtain the total reward. The formula for calculating the total reward is as follows: In the formula, Indicates time Total reward This indicates the external reward after softening. Indicates time Internal reward weighting, This represents the internal reward weighted by uncertainty and risk indicators; Step 7: Import the hyperparameters of all modules into the unmanned surface vessel (USV), start the training loop in the simulation environment, collect experience, assess risks, and dynamically adjust the exploration intensity and optimize the strategy; specifically: Step 701: No one starts training. The agent interacts with the environment at a fixed step size to obtain the state, action, next state, total reward and termination flag for this step; at the same time, it records the quality score and risk flag and writes the above information into the replay buffer with dual priority enhancement. Step 702: The unmanned vessel extracts a batch of samples from the playback based on the weighted average of successful segments and the quality score. At the same time, it retains a small number of uniform samples as a safety net to calculate the weighted coefficients for each sample, which are used for subsequent weighted updates to offset the bias caused by priority sampling. Step 703: The unmanned ship constructs a learning objective using the total reward for each sample and the estimated value at the next time step. The total reward calculated in step 603 shapes the commentator value estimation network in the maximum entropy policy learning in step 201. Step 704: The unmanned vessel uses the state and actions obtained from interacting with the environment, as well as the learning objective constructed in step 703, to perform weighted regression updates on the double Q network in the maximum entropy policy learning in step 201. Step 705: The unmanned vessel updates its strategy in the current state to make it more inclined to produce actions with higher evaluation values ​​and less entropy. Then, it automatically fine-tunes the temperature and the current success rate based on the target entropy to achieve an adaptive balance of exploration intensity. Step 706: Fill the priority of the sampled samples with the latest learning signal, and automatically decay the weighted success fragments according to the predetermined steps to maintain the sampling ratio of fresh sample tilt and uniform bottom protection. After the round ends, update the gate coefficient of the next stage according to the latest achievement power, and take effect simultaneously with the sliding estimation of the external reward softening scale in Step 6. Step 707: The unmanned vessel continues to repeat the above steps until the path control task is completed.

[0020] In the embodiment, the environment is a simulation marine environment built based on Gym, and the agent is an unmanned ship control system.

[0021] The overall training process of the reinforcement learning is as shown in the figure Figure 1 Specifically, first, the control action is generated by the policy network according to the current state of the policy network, and the marine environment is interacted, the next state and the original external reward are returned by the environment; the RND module calculates the feature residual of the next state to obtain the internal reward, and then combines the uncertainty evaluation and the risk score to weight and adjust the internal reward, and fuse it with the softened external reward to obtain the total reward for learning; subsequently, the current state, action, total reward, next state, risk label, sample quality score and other information are written into the experience replay area with double priority enhancement; in the network updating stage, small batches of samples are extracted from the replay area according to the sampling distribution combined with the successful segment weighting and uniform sampling, the importance weight is calculated, which is used for weighted updating of the critic value network and the policy network, the temperature parameter is adaptively adjusted, and the sample priority is rewritten according to the new learning error, and a round of training cycle is completed; through the repeated iteration of the above interaction, shaping, replay and updating, the continuous optimization and stable convergence of the unmanned ship control strategy are realized.

[0022] In addition, the double-priority experience replay process of the application is as shown in the figure Figure 2 First, the current state, action, total reward, next state and risk label and other information obtained by the interaction of the agent and the environment are written into the experience pool; when the network parameters need to be updated, the time difference error of the candidate sample is calculated, which is used as the basic priority; then, according to whether the sample is from a successful segment and the pre-given sample quality score, the basic priority is additionally amplified and weighted to highlight the successful experience and high-quality samples with better state distribution representation that contribute to the task; on this basis, the final sampling distribution of the experience extraction is constructed, which is used to extract training samples from the experience pool according to different priorities, and the importance weight of each sampled experience is calculated, so as to give greater gradient weight to high-priority samples when updating the network, and compensate for the sampling bias, thus completing the working process of the double-priority experience replay module.

[0023] In addition, the uncertainty perception and risk constraint internal reward construction process of the application is as shown in the figure Figure 3As shown, firstly, the current sea state information and the unmanned ship's own navigation state are collected; then the random network is used to extract the feature residual of the state, and the internal reward signal reflecting the novelty of the state is generated according to the residual size; at the same time, the risk score is calculated according to the speed, attitude and safety speed upper limit of the unmanned ship and other parameters, and the uncertainty score is obtained by observing the fluctuation of the feature residual in time; finally, the uncertainty score and the risk score are combined to weight and adjust the internal reward, and the exploration incentive is enhanced when the uncertainty is high and the risk is acceptable, and the internal reward is inhibited when the risk is high or the environment is familiar, so as to realize the adaptive control of the exploration intensity.

[0024] Figure 4 The reward softening and fusion process of the present application first takes the external reward returned by the environment as input, softens the external reward signal, compresses the excessive positive and negative amplitude to a stable interval, smooths the occasional peak reward, thereby avoiding excessive dependence on instantaneous extreme return in the training process; the softened external reward and the internal reward processed by the weighting are sent to the reward fusion module, the adjustment function is automatically adjusted in the module according to the training schedule, success rate gate and other information, the weights of internal and external rewards are adaptively distributed, and the total reward signal at the current time is generated; finally, the total reward is sent to the SAC algorithm to construct the target value of the Q network, and drive the update of the policy network and the temperature parameter.

[0025] Figure 5 The step flow chart of the unmanned ship path control method based on the random network distillation and uncertainty perception fusion of the reinforcement learning algorithm of the present application is shown.

Claims

1. A double-track regulation reinforcement learning unmanned ship path control method for random network distillation, characterized by, Comprise the following steps: (1) The unmanned ship builds a path tracking simulation environment and a dynamic model; (2) The unmanned ship builds a flexible action evaluation algorithm framework, integrates random network distillation and reinforcement learning to calculate internal and external rewards, and adjusts the balance of the two through a dynamic exploration mechanism to train the unmanned ship path control strategy; (3) The unmanned ship deploys a priority experience replay pool based on quality and success orientation to manage samples of different values and improve data utilization efficiency; (4) The unmanned ship introduces uncertainty perception and risk perception mechanisms to prevent the unmanned ship from mistakenly entering dangerous areas due to excessive exploration, affecting training efficiency; (5) The unmanned ship builds a success rate-based reward decay and cold start module for internal reward control; (6) The unmanned ship calculates the total reward and designs a reward softening mechanism to smooth the total reward; (7) The unmanned ship imports the hyperparameters of all modules, starts the training cycle in the simulation environment, collects experiences and evaluates risks, and dynamically adjusts the exploration intensity and optimization strategy.

2. The dual rail reinforcement learning method for path control of unmanned ships for stochastic network distillation according to claim 1, wherein, The operation steps of the unmanned ship in step (1) to build a path tracking simulation training environment are as follows: (11) The unmanned ship establishes the kinematics model of the unmanned ship, defines the state control and control input space of the unmanned ship, and ensures that its motion behavior accurately reflects the response characteristics under different control inputs; (12) The unmanned ship builds a path tracking simulation training system based on the Gym library in Python, introduces environmental disturbance models such as wind, waves, and currents, and combines the control error model to accurately reflect the influence of complex environments on path tracking performance; (13) The unmanned ship standardizes the pulse width instructions of the remote control channel, applies dead zones and amplitude limits, and inputs them into the first-order lag models of steering and propulsion, respectively, and superimposes the quadratic resistance related to the square of the speed, and then integrates the angular velocity and forward velocity with a fixed time step to advance the pose, so that the command-ship response in the simulation is consistent with the actual ship caliber; (14) The unmanned ship outputs the task indicators uniformly and generates end / success event markers on the simulation side as inputs for rewards and replay; (15) Zero adaptive clipping is performed on the observations to ensure the numerical stability of subsequent internal rewards and risk assessments.

3. The double-track reinforcement learning method for path control of USV based on random network distillation according to claim 2, wherein, The task indicators output in step (14) include path progress, lateral deviation, heading deviation, collision / boundary crossing.

4. The method of claim 1, wherein, The operation steps of step (2) are as follows: (21) Build a SAC reinforcement learning framework to establish three update channels for policy, value, and temperature, and update them through the interaction between the agent and the simulation environment to obtain state, action, and reward; (22) A random network distillation module is designed, which is completed by comparing the difference between the features of the predicted network output and the features of the target network output, and the mean square error is used to measure the difference between the two features to provide a reward for the novelty state, wherein the calculation formula of the internal reward is as follows: wherein represents an internal reward, represents a state at a time instant, and represent a target network feature map and a prediction network feature map in a stochastic network, respectively, represents a two-norm of a vector, and the superscript 2 outside the norm represents taking a square.

5. The method of claim 1, wherein, The operation steps of step (3) are as follows: (31) Establish a priority experience replay pool based on time difference error to improve the priority and importance sampling of high-value samples, thereby increasing the playback frequency of high-value samples; (32) On the basis of step (31), introduce success segment weighting and quality division guidance to build a dual priority enhancement module to amplify the frequency of high-quality samples with high completion in training, improve data utilization efficiency, accelerate convergence, and reduce training shock, The specific comprehensive priority calculation formula is as follows: In the formula, represents the enhanced priority of the experience sample , represents the basic priority of the sample , and respectively represent weight coefficients for adjusting the influence degree of the successful segment and the quality score, represents a successful segment indication function, taking 1 when the sample comes from a successful segment, and 0 otherwise, represents the quality score of the sample .

6. The method of claim 1, wherein, The operation steps of step (4) are as follows: (41) The unmanned ship adopts a characteristic residual variance estimation action value evaluation uncertainty score, and its calculation formula is as follows: In the formula, Indicates the uncertainty measure of time , Indicates the characteristic residual sequence in the latest time window, Indicates a normal number, which is used to prevent the denominator from being zero and to smooth the uncertainty; (42) The unmanned ship calculates the risk score at the current time based on the current forward speed and the safe speed upper limit and the current heading angle speed and the safe angle speed upper limit, and the calculation formula is as follows: wherein represents the risk score at time represents a sigmoid function that compresses the input to the interval [0, 1], represents the current forward speed, represents the set safe forward speed upper limit, and represent the weight coefficients for the speed term and the angular speed term, respectively, represents the current yaw rate, represents the set safe yaw rate upper limit;​ (43) The comprehensive uncertainty score and risk score of the unmanned ship are weighted to the internal reward, and the exploration is controlled, and the formula is as follows: In the formula, The effective internal reward weighted by the uncertainty and risk indicators is represented by The uncertainty amplification coefficient is represented by The uncertainty score is represented by The risk suppression coefficient is represented by The risk score is represented by The original internal reward is represented by (44) When the uncertainty or risk index exceeds the threshold, reduce the exploration weight and trigger the punitive early stop to prevent excessive exploration from leading to unstable training.

7. The method of claim 1, wherein, The operation steps of the step (5) are as follows: (51) Set a threshold, cold start the internal reward at the beginning of training, and start the internal reward module after the training step exceeds the threshold, and give up the external task, and the unmanned ship first collects successful experience; (52) Collect the success rate of arrival of the recent time window, set the weight β of the internal reward as a form that decreases with the increase of the success rate of arrival, realize the step-by-step weight reduction of the internal reward, and transfer the learning focus to the external task target when the success rate is high. The calculation formula of the weight β of the internal reward is as follows: wherein represents the time, represents the internal reward weight at the time, represents the risk suppression weight coefficient, represents the risk score, represents the initial weight, represents the arrival success rate in the recent time window, represents the exponential coefficient of the weight decay speed when the control success rate changes.

8. The method of claim 1, wherein, The operation steps of the step (6) are as follows: (61) The collected external rewards and internal rewards are softened to reduce their instantaneous fluctuations and make the training more stable. The softening calculation formula is as follows: In the formula, represents the softened external reward, represents the original external reward, and respectively represent the scale coefficients of the positive and negative external rewards, used to limit the amplitude upper limit of the softened reward, represents the hyperbolic tangent function; (62) When the fluctuation of the smoothed reward still exceeds the threshold, temporarily suspend the learning rate to ensure stable update of the policy and value estimation; (63) The total reward is obtained by fusing the inner and outer rewards after softening, and the total reward calculation formula is as follows: wherein the total reward at time the outer reward after softening, the inner reward weight at time the inner reward weighted by uncertainty and risk metrics.​​ 9. The method of claim 1, wherein, The operation steps of the step (7) are as follows: (71) The unmanned ship starts training, and the agent interacts with the environment according to the fixed step; (72) The unmanned ship extracts a batch of samples from the playback according to the successful fragment weighting and quality score, and retains a small amount of uniform sampling as a bottom line, calculates the weighted coefficient for each sample, and uses it for subsequent weighted update to offset the bias caused by priority sampling; (73) The unmanned ship constructs the learning target with the total reward of each sample and the estimated value of the next moment, and the critic value estimation network in the maximum entropy policy learning in step (21) is shaped by the total reward calculated in step (63); (74) The unmanned ship uses the state, action and learning target constructed in step (73) obtained by interacting with the environment to perform weighted regression update on the double Q network in the maximum entropy policy learning in step (21); (75) The unmanned ship updates the policy according to the current state distribution, so that it tends to produce actions with higher evaluation value and not too low entropy, and then adjusts the temperature and the current success rate according to the target entropy to realize adaptive balance of exploration intensity; (76) Backfill the priority of the sampled sample with the latest learning signal, automatically decay the success fragment weighting according to the predetermined step, maintain the sampling ratio of the fresh sample tilt and the uniform bottom line, and update the gating coefficient of the next stage according to the latest arrival success rate after the round is over, and the sliding estimation of the external reward softening scale in step (6) is synchronized to take effect; (77) The unmanned ship continues to cycle the above steps until the path control task is completed.

10. The dual-rail reinforcement learning method for path control of USV based on random network distillation according to claim 9, wherein, In the step (71), the state, action, next state, total reward and termination flag of the agent are obtained by interacting with the environment according to the fixed step, and the quality score and risk flag are recorded. The above information is written into the playback buffer with double priority enhancement.

Images