Model-free AUV depth control method based on reinforcement learning

Abstract

The invention particularly relates to a model-free AUV depth control method based on reinforcement learning. The method comprises the following steps: establishing an AUV kinetic model; based on an AUV dynamical model, establishing an interaction model of the AUV and the environment by adopting a Markov decision process, determining an action space variable and a state space variable of an AUV control system, and establishing an AUV depth control objective function and a sectional reward function; establishing a neural network model, respectively establishing a strategy network and an evaluation network, and performing model training on the control strategy by adopting a continuous PPO algorithm to obtain control strategy parameters of the AUV; the control strategy parameters of the AUV are used for realizing vertical plane depth-keeping control of the AUV; and through reinforcement learning, enabling the AUV to update the training network at each fixed step length in the set depth control training until convergence, and obtaining a final strategy network at a predetermined depth. The method has high real-time control capability and self-adaptive control characteristics, can effectively cope with environmental changes, and shows depth control robustness and depth control adaptability.

Description

Technical Field

[0001] The present invention relates to the field of AUV intelligent control, and in particular to a model-free AUV depth control method based on reinforcement learning. Background Art

[0002] For missions such as marine resource exploration, seabed surveys, and ocean mapping, where humans cannot reach the site for operation, ensuring the autonomy and controllability of AUV underwater motion is a fundamental and important functional requirement. However, as a multi-input, multi-output system, the AUV's dynamic model is highly nonlinear, strongly coupled, subject to input or state constraints, and time-varying. Furthermore, uncertainty in model parameters and the hydrodynamic environment complicates modeling of the AUV system. As AUV application scenarios continue to expand, higher requirements are being placed on the accuracy and stability of its motion control. Improving the control effectiveness of AUVs in various motion scenarios has become an important research direction.

[0003] Due to the uncertainty, randomness, and strong nonlinearity of both the AUV itself and its environment, many traditional control methods, such as PID control, fuzzy control, and adaptive control, suffer from weak adaptability, poor robustness, and poor control accuracy. In recent years, reinforcement learning technology has made significant progress. Faced with unknown or uncertain environments, it can achieve self-learning through interaction with the environment, accumulating experience through repeated work, enabling autonomous decision-making in the next encounter with the same or similar situations. For depth control of AUVs, the lack of an accurate dynamic model is the greatest obstacle to the application of traditional control methods. However, a controller based on reinforcement learning is a model-independent control method that does not require precise hydrodynamic coefficients. Instead, it can achieve autonomous learning and decision-making through continuous interaction with the environment, thereby achieving depth control of the AUV.

[0004] The prior art also discloses a control method based on reinforcement learning, but has the following problems:

[0005] (1) The reward function is designed to be simple: it cannot dynamically guide the strategy adjustment according to different control stages;

[0006] (2) Insufficient control accuracy: lack of high-precision fine-tuning when approaching the target, prone to overshoot or oscillation;

[0007] (3) Low training efficiency: Due to the lack of phased characteristics of the incentive signal, strategy learning is slow and difficult to converge quickly.

[0008] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0009] The present invention provides a model-free AUV depth control method based on reinforcement learning, which is used to solve the defects of the existing reinforcement learning technology in which a unified linear or nonlinear function is used to reward the entire control process, but the characteristics of different control stages cannot be distinguished, resulting in low training efficiency and slow convergence of the reinforcement learning model, and even easy to fall into local optimality.

[0010] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0011] According to a first aspect of the present invention, a model-free AUV depth control method based on reinforcement learning is provided, the method comprising:

[0012] The Markov decision process is used to establish the interaction model between the AUV and the environment, determine the action space variables and state space variables of the AUV control system, and establish the AUV depth control objective function and piecewise reward function;

[0013] A neural network model is established, and a policy network and an evaluation network are established separately. A continuous PPO algorithm is used to train the control strategy model to obtain the control strategy parameters of the AUV. The control strategy parameters of the AUV are used to achieve fixed-depth control of the AUV on the vertical plane. Through reinforcement learning, the AUV updates the training network at a fixed step size during the set depth control training until convergence, thereby obtaining the final policy network for the predetermined depth.

[0014] In some exemplary embodiments, the action space variable of the AUV control system is determined to be u k =[τ k ,δ k ] T , where τ k , δ k are the propeller thrust and rudder angle of the AUV, respectively. The subscript k represents the kth time step; τ k , δ k The value ranges are [0,τ max ] and [-δ max ,δ max ], τ max , δ max are the maximum propeller thrust and horizontal rudder angle, respectively, which are used to limit the range of the action space to comply with physical constraints.

[0015] In some exemplary embodiments, the state space variable of the AUV control system is determined as x=[zz r , θ-θ r, w, q], where z represents the current depth of the AUV in the inertial coordinate system, θ represents the angle between the AUV's forward direction and the x-axis in the inertial coordinate system, and q represents the angular velocity of the AUV's y-axis rotation in its own coordinate system. r represents the pitch angle error, w represents the vertical speed of the AUV in its own coordinate system, zz r Indicates the depth error between the AUV's current depth and the target depth, used to directly guide and control the target.

[0016] In some exemplary embodiments, establishing the AUV depth control objective function is specifically:

[0017]

[0018] Among them, J(π) is the objective function of strategy π, which represents the expected cumulative reward; r t represents the reward at step t, provided by the reward function r(s,a); γ∈[0,1] is a discount factor used to weigh short-term and long-term rewards; T is the number of time steps in the task, e depth ,e theta ,e psi ,u control They represent depth error, pitch angle error, heading error, and control input error respectively.

[0019] In some exemplary embodiments, establishing a piecewise reward function is specifically as follows:

[0020]

[0021] Among them, k1, k2, k3, and k4 represent the weight coefficients of each item, k1(zz r )+k2(θ-θ r ) ensures that the AUV reaches the specified depth while its forward direction is consistent with the x-axis, and k3w+k4q makes the longitudinal speed and rotational angular velocity of the AUV as small as possible.

[0022] In some exemplary embodiments, the input of the policy network is a state variable, and based on the current state, a probability distribution for selecting an action is generated.

[0023] In some exemplary embodiments, the evaluation network outputs a value estimate of the current state state, which represents an estimate of the long-term reward of the state, for optimizing the policy network.

[0024] In some exemplary embodiments, the continuous PPO algorithm is used to perform model training on the control strategy, specifically:

[0025] Set the size of the experience buffer used for batch sampling during training;

[0026] Define the relevant AUV motion state variables and initialize all AUV state parameters; randomly initialize the AUV state variables as x0 = [u0, v0, w0, p0, q0, r0] T , and let x t =x0;

[0027] At time step t, based on the current state state, the policy network generates a deterministic action μ(s t |θ), and obtain the final control action u by adding exploration noise Δu t =μ(s t |θ)+Δu t ; The policy network outputs the action probability distribution π(a|s t ), from which action a is sampled t ~π(a|s t );Control action u t Or sampling action a t Input into the AUV dynamics model, interact with the environment and update to the next state s t+1 , and get instant rewards t ; interactively generated empirical data (s t ,u t ,r t ,s t+1 ) is stored in the experience buffer R for subsequent policy optimization and value estimation; in this way, the policy network strikes a balance between exploration and exploitation, gradually optimizing the control strategy to achieve the target task;

[0028] Count the training step length n-steps and update the current network model. When n-steps> Thornical, update the evaluation network weight according to the following formula:

[0029] TD Target t =r t +γV(s t+1 )Critic

[0030]

[0031] Among them, r t is the reward at the current moment, γ is the discount factor, V(s t+1 ) is the estimate of the state at the next moment; λ is the regularization coefficient, w i is the weight in the Critic network, α is the learning rate;

[0032] Update the target evaluation network weight and target strategy network weight according to the following formula:

[0033]

[0034] In each training iteration, the AUV randomly selects an initial position and begins a new round of fixed-depth control tasks. This process aims to enhance the generalization ability of the policy network through diversified initial conditions, enabling it to adapt to different starting states. If the predetermined training termination condition is not reached in the current iteration, the next iteration will be continued; otherwise, the training process ends.

[0035] At the end of the iteration, the current policy network μ(s t |θ) is the final trained policy network, denoted as μ * (s t |θ); The network is optimized through multiple iterations and is able to t Output the optimal control action a t , thereby achieving depth control of AUV;

[0036] If the training termination conditions are met, training can be stopped. The training target requirements are that the depth error is less than 0.01 meters, the pitch angle error is less than 0.01 degrees, and the training is stable for more than 1000 steps. At this time, the training is considered to have met the target requirements. Other termination conditions are safety assessment criteria for the navigation process. The specific training termination conditions are as follows:

[0037]

[0038] According to a second aspect of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning-based model-free AUV depth control method described in the first aspect is implemented.

[0039] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning-based model-free AUV depth control method described in the first aspect is implemented.

[0040] According to a fourth aspect of the present invention, there is provided an electronic device, comprising:

[0041] processor; and

[0042] a memory for storing executable instructions of the processor;

[0043] Wherein, the processor is configured to implement the reinforcement learning-based model-free AUV depth control method described in the first aspect above by executing the executable instructions.

[0044] The reinforcement learning-based model-free AUV depth control method provided by the embodiments of the present invention realizes the set depth control of the AUV when the AUV dynamic model is completely unknown. Through the reinforcement learning method, the AUV can interact with the environment and optimize the control strategy without relying on an accurate dynamic model. This model-free control method enables the AUV to autonomously explore and utilize in the state space and action space through strategy optimization and the guidance of reward signals, thereby achieving depth maintenance and improving depth control accuracy. The present invention has high real-time control capabilities and adaptive control characteristics, can effectively respond to environmental changes, demonstrates depth control robustness and depth control adaptability, and has significant practical value.

[0045] Compared with the single linear reward function scheme used in the existing technology, this paper proposes a novel piecewise reward function. Combining the characteristics of each stage in the deep control task, the control process is divided into coarse adjustment stage, fine adjustment stage and fine adjustment stage, and differentiated reward strategies are designed for each stage, thereby improving the learning efficiency and control accuracy of the model.

[0046] The control method proposed in this paper requires no prior knowledge and instead autonomously learns through continuous interaction with the environment, ultimately achieving the optimal control strategy and thus controlling the depth of the underwater robot. To address the learning speed issue, extensive offline training can be conducted in a simulation environment to cover as many possible state values ​​as possible. This training can then be put into practical use, enabling real-time control of the underwater robot while maintaining the ability to continuously adapt to environmental changes during actual control.

[0047] The present invention does not rely on precise kinematic and dynamic models of the controlled object, and can achieve effective control of the underwater robot in a complex environment only through continuous interaction with the environment.

[0048] Since the method of the present invention adopts multiple evaluation networks, its learning process is not easily affected by the bad historical depth trajectory of the AUV, has good robustness, and the learning process is stable.

[0049] The method of the present invention combines reinforcement learning with neural networks, has strong self-learning ability, can achieve high-precision adaptive control of AUVs in uncertain deep-sea environments, and has good application prospects in scenarios such as AUV trajectory tracking and underwater obstacle avoidance.

[0050] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0052] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0053] Figure 2 It is a diagram describing the motion relationship of the AUV in two coordinate systems in the present invention;

[0054] Figure 3 It is the basic framework diagram of reinforcement learning of the method of the present invention;

[0055] Figure 4 It is a principle block diagram of the PPO algorithm of the method of the present invention;

[0056] Figure 5 From top to bottom are the AUV depth and current depth comparison diagram of the method of the present invention, the AUV pitch angle control diagram, the convergence diagram of the reward function reward, and the rudder angle change diagram. DETAILED DESCRIPTION

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0058] In addition, the accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0059] In related technologies, a reinforcement learning-based fixed-depth control method for an underwater autonomous robot has been proposed. First, a Markov decision process model for fixed-depth control of the underwater autonomous robot is constructed, and expressions for the state variables, control variables, transition model, and one-step loss function are obtained. A decision network and evaluation network are then established. Through reinforcement learning, the decision network and evaluation network are continuously updated with each step forward in fixed-depth control training until convergence is achieved, resulting in a final decision network for fixed-depth control. However, this method often uses a unified linear or nonlinear function to reward the entire control process, failing to distinguish the characteristics of different control stages. This results in low training efficiency, slow convergence, and even a tendency to fall into local optimality for the reinforcement learning model. Because it fails to provide more precise guidance signals near the target depth, the trained strategy exhibits large fluctuations near the target point, resulting in insufficient control accuracy.

[0060] In response to the shortcomings and deficiencies of the existing technology, this example implementation provides a model-free AUV depth control method based on reinforcement learning, proposes a novel piecewise reward function, and combines the characteristics of each stage in the depth control task to divide the control process into coarse adjustment stage, fine adjustment stage and fine adjustment stage, and designs differentiated reward strategies for each stage, thereby improving the learning efficiency and control accuracy of the model.

[0061] Specifically, the following steps may be included:

[0062] Step S1, using the Markov decision process to establish an interaction model between the AUV and the environment, determine the action space variables and state space variables of the AUV control system, and establish the AUV depth control objective function and piecewise reward function;

[0063] Step S2: Establish a neural network model, establish a policy network and an evaluation network respectively, use a continuous PPO algorithm to train the control strategy model, and obtain the control strategy parameters of the AUV; the control strategy parameters of the AUV are used to achieve fixed depth control of the AUV on the vertical plane; through reinforcement learning, the AUV updates the training network at a fixed step size in the set depth control training until convergence, and obtains the final policy network for the predetermined depth.

[0064] Below, each step of the phased array radar design method in this example implementation will be described in more detail with reference to the accompanying drawings and examples.

[0065] In step S2, the Markov decision process is used to establish the interaction model between the AUV and the environment, determine the action space variables and state space variables of the AUV control system, and establish the AUV depth control objective function and piecewise reward function.

[0066] For example, the Markov Decision Process (MDP) provides a systematic framework that incorporates the AUV's state transitions, control decisions, and environmental disturbances into a unified model, guiding the design of the control strategy through the form of objective function minimization. Defining the AUV depth control problem includes the following four parts: determining the action space variables of the AUV control system; defining the AUV state space variables, defining the depth control error and establishing the AUV depth control target; and determining the reward function. The specific steps are as follows:

[0067] 1-1) Determine the AUV control system action space: vector u k =[τ k ,δ k ] T , where τ k , δ k are the propeller thrust and rudder angle of the AUV, respectively. The subscript k represents the kth time step; τ k , δ k The value ranges are [0,τ max ] and [-δ max ,δ max ], τ max , δ max are the maximum propeller thrust and horizontal rudder angle, respectively, which are used to limit the range of the action space to comply with physical constraints;

[0068] 1-2) Determine the state space of the AUV control system: z represents the depth of the AUV in the inertial coordinate system, θ represents the angle between the AUV's forward direction and the X-axis in the inertial coordinate system, u represents the longitudinal speed of the AUV in its own coordinate system, and q represents the angular velocity of the AUV's rotation about the y-axis in its own coordinate system. These parameters reflect the dynamic state of the AUV. Assume that the target depth of the AUV in the inertial coordinate system is z r , then the depth error between the current depth of the AUV and the target depth is zz r , used to directly guide the control target. Then the AUV state variable expression is: x = [zz r , θ-θ r ,w,q];

[0069] 1-3) Determine the objective function: Depth control for AUVs is a core task in navigation and operation. Its goal is to ensure that the AUV can accurately reach and stably maintain the target depth to adapt to complex underwater environments and operational requirements. To achieve this goal, the expected cumulative loss function is often introduced as the optimization objective. The objective function expression is as follows:

[0070]

[0071] Among them, J(π) is the objective function of strategy π, which represents the expected cumulative reward; r t represents the reward at step t, provided by the reward function r(s,a); γ∈[0,1] is a discount factor used to weigh short-term and long-term rewards; T is the number of time steps in the task.

[0072] 1-4) Determine the Reward Function: The reward function is used to evaluate the effectiveness of the AUV's control variables at a given moment, represented by reward. This paper proposes a novel segmented reward function that allocates control rewards to different control stages to achieve more precise regulation. The specific reward function settings are as follows:

[0073]

[0074] Among them, k1, k2, k3, and k4 represent the weight coefficients of each item (the value of each coefficient is between [0, 1]). r )+k2(θ-θ r ) ensures that the AUV reaches the specified depth while its forward direction is consistent with the x-axis, and k3w+k4q makes the longitudinal speed and rotational angular velocity of the AUV as small as possible.

[0075] In step S2, a neural network model is established, and a policy network and an evaluation network are established respectively. The control strategy is trained using a continuous PPO algorithm to obtain the control strategy parameters of the AUV; the control strategy parameters of the AUV are used to achieve fixed-depth control of the AUV on the vertical plane; through reinforcement learning, the AUV updates the training network at a fixed step size in the set depth control training until convergence, and the final policy network for the predetermined depth is obtained.

[0076] For example, to achieve stable depth control of an AUV when the dynamic model is completely unknown, the present invention constructs an evaluation network and a policy network, and performs collaborative optimization based on a reinforcement learning framework. The evaluation network is used to estimate the state value function or action value function, and provides feedback for policy optimization by evaluating the long-term cumulative reward of the AUV in a specific state. The policy network directly outputs control actions, mapping the current state of the AUV into specific control instructions to achieve depth maintenance and improve depth control accuracy.

[0077] 2-1) The evaluation network provides state value estimates for the policy network, helping to calculate the advantage function and thus guide policy optimization. Updates to the evaluation network are performed alternately with updates to the policy network to ensure that the two remain synchronized during training. Based on the current state, it outputs a value estimate V(s) for that state, representing the estimated long-term reward for that state. This is used to optimize the policy network. The evaluation network is expressed as follows:

[0078] V(s)=WT h(s)+b

[0079] Among them, V(s) is the value estimate of state s; W is the weight parameter of the network; h(s) is the result of state s being processed by the hidden layer activation function; b is the bias term.

[0080] 2-2) The policy network is responsible for generating actions and optimizing the policy by maximizing the objective function (including policy improvement and constraints). The policy network takes the state variable as input and generates a probability distribution (Gaussian distribution) for selecting actions based on the current state. It then samples actions based on the distribution. The probability density function of the Gaussian distribution is as follows:

[0081]

[0082] For example, a reinforcement learning-based continuous proximal policy optimization (PPO) algorithm is used to achieve stable control and precise maintenance of an underwater autonomous robot at the target depth. By constructing a policy network and an evaluation network, the policy network is responsible for generating control instructions in the continuous action space, such as propeller speed and rudder angle adjustment, while the evaluation network is used to estimate the value function of the current state and provide feedback on the control strategy. The specific steps are as follows:

[0083] 3-1) Establish an AUV dynamics model as an environmental interaction model for reinforcement learning, so that reinforcement learning can interactively train algorithms and parameters to achieve the set navigation mission:

[0084]

[0085] Where v is the AUV's six-degree-of-freedom velocity vector; M is the inertia matrix; C(v) is the Coriolis centripetal force matrix; D(v) is the fluid damping matrix; g(η) is the restoring force and torque vector; τ is the force and torque vector generated by the actuator, τ d is the external environment interference force and torque vector.

[0086] 3-2) Reward function design: Construct a suitable reward function by analyzing the predetermined navigation goals and AUV feedback status;

[0087] 3-3) Adding interference: During the training of the control strategy, the environmental interference factor τ is added in a given round d , used to test the anti-interference ability of the control strategy;

[0088] 3-4) Design the critic network and policy network, and initialize them separately. Each network has its own optimizer (actor_optimizer and critic_optimizer). Create your own optimizer to update the critic network's parameters during subsequent training.

[0089] 3-5) Set the size of the experience buffer to facilitate batch sampling during subsequent training. Set the experience buffer R (state, action, reward, next state, etc.) to be empty. As training progresses, the agent's interaction experience with the environment will be gradually added to the experience buffer R.

[0090] 3-6) Define the relevant AUV motion state variables and initialize all AUV state parameters; randomly initialize the AUV state variables to x0 = [u0, v0, w0, p0, q0, r0] T , and let x t =x0;

[0091] 3-7) At time step t, based on the current state state, the policy network generates a deterministic action μ(s t |θ), and obtain the final control action u by adding exploration noise Δu t =μ(s t |θ)+Δu t At the same time, the policy network outputs the action probability distribution π(a|s t ), from which action a is sampled t ~π(a|s t ). The control action u t Or sampling action a t Input into the AUV dynamics model, interact with the environment and update to the next state s t+1 , and get instant rewards t . Interactively generated empirical data (s t ,u t ,r t ,s t+1 ) is stored in the experience buffer R for subsequent policy optimization and value estimation. In this way, the policy network strikes a balance between exploration and exploitation, gradually optimizing the control strategy to achieve the target task.

[0092] 3-8) Count the training step length n-steps and update the current network model. When n-steps> Thornical, update the evaluation network weight according to the following formula:

[0093] TD Target t =r t +γV(s t+1 )Critic

[0094]

[0095] Among them, r t is the reward at the current moment, γ is the discount factor, V(s t+1) is the estimate of the state at the next moment; λ is the regularization coefficient, w i is the weight in the Critic network, and α is the learning rate.

[0096] 3-9) Update the target evaluation network weight and target strategy network weight according to the following formula:

[0097]

[0098] 3-10) During each training iteration, the AUV randomly selects an initial position and begins a new round of fixed-depth control. This process aims to enhance the generalization ability of the policy network by diversifying initial conditions, enabling it to adapt to different starting states. If the predetermined training termination criteria (such as the maximum number of iterations or performance convergence) are not reached in the current iteration, the training process continues to the next iteration. Otherwise, the training process ends.

[0099] 3-11) When the iteration ends, the current policy network μ(s t |θ) is the final trained policy network, denoted as μ * (s t |θ). After multiple iterations of optimization, the network is able to t Output the optimal control action a t , thereby achieving depth control of the AUV.

[0100] 3-12) Training Termination Conditions: If the set requirements are met, training can be stopped. The training target requirements set by this method are depth error less than 0.01 meters, pitch angle error less than 0.01 degrees, and stable operation for more than 1000 steps. At this point, the training is considered to have met the target requirements. Other termination conditions are based on the safety assessment criteria of the navigation process. The specific training termination conditions are as follows:

[0101]

[0102] Example 1

[0103] Step 1: mathematically model the underwater motion control of the underwater robot and establish a six-degree-of-freedom spatial motion model and a six-degree-of-freedom dynamic model of the underwater robot;

[0104] Step 2: Use the PyTorch framework in Python to build an agent-environment interaction model in Windows. Build a visual environment interface to train the policy network by defining the action space and state space.

[0105] Step 3: Model the underwater robot depth control problem as a Markov decision problem. Based on the principle of minimizing the depth error and pitch angle error, set a reasonable reward function and exploration strategy.

[0106] Step 4: Add random perturbations to the reinforcement learning training environment to prevent the controller from over-simulating a single environment. Add environmental disturbances to a given round to test the control strategy's ability to resist disturbances.

[0107] Step 5: Construct four neural networks: Actor network, Critic network, Target Actor network, and Target Critic network, and initialize the four neural networks. Use the PPO algorithm to train the control strategy model. The neural network models update their parameters according to their respective update methods to obtain the control strategy parameters of the underwater robot.

[0108] Step 6: Implement AUV depth control according to the designed reinforcement learning control strategy parameters.

[0109] Example 2

[0110] The initial depth of AUV is a random depth (0, 2), the initial heading is due north (0°), the given target pitch angle is 0°, and the given target depth is 20m. The PPO method is applied to the fixed depth control method of AUV, and a neural network with a suitable structure is designed for the AUV control problem. The PPO algorithm is a reinforcement learning method for processing continuous action space. It assumes a deterministic policy function and updates the policy along the gradient direction of the maximized long-term loss function. The present invention focuses on the control problem of AUV at a fixed depth, specifically describing how to design a control strategy to drive the AUV to and maintain at a specified depth. First, we model the problem as a Markov decision process, and then use the reinforcement learning method to solve the optimal strategy. The overall process of this method is shown in the attached figure. Figure 1 As shown, the following steps are included:

[0111] 1. Formulate the AUV's deep control problem as a Markov decision process and construct a reinforcement learning training environment;

[0112] The AUV depth control problem is mainly decomposed into four modules: state variable design, control variable design, transfer model establishment, and reward function design. The design steps of the specific modules are as follows:

[0113] (1) State variable module design

[0114] State variables are key parameters that describe the dynamic behavior of a controlled object during control. Their design directly determines the control system's perception and decision-making accuracy. In this paper, addressing the depth control problem of an AUV, it is necessary to first clearly define the core physical variables that describe the AUV's depth variations. These variables should be able to fully characterize the AUV's vertical motion state, including but not limited to depth position, vertical velocity, pitch angle, and its rate of change.

[0115] Two reference frames are usually used to describe the motion state of an AUV: the first is called the inertial coordinate system, which is a ground reference system that ignores factors such as the rotation of the earth. The inertial coordinate system is used to determine the absolute position and attitude of the AUV in the global environment; the rotating coordinate system is a coordinate system fixed to the AUV, with the AUV's axis as the reference axis. The rotating coordinate system is used to describe the AUV's own rotational motion and attitude changes. For details, see the attached Figure 2 . Figure 2 Two reference coordinate systems are used to describe the motion of the AUV. The E-ξηζ inertial coordinate system describes the position coordinates of the AUV in the inertial coordinate system. The G-XYZ body coordinate system is used to describe the AUV's rotational motion and attitude transformation. The origin G is usually selected from the center of gravity or buoyancy of the AUV.

[0116] In the present invention's research on AUV depth control, we only focus on analyzing the motion characteristics of the AUV in the XZ plane of the inertial coordinate system. The series of key physical variables set are all parameters related to the motion characteristics in the XZ plane. Specifically, z represents the displacement of the AUV in the inertial coordinate system along the depth direction; θ represents the angle between the AUV's forward direction and the X-axis of the inertial coordinate system. This parameter is crucial for understanding the AUV's navigation trajectory; w is the longitudinal velocity component of the AUV perpendicular to the forward direction in its own coordinate system, which reflects the dynamic changes of the AUV in the depth direction; and q represents the angular velocity of the AUV around its vertical axis, that is, the rate of change of the angle θ. The above-mentioned physical parameters play a key role in controlling the steering and attitude stability of the AUV. In the present invention, we assume that the forward speed of the AUV remains constant in order to explore in more depth the impact of the above-mentioned physical variables on the depth control of the AUV.

[0117] The core goal of this invention is to enable the AUV to accurately reach the specified depth and have a certain anti-interference ability. And to ensure that its pitch angle eventually converges to 0, we introduce the concepts of relative depth and relative pitch angle. Specifically, let the target depth of the AUV in the inertial coordinate system be z r , the target pitch angle is θ r (Here θ r Theoretically, it should be 0, representing a horizontal attitude), then the current depth z of the AUV is different from the target depth z r The difference between them is defined as the relative depth, i.e., zz r ; Similarly, the AUV current pitch angle θ and the target pitch angle θ r The deviation between them is defined as the relative pitch angle, θ-θ r (In practical applications, θ rThrough this transformation, even if the target depth changes, the method proposed in the present invention can still develop an effective control strategy to ensure that the AUV can accurately reach the new target depth and maintain a horizontal attitude.

[0118] With regard to the depth control problem of the AUV with anti-disturbance in the present invention, after the above theoretical analysis, the final setting of the state space variable is as follows: s = [zz r ,θ-θ r ,w,q] T .

[0119] (2) Control variable module design

[0120] In the AUV control system, the design control variable expression is u k =[τ k ,δ k ] T , these variables are key parameters for adjusting the motion state of AUV, among which δ k represents the angle of the AUV horizontal rudder, τ k Indicates the thrust applied in the longitudinal direction of the AUV.

[0121] (3) Determine the transfer model;

[0122] In the AUV depth control method based on reinforcement learning, since the dynamic model of the AUV is completely unknown, the transfer model cannot be expressed by an exact mathematical equation, but a time-discrete function s is used instead. t+1 =f(s t ,a t ,ξ,Δt) to approximate the state transition relationship. Among them, s t Represents the state variable at the current moment, a t represents the control variable at the current moment, ξ represents uncertain disturbances such as environmental noise, Δt represents the uncertainty of the model error in the transfer model, and s t+1 represents the state variables at the next moment. This transition model can be learned in a data-driven manner and describes how the system state evolves to the next moment given the current state and control variables. By introducing uncertain perturbations, the transition model can better simulate the state transition process in a real environment, thereby enhancing the robustness and adaptability of the control strategy.

[0123] (4) Reward function module setting;

[0124] Determine the reward function: The reward function is used to evaluate the effect of the AUV executing the control variables at a certain moment, represented by reward. This paper proposes a novel piecewise reward function to distribute control rewards to different control stages to achieve a more accurate regulation effect. The specific reward function settings are as follows:

[0125]

[0126] Among them, k1, k2, k3, and k4 represent the weight coefficients of each item (the value of each coefficient is between [0, 1]). r )+k2(θ-θ r ) ensures that the AUV reaches the specified depth while its forward direction is consistent with the x-axis, and k3w+k4q makes the longitudinal speed and rotational angular velocity of the AUV as small as possible.

[0127] (5) Objective function module setting;

[0128] The goal of AUV depth control is to ensure that the AUV can accurately reach and stably maintain the preset depth value. To achieve this goal, we adopt the Markov decision process as the theoretical framework and design an expected cumulative loss function to quantify the control effect. Specifically, this function aims to measure the deviation between the AUV and the specified depth during actual operation, as well as the control cost incurred to reach and maintain that depth. Therefore, in the context of the Markov decision process, our objective function is redefined as: seeking an optimal strategy to minimize this expected cumulative loss function, thereby ensuring that the AUV can complete the depth control task efficiently and accurately.

[0129]

[0130] Among them, J(π) is the objective function of strategy π, which represents the expected cumulative reward; r t represents the reward at step t, provided by the reward function r(s,a); γ∈[0,1] is a discount factor used to weigh short-term and long-term rewards; T is the number of time steps in the task.

[0131] The evaluation network provides state value estimates for the policy network, helping to calculate the advantage function and thus guide policy optimization. Updates to the evaluation network are performed alternately with updates to the policy network to ensure that the two remain synchronized during training. Based on the current state, the value estimate V(s) of the state is output, representing the long-term reward estimate for that state. This is used to optimize the policy network. The evaluation network expression is as follows:

[0132] V(s)=W T h(s)+b

[0133] Among them, V(s) is the value estimate of state s; W is the weight parameter of the network; h(s) is the result of state s being processed by the hidden layer activation function; b is the bias term.

[0134] The policy network is responsible for generating actions and optimizing the policy by maximizing the objective function (including policy improvement and constraints). The policy network takes the state variable as input and generates a probability distribution (Gaussian distribution) for selecting actions based on the current state. It then samples actions based on the distribution. The probability density function of the Gaussian distribution is as follows:

[0135]

[0136] The loss function is used to measure the effectiveness of the current policy and guide policy updates. By limiting the magnitude of policy updates, it indirectly influences the optimization process of the loss function. Specifically, by limiting the difference between the new and old policies, the loss function ensures that excessive policy updates prevent the loss function from falling into a local optimum or causing the algorithm to diverge. This method optimizes the objective through a pruned policy update. The specific loss function expression is as follows:

[0137]

[0138] in, is the strategy importance sampling ratio, which indicates the relative probability of sampling u between the new strategy and the old strategy in state st; A t =Q(s t ,u t )=V(s t ) is the advantage function, which measures the advantage of the current action u relative to the strategy; ∈ is the clipping threshold, which is used to limit the strategy update amplitude and ensure the stability of the optimization process.

[0139] The structure of the evaluation network in this method, the Critic network, is used to evaluate the value of a given state and is typically used for value function estimation in reinforcement learning. The network's input is the state, which is processed through two hidden layers with a tanh activation function, and finally outputs a scalar value representing the evaluation value of the state. The Critic network helps the agent better evaluate the effectiveness of the current strategy by predicting the "value" or "reward" of the environment in a certain state. During training, the network is typically updated by minimizing the error between the predicted value and the actual reward. The output of the Critic network can be used to improve the behavior of the policy network (such as the Actor network) so that the strategy can select better actions to maximize the cumulative reward.

[0140] In this method, the policy network is a Gaussian distributed policy network with a fixed standard deviation and only outputs the mean mu of the action. The evaluation network consists of an input layer, two hidden layers, and an output layer. The activation function of each hidden layer is ReLU, and the output layer uses a sigmoid function to limit mu to between (0,1). In the constructor, the weights and biases of the output layer are scaled to ensure that the action output is small at initialization to prevent excessive action perturbations. In addition, the network contains a learnable parameter action_log_std, which represents the logarithmic standard deviation of the action, which remains unchanged during training. Through the get_dist method, the network returns a Gaussian distribution object based on mu and action_std for generating random actions. The deterministic_act method directly returns the action mean mu for deterministic action selection.

[0141] 2. Perform the AUV depth control training task with disturbances using the PPO algorithm based on reinforcement learning. The specific steps are as follows:

[0142] (1) Parameter setting of PPO training algorithm based on reinforcement learning;

[0143] The maximum number of iterations is M = 200, the maximum number of experimental steps per iteration is T = 10000, the AUV state space dimension state_dim = 12, the action space dimension action_dim = 3, the neural network hidden layer size hidden_size = 400, the maximum action value max_action = 10, and the regularization coefficient l2_reg = 1e -3 , generalized advantage estimation parameter lambd = 0.95, clipping rate clip_rate = 0.01, the maximum value of the experience replay buffer N = 100000, the learning rates of the evaluation network and the policy network are lr_actor = 2e -4 and lr_critic = 2e -3 , the discount factor is γ = 0.99;

[0144] (2) Initialize the evaluation network and the strategy network separately;

[0145] Let ζ and θ represent the weight parameters of the critic network and the policy network, respectively. The critic network's function is described by the parameterized expression Q(s,u|ζ), which evaluates the value of a state s and an action a. The policy network, represented by the parameterized expression μ(s|θ), generates an action policy based on the state s. Initially, the weight parameters of both networks are set to 0, meaning they are in their initial state. We refer to this initial state of the critic network as the current critic network, and similarly, the initial state of the policy network as the current policy network.

[0146] Construct an experience buffer with a maximum value of N = 100,000 and name it the experience queue set of ReplayBuffer to store the experience data during training. At initialization, this queue set is empty.

[0147] (3) Define the relevant AUV motion state variables and initialize all AUV state parameters; randomly initialize the AUV state variables as x0 = [u0, v0, w0, p0, q0, r0] T , and let x t =x0;

[0148] (4) During the training of the control strategy, the environmental interference factor τ is added in a given round d , used to test the anti-interference ability of the control strategy;

[0149] (5) Use the policy network μ(s|θ) to generate the control variable at time t and add exploration noise recorded as u t =μ(s|θ)+Δu t ; AUV passes the obtained dynamic transfer model s t+1 =f(s t ,a t ,ξ,Δt) returns the next moment state s t+1 Use the one-step loss function to calculate the loss function of the transfer step;

[0150] (6) Update the evaluation network weight according to the following formula:

[0151] TD Target t =r t +γV(s t+1 )Critic

[0152]

[0153] (7) The PPO algorithm is used to optimize the target evaluation network weights through the following objective function:

[0154]

[0155] (8) The PPO algorithm is used to optimize the target strategy network weight through the following objective function:

[0156]

[0157] Record the updated policy network as the new current policy network;

[0158] After the update is complete, proceed to step 3-11);

[0159] (9) Let t = t + 1 and determine whether the model training of this episode is completed: if t < n_steps, then return to step (5) and the AUV continues the next model training step; otherwise, go to step (12) to perform the model training of a new episode;

[0160] (10) If a new depth control model training is required, the AUV will select a new initial position and start the above steps again to train the model. Otherwise, the iteration ends. When the iteration ends, the current strategy network μ(s t |θ) is the final trained policy network, denoted as μ * (s t |θ). After multiple iterations of optimization, the network is able to t Output the optimal control action a t , thereby achieving depth control of the AUV.

[0161] The present invention proposes a PPO model-free underwater AUV anti-interference depth control method based on reinforcement learning. The simulation effect analysis is as follows:

[0162] Although the present invention aims to achieve fixed depth control without relying on prior knowledge of the AUV dynamics model, in order to evaluate the effectiveness of the proposed method, the motion of a real AUV when performing a heading mission is simulated based on the dynamics equations of a common AUV. The kinematic and dynamic models of a common AUV are shown below:

[0163]

[0164] In the above dynamic equations, ξ, η, and ζ represent the position of the AUV in the inertial coordinate system. φ, θ, and ψ represent the roll, pitch, and yaw angles of the AUV. u, v, and w represent the velocity of the AUV in the rotating coordinate system. p, q, and r represent the angular velocity of the AUV in the rotating coordinate system. X, Y, and Z represent the forces exerted by the AUV's actuators in the inertial coordinate system. K, M, and N represent the torques output by the actuators in the inertial coordinate system at the moment the AUV is moving. g * and f * Represents the effects of residual buoyancy and environmental disturbance on AUV. * Represents the inertia of the AUV at a certain moment. The meanings of other parameters are shown in Table 1. W and B represent the gravity and buoyancy of the AUV. Assuming that the AUV's own coordinate system coincides with the buoyancy center, then x B =y B =z B =0; center of gravity (x G ,y G ,z G ) T =(0,0,0.01)T .

[0165] REMUS model AUV; Table 1 is the model parameter table of REMUS model AUV.

[0166]

[0167] Attachment Figure 5 This is the simulation result of the method of the present invention on the AUV depth control problem. Figure 5 It shows the changing trend of the motion state of the AUV during the training process of the method of the present invention. As can be seen from the figure, in the 0-4000 training times, the underwater robot has been looking for the optimal action strategy, and the reward value varies greatly. The method is close to convergence at around 4000 training iterations, and the reward value converges to a certain range. According to the error between the current position of the AUV and the target depth, the control process is dynamically divided into multiple stages, so that the reward function can guide the learning strategy more specifically, indicating that the reward function is reasonably designed. Compared with the existing technology, a different reward function curve is set for each stage. For example, in the coarse adjustment stage, rapid approach to the target is encouraged, and in the fine adjustment stage, stability and minimum error are emphasized, which solves the problem of "one-size-fits-all" reward design in traditional methods; Figure 5 The figure shows the changing trends of various states when the final policy network obtained using the method of the present invention controls an AUV. The figure shows that all states have reached convergence, and the depth has converged to the specified target depth. To address the shortcomings of existing technologies, an error threshold micro-reward is introduced when approaching the target depth, allowing the policy to focus on detailed adjustments and avoiding "jumpy" over-adjustments. The changing trends of the control inputs and outputs show that control variables such as depth, pitch angle, and reward value remain in a stable convergence state after the AUV reaches the target depth.

[0168] These results fully demonstrate the effectiveness and superiority of the proposed piecewise reward function in the depth control task of the REMUS AUV. Even under non-ideal ocean conditions, such as those simulating complex water disturbances, the designed reinforcement learning control strategy achieves rapid and stable convergence, demonstrating excellent adaptability and convergence robustness. The AUV's actual operating depth is consistently maintained near the preset target depth, while the fluctuations in key state variables such as pitch angle and longitudinal velocity are significantly reduced, significantly improving overall control accuracy and stability.

[0169] This invention constructs a phased, multi-level reward function system to dynamically adjust the incentive strategy as the AUV gradually approaches the target depth. For example, during the coarse adjustment phase, when moving away from the target, rapid response is enhanced; during the fine and fine adjustment phases, when approaching the target, stability and accuracy are emphasized. This creates a coarse-to-fine, step-by-step convergence control path throughout the entire control process. This design effectively improves training efficiency, enabling the model to achieve a high-precision, low-volatility control strategy while significantly reducing training time.

[0170] In typical test tasks, the control strategy of the present invention achieved significantly lower average steady-state error after reaching the target depth than the method described in the comparative document. The control process was smoother, with no significant oscillation or overshoot, demonstrating enhanced control accuracy and robustness. Experiments also showed that the method maintained good performance under various operating conditions, including varying initial depths and disturbance intensities, demonstrating enhanced generalization and engineering adaptability, and promising broad practical application prospects.

[0171] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0172] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the claims.

[0173] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings and that various modifications and variations can be made without departing from the scope thereof, which is limited only by the appended claims.

Claims

1. A model-free AUV depth control method based on reinforcement learning, characterized in that: The method comprises: Establish AUV dynamics model; Based on the AUV dynamics model, the Markov decision process is used to establish the interaction model between the AUV and the environment, determine the action space variables and state space variables of the AUV control system, and establish the AUV depth control objective function and piecewise reward function; A neural network model is established, and a policy network and an evaluation network are established separately. A continuous PPO algorithm is used to train the control strategy model to obtain the control strategy parameters of the AUV. The control strategy parameters of the AUV are used to achieve fixed-depth control of the AUV on the vertical plane. Through reinforcement learning, the AUV updates the training network at a fixed step size during the set depth control training until convergence, thereby obtaining the final policy network for the predetermined depth.

2. The method according to claim 1, characterized in that The action space variable of the AUV control system is determined as u k =[τ k ,δ k ] T , where τ k , δ k are the propeller thrust and rudder angle of the AUV, respectively. The subscript k represents the kth time step; τ k , δ k The value ranges are [0,τ max ] and [-δ max ,δ max ], τ max , δ max are the maximum propeller thrust and horizontal rudder angle, respectively, which are used to limit the range of the action space to comply with physical constraints.

3. The method according to claim 1, characterized in that The state space variable of the AUV control system is determined as x=[zz r , θ-θ r , w, q], where z represents the current depth of the AUV in the inertial coordinate system, θ represents the angle between the AUV's forward direction and the x-axis in the inertial coordinate system, and q represents the angular velocity of the AUV's y-axis rotation in its own coordinate system. r represents the pitch angle error, w represents the vertical speed of the AUV in its own coordinate system, zz r Indicates the depth error between the AUV's current depth and the target depth, used to directly guide and control the target.

4. The method according to claim 1, wherein The AUV depth control objective function is established as follows: Among them, J(π) is the objective function of strategy π, which represents the expected cumulative reward; r t represents the reward at step t, provided by the reward function r(s,a); γ∈[0,1] is a discount factor used to weigh short-term and long-term rewards; T is the number of time steps in the task, e depth ,e theta ,e psi ,u control They represent depth error, pitch angle error, heading error, and control input error respectively.

5. The method according to claim 1, wherein The establishment of the piecewise reward function is specifically as follows: Among them, k1, k2, k3, and k4 represent the weight coefficients of each item, k1(zz r )+k2(θ-θ r ) ensures that the AUV reaches the specified depth while its forward direction is consistent with the x-axis, and k3w+k4q makes the longitudinal speed and rotational angular velocity of the AUV as small as possible.

6. The method according to claim 1, characterized in that The input of the policy network is the state variable, and based on the current state, a probability distribution for selecting an action is generated.

7. The method according to claim 6, characterized in that The evaluation network outputs a value estimate of the current state based on the state, which represents the long-term reward estimate of the state and is used to optimize the policy network.

8. The method according to claim 1, characterized in that The continuous PPO algorithm is used to train the control strategy model, specifically: Set the size of the experience buffer used for batch sampling during training; Define the relevant AUV motion state variables and initialize all AUV state parameters; randomly initialize the AUV state variables as x0 = [u0, v0, w0, p0, q0, r0] T , and let x t =x0; At time step t, based on the current state state, the policy network generates a deterministic action μ(s t |θ), and obtain the final control action u by adding exploration noise Δu t =μ(s t |θ)+Δu t ; The policy network outputs the action probability distribution π(a|s t ), from which action a is sampled t ~π(a|s t );Control action u t Or sampling action a t Input into the AUV dynamics model, interact with the environment and update to the next state s t+1 , and get instant rewards t ; interactively generated empirical data (s t ,u t ,r t ,s t+1 ) is stored in the experience buffer R for subsequent policy optimization and value estimation; in this way, the policy network strikes a balance between exploration and exploitation, gradually optimizing the control strategy to achieve the target task; Count the training step length n-steps and update the current network model. When n-steps> Thornical, update the evaluation network weight according to the following formula: TD Target t =r t +γV(s t+1 )Critic Among them, r t is the reward at the current moment, γ is the discount factor, V(s t+1 ) is the estimate of the state at the next moment; λ is the regularization coefficient, w i is the weight in the Critic network, α is the learning rate; Update the target evaluation network weight and target strategy network weight according to the following formula: In each training iteration, the AUV randomly selects an initial position and begins a new round of fixed-depth control tasks. This process aims to enhance the generalization ability of the policy network through diversified initial conditions, enabling it to adapt to different starting states. If the predetermined training termination condition is not reached in the current iteration, the next iteration will be continued; otherwise, the training process ends. At the end of the iteration, the current policy network μ(s t |θ) is the final trained policy network, denoted as μ * (s t |θ); The network is optimized through multiple iterations and is able to t Output the optimal control action a t , thereby achieving depth control of AUV; If the training termination conditions are met, training can be stopped. The training target requirements are that the depth error is less than 0.01 meters, the pitch angle error is less than 0.01 degrees, and the training is stable for more than 1000 steps. At this time, the training is considered to have met the target requirements. Other termination conditions are safety assessment criteria for the navigation process. The specific training termination conditions are as follows:

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model-free AUV depth control method based on reinforcement learning is implemented according to any one of claims 1 to 8.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the reinforcement learning-based model-free AUV depth control method according to any one of claims 1 to 8 is implemented.

Images