AUV fault-tolerant control method based on near-end strategy optimization and tubular model prediction

Abstract

The invention relates to a fault-tolerant control technology, and aims to provide an AUV fault-tolerant control method based on near-end strategy optimization and tubular model prediction. Comprising the following steps: establishing an underwater robot mathematical model, and discretizing into a nominal system and a real system; nonlinear model predictive control is achieved for a nominal system, and a nominal control law is obtained through solution and rolling optimization; introducing a near-end strategy optimization algorithm based on tubular model predictive control, outputting an auxiliary feedback control law based on an actor and commentator network, and adding the auxiliary feedback control law with a nominal control law to obtain a fault-tolerant control law for the underwater robot propeller; keeping the parameters of the nominal control law unchanged; meanwhile, an auxiliary feedback control law is adaptively updated based on a strategy obtained through near-end strategy optimization training, and online fine adjustment of fault-tolerant control of the propeller is achieved. Influences of faults and delay of the propeller are passively processed by means of inherent robustness of the controller, the design of a fault device control law is simplified while the stability is guaranteed, and the conservative property of a classical tubular model is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of fault-tolerant control of autonomous underwater vehicles (AUVs), and more specifically to a fault-tolerant control method for AUVs based on proximal policy optimization (PPO) and tube model predictive control (Tube MPC), which is used to solve the problem of stable control of AUVs under propeller failure and delay conditions. Background Art

[0002] With the rapid development of science and technology, autonomous underwater vehicles (AUVs), or underwater robots (UUVs), have become essential tools for exploring the ocean and protecting marine resources. Research on AUV control technology is a crucial branch of marine science and technology and a driving force behind marine innovation. In complex marine environments, the propellers of AUVs are susceptible to wear, aging, and entanglement with foreign objects, leading to sudden or progressive failures. Fault-tolerant control technology, as a key technology for improving the reliability and safety of AUVs, has important practical implications for enhancing the ability to develop marine resources.

[0003] Model predictive control is increasingly being used in fault-tolerant control schemes for underwater vehicles, often within the framework of active fault-tolerant control. While effective, these designs are inherently suboptimal due to the necessary decoupling between fault detection, isolation, and recovery architectures within the controller. In contrast, passive fault-tolerant control is a control strategy that does not require real-time fault detection and diagnosis. When a system fault occurs, it relies on a pre-designed robust controller to maintain system stability and performance through fault tolerance and inherent robustness.

[0004] Tube-based Robust Model Predictive Control (RMPC) is an advanced control method for uncertain systems. Its core idea is to ensure that the system state always remains within a predefined "safety pipe" within the prediction time domain through dynamic constraints, thereby achieving robustness against disturbances and model uncertainties. To address the complex design of active fault-tolerant control, passive fault-tolerant control methods based on tube-based model prediction have been applied. Its advantage / feature is that it conservatively handles uncertainty through the robustness of the controller itself. However, this method still has the following shortcomings: First, the fixed parameters of the tube lack the ability to adaptively adjust, resulting in conservative control performance and an inability to dynamically adapt to environmental changes; second, the impact of thruster delay is not fully considered. Existing methods usually incorporate delay into nominal system modeling, but the actual delay is difficult to accurately predict.

[0005] Therefore, it is urgent to propose a fault-tolerant control method that can dynamically adjust pipe parameters and jointly handle faults and delays to meet the actual needs of passive fault-tolerant control of underwater robots. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and propose a thruster fault-tolerant control method based on proximal strategy optimization and tubular model prediction.

[0007] To solve the technical problem, the solution of the present invention is:

[0008] A thruster fault-tolerant control method based on proximal strategy optimization and tubular model prediction is provided, comprising:

[0009] (1) Establish a mathematical model of the underwater robot including kinematic model and dynamic model, and use Euler method and Taylor expansion to linearly discretize it into a nominal system without external interference and a real system with bounded interference;

[0010] (2) The fault-tolerant control of the underwater robot propulsion system consists of two parts: the outer loop and the inner loop. The former implements nonlinear model predictive control for the nominal system. By solving the nonlinear model predictive optimization problem, the nominal control law at the current sampling moment is obtained through rolling optimization. The latter introduces a proximal policy optimization algorithm based on the tubular model predictive control. By calculating the state difference between the nominal system and the real system, it is used as the loss function of the proximal policy optimization algorithm. After training a strategy that minimizes the difference between the two systems, an auxiliary feedback control law is output based on the actor-critic network.

[0011] (3) The nominal control law and the auxiliary feedback control law are added together to obtain the fault-tolerant control law for the underwater robot thruster. In the process of implementing fault-tolerant control, the parameters of the nominal control law are kept fixed. At the same time, the auxiliary feedback control law is adaptively updated based on the strategy obtained by proximal strategy optimization training, thereby achieving online fine-tuning of the thruster fault-tolerant control.

[0012] Description of the invention principle:

[0013] This paper proposes a tubular model predictive fault-tolerant control method based on proximal policy optimization to address the control challenges of underwater robot propulsion systems under dynamic and uncertain conditions, including faults and time delays. This method innovatively integrates proximal policy optimization with tubular model predictive control to create a novel passive fault-tolerant control architecture.

[0014] This invention combines nonlinear model predictive control with an auxiliary feedback control law, treating the effects of environmental disturbances, fault disturbances, and time delay disturbances as part of the uncertain dynamics. It relies on the robustness of the controller itself to control the real system closer to the nominal system. Furthermore, this invention employs proximal strategy optimization to train the optimal strategy and adaptively adjusts the parameters of the auxiliary feedback control law to reduce the conservatism of tubular model predictive control.

[0015] Compared with the prior art, the present invention has the following beneficial effects and advantages:

[0016] 1. The proposed tubular model predictive fault-tolerant control method based on proximal strategy optimization relies on the inherent robustness of the controller to passively handle the impact of thruster failures and delays, while ensuring its stability and simplifying the design of the fault controller control law.

[0017] 2. This invention utilizes proximal strategy optimization to adaptively adjust the parameters of the auxiliary feedback control law, reducing the conservatism of the classic tubular model predictive control method. Furthermore, it fully considers the impact of thruster delay on the control system, complementing the existing tubular model predictive control framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a block diagram of the control system implemented in the present invention.

[0019] Figure 2 This is the architecture diagram of the proximal strategy optimization-tubular model prediction algorithm designed for the present invention. DETAILED DESCRIPTION

[0020] First of all, it should be noted that the present invention relates to fault-tolerant control technology and signal processing technology. In the process of implementing the present invention, the principles and applications of multiple basic algorithms may be involved. The applicant believes that after carefully reading the application documents and accurately understanding the implementation principles and invention objectives of the present invention, in combination with the existing publicly known technology, those skilled in the art can fully use the algorithm writing capabilities they have mastered to implement the present invention. All that is mentioned in the application documents of the present invention fall into this category, and the applicant does not list them one by one. In addition, the implementation of the patent of the present invention relies on a variety of computers, single-chip microcomputers, sensor modules and communication modules, and these instruments are all existing technologies, and there are mature products available on the market. Secondly, the use of application-specific integrated circuits (including but not limited to ASICs) or field programmable gate arrays (including but not limited to FPGAs) to deploy the fault-tolerant control method described in the present invention will help to achieve hardware acceleration.

[0021] The implementation process of the present invention is described in detail below with reference to the accompanying drawings.

[0022] 1. Control Framework Description

[0023] The control system block diagram implemented by the present invention is as follows Figure 1 As shown, on this basis, we further propose Figure 2 The design architecture of the proximal policy optimization-tubular model prediction algorithm shown.

[0024] In the present invention, the proximal policy optimization algorithm is only responsible for the training of the auxiliary feedback control law in the tubular model predictive control, hoping to control the real system to fluctuate around the nominal system through the auxiliary feedback control law to maintain dynamic stability. Within the framework of the proximal policy optimization algorithm, the actor network uses the predicted system trajectory to calculate the control input and generate a control strategy; this strategy can compensate for the effects of time delays, faults and environmental interference, and maintain the stability and task completion ability of the robot. The critic network is used to evaluate the effect of the control input selected by the actor network. It calculates the value function of the state-action pair, evaluates the expected reward that can be obtained after performing a certain action, and feeds this evaluation result back to the actor network to help the actor network adjust its strategy. The advantage function calculated by the critic network provides the actor network with an improvement direction, allowing the actor network to continuously optimize its strategy, so that the control system can gradually improve its control performance when facing faults or time delay interference.

[0025] Based on the above design, the present invention proposes an AUV thruster fault-tolerant control system based on proximal strategy optimization and tubular model prediction, including a hardware system module and a software function module.

[0026] The hardware system module includes sensors, signal processing and transmission devices, memory and processor. Among them, several sensors are installed on the underwater robot to collect status data generated during operation; the signal processing and transmission device is used to collect the status data collected by the sensors in real time, and transmit it to the processor through a communication interface (including but not limited to a serial port, a network port, and a bus) after pre-processing; the memory is used to store the historical status data of the underwater robot and deploy software function modules; the processor is configured to be able to execute instructions, call the software function modules from the memory, and use the real-time status data and historical status data to implement the AUV thruster fault-tolerant control method, and finally obtain the control signal when the underwater robot fails during operation. The sensors are attitude sensors and position sensors installed on the underwater robot. The collected status data includes at least the underwater robot's position / angle, speed / angular velocity, acceleration / angular acceleration and thruster control signal, and the sensor data is transmitted in real time via the MavLink communication protocol. The processor can be a single-chip microcomputer, Raspberry Pi or FPGA.

[0027] The software functional modules include a data processing module, a model building module, a proximal policy optimization module, and a controller module. The data processing module is used to receive signals from various sensors and convert and process them into applicable data formats. The model building module is used to construct a mathematical model of the underwater robot, including a kinematic model and a dynamic model, and discretize the nominal system and the real system. The proximal policy optimization module includes a tubular model prediction network and an actor-critic network, and adopts a Markov decision process corresponding to underwater robot control. The controller module is used to add the nonlinear model prediction nominal control law for the nominal system to the policy adaptive update auxiliary feedback control law obtained based on proximal policy optimization training to obtain a fault-tolerant control law for the AUV thruster, thereby achieving online fine-tuning of the fault-tolerant control.

[0028] 2. Specific Implementation of AUV Fault-Tolerant Control Method

[0029] 1. Establish a mathematical model of the underwater robot including kinematic model and dynamic model, and use the Euler method to linearly discretize it into a nominal system without external interference and a real system with bounded interference.

[0030] In establishing the mathematical model of the underwater robot, the kinematic and dynamic models were linearly discretized into a nominal system without external disturbances and a system with bounded disturbances using the Euler method and Taylor expansion. The hydrodynamic coefficients in the dynamic model were calculated using a combination of computational fluid dynamics simulation and experiments.

[0031] The nominal system is represented as follows:

[0032]

[0033] in, is the derivative of the nominal system state; A p is the state transfer matrix; B p Assign the matrix to the thrust; is the state of the nominal system at the current time; is the control input of the nominal system at the current time;

[0034] The state equation is linearly discretized using Euler's formula:

[0035]

[0036] Among them, A d is the discretized state transfer matrix; B d is the discretized thrust distribution matrix;

[0037] Substituting into the above formula we can get:

[0038]

[0039] Where k is the time; It refers to the nominal system state at the next moment after discretization; It refers to the nominal system state at the current moment after discretization; It refers to the control input of the nominal system at the current moment after discretization;

[0040] Introducing environmental interference F env 、Fault interference F fau Interference with time delay F delay , then the real system is expressed as follows:

[0041] x(k+1)=A d x(k)+B d u(k)+F delay +F fau +F env

[0042] Among them, x(k+1) is the state of the real system at the next moment; x(k) is the state of the real system at the current moment; u(k) is the control input of the real system at the current moment;

[0043] Environmental interference F env : Specifically includes environmental forces caused by wind, waves and currents, expressed as thrust / torque;

[0044] Fault interference F fauFailures during underwater robot operation are considered efficiency losses in the propulsion system. The propulsion system consists of several propellers and corresponding motors, and may be affected by both sudden and slow-changing faults. Assuming that after a fault occurs, the fault coefficient matrix w represents the severity of the fault, and the original thrust is u, the actual thrust output of the propeller is:

[0045] u fau =w×u

[0046] The thrust is multiplied by the thrust distribution matrix w to obtain the fault interference F fau .

[0047] Delay Interference F delay This paper innovatively incorporates propeller delay into the real-world underwater robot system. This delay primarily results from the communication time between the controller and the propeller, as well as the controller's computational time. This paper uses a formula to adjust the thrust distribution matrix during model discretization to account for the impact of this delay.

[0048] B d u(k) is converted to B d0 (Δt)u(k)+B d1 (Δt)u(k-1), where

[0049]

[0050] F delay =B d0 (Δt)u(k)+B d1 (Δt)u(k-1)-B d u(k)

[0051] Due to the influence of time delay, the thrust distribution matrix is ​​modified into two parts, where B d0 (Δt) corresponds to the thrust distribution matrix when the controller executes the current control input, B d1 (Δt) corresponds to the thrust distribution matrix when the controller executes the control input at the previous moment in the current time step.

[0052] 2. The fault-tolerant control of the underwater robot thruster consists of two parts: an outer loop and an inner loop. The former implements nonlinear model predictive control for the nominal system. By solving the nonlinear model predictive optimization problem, rolling optimization is performed to obtain the nominal control law at the current sampling moment. The latter introduces a proximal policy optimization algorithm based on tubular model predictive control. By calculating the state difference between the nominal system and the real system, it is used as the loss function of the proximal policy optimization algorithm. After training a strategy that minimizes the difference between the two systems, an auxiliary feedback control law is output based on the actor-critic network.

[0053] 2.1 Nonlinear Model Predictive Control of the Outer Loop

[0054] (1) is the tracking target trajectory x of the control nominal system d , set the nonlinear model prediction optimization problem as:

[0055]

[0056] in,

[0057]

[0058]

[0059] In the above formulas, is the optimization variable, which refers to the control input at time h during the optimization solution process; is the nominal system state; is the nominal system control input; m is the control time domain; p is the prediction time domain; t is the current time; τ is the time variable (for integration); P is the terminal weight matrix, positive definite and symmetric; is the desired state (reference trajectory); is the desired control input; Q and R are the weighted matrices of state and control input respectively, both are positive definite symmetric matrices; The feasible set in the state Inside, In the control input constraint set Inside; represents the stage cost; T is the symbol of matrix transpose;

[0060] (2) In the process of solving the nonlinear model prediction optimization problem, the following constraints need to be met:

[0061]

[0062] In the above formulas, is the derivative of the nominal system state; f is the system dynamic function; Input constraint set for control; is the feasible set of states; Ω is the terminal area;

[0063] Equation (a) indicates that the state variables in the prediction model are initialized from the state variables of the actual system. Equation (b) represents the constraints on the input variables in the prediction model, which are determined by the constraints of the actual system. Equation (c) indicates that the predicted control variable is the last constant solution in the control horizon. Equation (d) represents the terminal inequality constraint, which ensures that the system state returns to the terminal region Ω at the end of the finite horizon. Both Ω and P need to be calculated offline to ensure system stability.

[0064] 2.2 Tubular Model Predictive Control of the Inner Loop

[0065] If there exists a matrix E>0 and M>0, the following linear matrix inequality is satisfied:

[0066]

[0067] Then F=EM -1 , making the system stable;

[0068] By solving the following optimization problem, we obtain the robust positive invariant set S:

[0069]

[0070]

[0071] The state constraints and input constraints of the nominal system need to satisfy:

[0072]

[0073] In the above formulas, E is a symmetric positive definite matrix in LMI, which is used to construct the feedback gain matrix; M is an auxiliary variable in LMI, which must satisfy M>0; LMI refers to linear matrix inequality; A is the state transfer matrix (system matrix) of the discrete-time system; B is the input matrix of the discrete-time system; T is the sign of the matrix transpose; F is the feedback gain matrix; S is the robust positive invariant set; s is a vector, including s1 to s6, which is the decision variable of the optimization problem; s i is the polyhedron boundary parameter that defines S and the optimization variable; D is the perturbation set (representing the external perturbations that the system may be subjected to); is the feasible set of states of the nominal system; X is the feasible set of states of the real system; is the control input constraint set of the nominal system; U is the input constraint set of the real system.

[0074] 2.3 The proximal strategy optimization algorithm of the inner loop, the specific implementation process includes:

[0075] (1) Data preprocessing

[0076] By executing the current policy π in the environment θ To collect a set of interaction data, including state s t 、Action a t , reward r t and the next state s t+1 ;

[0077] Importance sampling method is used to calculate probability ratios to improve data utilization:

[0078]

[0079] Where r t (θ) is the importance sampling ratio, which is used to correct the difference in action selection probability under the new and old strategies; θ is the parameter of the current strategy; t is the current time step; a t The action taken for the current time step; s t is the current state; θ old is the parameter of the old strategy; π θ For the current strategy; For the old strategy.

[0080] Normalize the state data to have a mean of zero and a variance of 1, and perform a discounted accumulation calculation on the reward signal:

[0081]

[0082] Among them, R t is the cumulative discounted reward starting from time step t; γ is the discount factor, γ∈(0,1); k is the offset time step, indexing the future reward; t is the current time step;

[0083] (2) Calculate the advantage estimate

[0084] The advantage function is calculated based on the generalized advantage estimation method to measure how good the current action is relative to the average level. The calculated advantage estimate is used for subsequent strategy optimization.

[0085] The generalized odds estimate is defined as follows:

[0086] A t =δ t +(γλ)δ t+1 +…(γλ) T-t+1 δ T-1

[0087] δ t =r t +γV(s t+1 )-V(s t )

[0088] Among them, A t is the advantage function, which is used to measure the quality of the current action relative to the average strategy; γ is the discount factor; δ t is the time difference error; T is the trajectory end time or truncation time; t is the current time step; λ is the weight parameter for balancing deviation and variance; V(s t ) is the state value function;

[0089] (3) Optimize the objective function

[0090] Use the pruning strategy to optimize the objective function and prevent the strategy from updating too much.

[0091] The optimization objective function of proximal policy optimization is defined as:

[0092]

[0093] Among them, L(θ) is the optimization objective function, which represents the loss function under the parameter θ; is the expected value, which represents the average of time step t; r t (θ) is the return or reward estimate at time step t under the parameters θ; A t is the advantage function; ∈ is the clipping range; clip(·) means clipping the probability ratio; the objective function contains both the value function loss term and the entropy regularization term;

[0094] (4) Update strategy

[0095] Update the policy parameters using the gradient ascent method, and then repeat the above steps with the new policy parameters until the policy performance no longer improves or the set number of iterations has been reached.

[0096] 2.4. Before using the proximal policy optimization algorithm, train it as follows:

[0097] The interaction data of the underwater robot during normal operation and under different fault delay conditions, including state, action, reward and other information, are collected and preprocessed. The data are normalized and combined into multiple sets of matrix data as training data, and a training set and a validation set are established. The training set is used to train the policy network based on proximal policy optimization to ensure that the policy can effectively adapt to the control requirements under different states, and the generalized advantage estimate is calculated to optimize the policy. The clipping policy loss function is used in the training process to limit the policy update amplitude and improve the stability of the training. After the training is completed, the policy parameters of the optimal model are retained and applied to the construction of the auxiliary feedback control law.

[0098] Before the underwater robot is put into actual operation, the trained strategy is deployed in the embedded computing platform of the underwater robot system to meet the computing requirements of real-time control; during the operation of the system, the current state data is used to input the trained proximal strategy optimization policy network to obtain the optimal control action, and it is combined with traditional control methods to form an enhanced feedback control law; during the operation of the underwater robot, the execution effect of the control strategy is continuously recorded, the actual operation data is collected, and the strategy network is fine-tuned online using this data to improve its adaptability; when the robot encounters new faults or environmental changes, the updated data can be used to perform incremental training on the proximal strategy optimization strategy to give it stronger generalization capabilities, thereby forming a stable and reliable fault adaptive control strategy.

[0099] 2.5 In the implementation of the proximal policy optimization algorithm, the Markov decision process is used:

[0100] (1) Confirming the observation space based on the underwater robot's information

[0101] The information observed by the underwater robot consists of four parts: its true position, its true mission status, the position of the nominal system, and the current mission status of the nominal system, namely:

[0102]

[0103] In the formula, o i is the observation space; x is the state of the real system; u is the control input of the real system; is the state of the nominal system; is the control input of the nominal system;

[0104] (2) Confirm the motion space at each time point by calculating the thrust correction value of the thruster:

[0105] Taking a six-thruster underwater robot as an example, the action at each time point includes the thrust correction value u of the six thrusters c However, in order to improve the training speed and ensure the stability of control, we do not directly optimize the training results through the proximal strategy to obtain u c Instead, we first obtain F at each moment through training, and then calculate u based on the following formula: c .

[0106]

[0107] Where u c is the thrust correction value of the propeller; F is the auxiliary feedback control gain; x is the state of the real system; is the state of the nominal system;

[0108] (3) During training, rewards are immediately available once a new action is taken. Since rewards directly enable the agent to optimize its decision-making policy, the reward function should be designed based on the constraints and objectives of the underwater robot control problem. The instantaneous reward obtained by the underwater robot during training depends on the error between its true system and the nominal system. The smaller the error, the higher the reward.

[0109]

[0110] Where r i is the reward function; x is the state of the real system; is the state of the nominal system;

[0111] The instantaneous reward obtained during training depends on the error between the true system and the nominal system. The smaller the error, the higher the reward.

[0112] 3. The nominal control law and the auxiliary feedback control law are added together to obtain a fault-tolerant control law for the underwater robot thruster. During the fault-tolerant control process, the parameters of the nominal control law are kept fixed. At the same time, the auxiliary feedback control law is adaptively updated based on the strategy obtained from the proximal policy optimization training, thereby achieving online fine-tuning of the thruster's fault-tolerant control.

[0113] In this invention, the control law of the nominal system is obtained by solving the nonlinear model prediction optimization Then, based on the tubular model predictive control rolling optimization and the introduction of the proximal strategy optimization algorithm in the fault-tolerant control law design to achieve adaptive adjustment of the auxiliary feedback control law, the fault-tolerant control law of the underwater robot is obtained; that is, the sum of the nonlinear model predictive control law and the auxiliary feedback control law. The specific calculation follows the following formula:

[0114]

[0115] Where u is the control law of the real system; is the control law of the nominal system; u c is the auxiliary feedback control law.

[0116] 3. Specific application examples

[0117] This example utilizes the proposed method for predictive fault-tolerant control of an underwater robot using a tubular model based on proximal policy optimization. This method implements fault-tolerant control of the underwater robot, encompassing four state types: sudden failure, gradual failure, short delay, and long delay. Propeller failures and delays are artificially configured in the control system to simulate their occurrence without damaging the robot. The underwater robot is affected by wave disturbances during its motion.

[0118] This embodiment builds a mathematical model of an underwater robot in a Python environment with reference to the above content, in order to verify the specific feasibility and corresponding technical effects of the present invention. Specifically, it includes the following steps:

[0119] In the first step, the relevant operating conditions were set in the simulation model: the underwater robot numbered A was used to verify the method, the sudden fault was set to a 20% loss of propeller efficiency, the gradual fault was set to a propeller efficiency loss gradually increasing from 20% to 50%, the short delay was set to 0.01s, and the long delay was set to 0.1s;

[0120] Step 2: The hydrodynamic coefficient of the underwater robot is obtained by combining computational fluid dynamics simulation with experiments;

[0121] Step 3: Based on the Euler method and Taylor expansion, the mathematical model of the underwater robot is converted into the form of state equation;

[0122] In the fourth step, the environment, faults, and time delay disturbances that are considered as part of the uncertain dynamics of the real system are ignored, and the nominal control law is calculated under the constraints by predicting the optimization problem through the nonlinear model;

[0123] Step 5: The motion data of the underwater robot is collected through sensors. After sampling, the optimal strategy of the auxiliary feedback control law is obtained based on the proximal strategy optimization algorithm training.

[0124] Step 6: Deploy the optimal strategy in the control system of the underwater robot and adaptively adjust the auxiliary feedback control law;

[0125] Step 7: The passive fault-tolerant control law is obtained by adding the nonlinear model predictive control law and the auxiliary feedback control law;

[0126] In step 8, follow steps (2)-(7) to obtain the control law at each moment, and convert it into a control signal according to the propeller parameters of the underwater robot to achieve fault-tolerant control of the underwater robot.

[0127] This example first models environmental disturbances, faults, and time delays as interfering forces in the real system and handles them passively. Next, a nonlinear model predictive control (MPC) optimization problem for the nominal system is designed. Strict constraints are established, and a rolling optimization approach is used to obtain the nominal control law at each sampling moment, along with the corresponding center trajectory of the pipe. A proximal policy optimization algorithm is then used to calculate the advantage estimate and optimization objective function, and the policy is updated. Based on this strategy, the auxiliary feedback control law is adaptively adjusted to guide the real system to fluctuate around the center of the pipe, achieving dynamic stability for the underwater robot under the influence of faults and time delays.

[0128] In the Python simulation environment, simulation was performed according to the steps in the specific application example. The data obtained from the simulation experiment are shown in Tables 1 and 2.

[0129] Table 1. Tracking error of underwater robot control under short delay

[0130]

[0131] Table 2 Underwater robot control under long delay

[0132]

[0133] In the above table, “ / ” indicates control method failure, and “|” separates the RMS errors of the three stages. The initial stage is affected only by delay, the second stage is affected by delay and sudden faults, and the third stage is affected by delay and slowly deteriorating faults.

[0134] The above simulation experiments are of great help in solving the problem of fault-tolerant control of underwater robots with fault uncertainty and time delay uncertainty. According to the actual test verification of the applicant's research team in actual application scenarios, the present invention is also effective for the fault-tolerant control method of underwater robots.

Claims

1. A fault-tolerant control method for AUV based on proximal strategy optimization and tubular model prediction, characterized in that: include: (1) Establish a mathematical model of the underwater robot including kinematic model and dynamic model, and use Euler method and Taylor expansion to linearly discretize it into a nominal system without external interference and a real system with bounded interference; (2) The fault-tolerant control of underwater robot thrusters consists of two parts: the outer loop and the inner loop; The former implements nonlinear model predictive control for the nominal system. By solving the nonlinear model predictive optimization problem, the nominal control law at the current sampling moment is obtained through rolling optimization. The latter introduces a proximal policy optimization algorithm based on the tubular model predictive control. The state difference between the nominal system and the real system is calculated and used as the loss function of the proximal policy optimization algorithm. After training a strategy that minimizes the difference between the two systems, an auxiliary feedback control law is output based on the actor-critic network. (3) Add the nominal control law and the auxiliary feedback control law to obtain the fault-tolerant control law for the underwater robot propulsion system; In the process of implementing fault-tolerant control, the parameters of the nominal control law are kept fixed; at the same time, the auxiliary feedback control law is adaptively updated based on the strategy obtained from the proximal strategy optimization training, thereby realizing online fine-tuning of the thruster fault-tolerant control.

2. The method according to claim 1, characterized in that The nominal system is expressed as: in, is the derivative of the nominal system state; A p is the state coefficient matrix; B p Assign the matrix to the thrust; is the state of the nominal system at the current time; is the control input of the nominal system current time; The real system is represented as: x(k+1)=Ax(k)+B d u(k)+F delay +F fau +F env Among them, x(k+1) is the state of the real system at the next moment; k is the moment; A is the discretized state coefficient matrix; B d is the discretized thrust distribution matrix; x(k) is the state of the real system at the current moment; u(k) is the control input of the real system at the current moment; F env is the environmental interference, F fau is the fault interference, F delay It is time delay interference.

3. The method according to claim 2, characterized in that The environmental interference F env Specifically, they include environmental forces caused by wind, waves and currents, expressed as thrust / torque; The fault interference F fau Specifically expressed as the product of the thrust actually output by the thruster and the thrust distribution matrix; The delay interference F delay Coming from the communication time between the controller and the thruster and the calculation time of the controller, the thrust distribution matrix is ​​adjusted by the formula method to represent the influence of time delay.

4. The method according to claim 1, wherein The nonlinear model predictive control of the outer loop specifically includes: (1) is the tracking target trajectory x of the control nominal system d , set the nonlinear model prediction optimization problem as: in, In the above formulas, is the optimization variable, which refers to the control input at time h during the optimization solution process; is the nominal system state; is the nominal system control input; m is the control time domain; p is the prediction time domain; t is the current time; τ is the time variable; P is the terminal weight matrix, positive definite symmetric; is the desired state; is the desired control input; Q and R are the weighted matrices of state and control input respectively, both are positive definite symmetric matrices; The feasible set in the state Inside, In the control input constraint set Inside; represents the stage cost; T is the symbol of matrix transpose; (2) In the process of solving the nonlinear model prediction optimization problem, the following constraints need to be met: In the above formulas, is the derivative of the nominal system state; f is the system dynamic function; Input constraint set for control; is the feasible set of states; Ω is the terminal area; Formula (a) indicates that the state variables in the model are initialized from the state variables of the actual system; Formula (b) indicates that the constraints on the model input variables depend on the constraints of the actual system; Formula (c) indicates that the obtained predicted control variable is the last constant solution in the control time domain; Formula (d) represents the terminal inequality constraint, which is used to ensure that the system state returns to the terminal region Ω at the end of the finite time domain.

5. The method according to claim 1, wherein In the tubular model predictive control, if there exists a matrix E>0 and M>0, the following linear matrix inequality is satisfied: Then F=EM -1 , making the system stable; By solving the following optimization problem, we obtain the robust positive invariant set S: The state constraints and input constraints of the nominal system need to satisfy: In the above formulas, E is the symmetric positive definite matrix in LMI, which is used to construct the feedback gain matrix; M is the auxiliary variable in LMI, which must satisfy M>0; LMI refers to linear matrix inequality; A is the state transition matrix of the discrete-time system; B is the input matrix of the discrete-time system; T is the symbol of the matrix transpose; F is the feedback gain matrix; S is the robust positive invariant set; s is a vector, including s1 to s6, which is the decision variable of the optimization problem; s i is the polyhedron boundary parameter that defines S and the optimization variable; D is the perturbation set, which represents the external perturbations that the system may be subjected to; is the feasible set of states of the nominal system; X is the feasible set of states of the real system; is the control input constraint set of the nominal system; U is the input constraint set of the real system.

6. The method according to claim 1, characterized in that Before using the proximal policy optimization algorithm, train it as follows: (1) Collect interaction data of the underwater robot during normal operation and under different fault delay conditions, including information on status, action, and reward; Perform data preprocessing, normalize and combine into multiple sets of matrix data, use them as training data and establish training set and validation set; (2) Using the training set to train the policy network based on proximal policy optimization to ensure that the policy can effectively adapt to the control requirements under different states, and calculate the generalized advantage estimate to optimize the policy; using the clipping policy loss function during the training process to limit the policy update range and improve the stability of the training; After the training is completed, the policy parameters of the optimal model are retained and applied to the construction of the auxiliary feedback control law.

7. The method according to claim 1, characterized in that The implementation process of the proximal strategy optimization algorithm includes: (1) Data preprocessing Collect a set of interaction data by executing the current policy in the environment, including the state s t 、Action a t , reward r t and the next state s t+1 ; Importance sampling method is used to calculate probability ratios to improve data utilization: Where r t (θ) is the importance sampling ratio, which is used to correct the difference in action selection probability under the new and old strategies; θ is the parameter of the current strategy; t is the current time step; a t The action taken for the current time step; s t is the current state; θ old is the parameter of the old strategy; π θ For the current strategy; For the old strategy; Normalize the state data to have a mean of zero and a variance of 1, and perform a discounted accumulation calculation on the reward signal: Among them, R t is the cumulative discounted reward starting from time step t; γ is the discount factor, γ∈(0,1); k is the offset time step, indexing the future reward; t is the current time step; (2) Calculate the advantage estimate The generalized odds estimate is defined as follows: A t =d t +(gl)d t+1 +…(cl) T-t+1 d T-1 δ t =r t +γV(s t+1 )-V(s t ) Among them, A t is the advantage function, which is used to measure the quality of the current action relative to the average strategy; γ is the discount factor; δ t is the time difference error; T is the trajectory end time or truncation time; t is the current time step; λ is the weight parameter for balancing deviation and variance; V(s t ) is the state value function; The advantage function is calculated based on the generalized advantage estimation method to measure the performance of the current action relative to the average level and to optimize the subsequent strategy. (3) Optimize the objective function The optimization objective function of proximal policy optimization is defined as: Among them, L(θ) is the optimization objective function, which represents the loss function under the parameter θ; is the expected value, which represents the average of time step t; r t (θ) is the return or reward estimate at time step t under the parameters θ; A t is the advantage function; ∈ is the clipping range; clip(·) means clipping the probability ratio; The objective function includes both the value function loss term and the entropy regularization term. The objective function is optimized using a pruning strategy to prevent the strategy from updating too much. (4) Update strategy Update the policy parameters using the gradient ascent method, and then repeat the above steps with the new policy parameters until the policy performance no longer improves or the set number of iterations has been reached.

8. The method according to claim 1, characterized in that The Markov decision process is used in the implementation of the proximal policy optimization algorithm, which specifically includes: (1) Confirming the observation space based on the information of the underwater robot: In the formula, o i is the observation space; x is the state of the real system; u is the control input of the real system; is the state of the nominal system; is the control input of the nominal system; (2) Confirm the motion space at each time point by calculating the thrust correction value of the thruster: Where u c is the thrust correction value of the propeller; F is the auxiliary feedback control gain; x is the state of the real system; is the state of the nominal system; In order to improve the training speed and ensure the control stability, the auxiliary feedback control gain F at each time point is obtained through training, and then the thrust correction value u is calculated. c ; (3) Design a reward function based on the constraints and goals of the underwater robot control problem to ensure that rewards are immediately obtained once a new action is taken during training; Where r i is the reward function; x is the state of the real system; is the state of the nominal system; The instantaneous reward obtained during training depends on the error between the true system and the nominal system. The smaller the error, the higher the reward.

9. An AUV fault-tolerant control system based on proximal strategy optimization and tubular model prediction, characterized in that: Including hardware system modules and software function modules; The hardware system module includes sensors, signal processing and transmission devices, memory and processors: Several sensors are installed on the underwater robot to collect status data generated during operation; Signal processing and transmission device, used to collect status data collected by sensors in real time, and transmit it to the processor through the communication interface after preprocessing; Memory, used to store historical status data of the underwater robot and deploy software function modules; A processor configured to execute instructions, call a software function module from the memory, and implement the method described in any one of claims 1 to 8 using real-time status data and historical status data, thereby ultimately obtaining a control signal when a fault occurs during operation of the underwater robot; The software function modules include a data processing module, a model building module, a proximal strategy optimization module and a controller module; wherein, The data processing module is used to receive the signals obtained by each sensor, and convert and process them into applicable data forms; The model building module is used to build a mathematical model of the underwater robot including a kinematic model and a dynamic model, and to discretize the nominal system and the real system; The proximal policy optimization module includes a tubular model prediction network and an actor-critic network, using a Markov decision process corresponding to underwater robot control; The controller module is used to add the nominal control law predicted by the nonlinear model for the nominal system to the policy adaptive update auxiliary feedback control law obtained based on proximal policy optimization training to obtain the fault-tolerant control law for the AUV thruster and realize online fine-tuning of the fault-tolerant control.

10. The system according to claim 1, wherein: The sensors are attitude sensors and position sensors installed on the underwater robot; the status data collected by the sensors include at least: the underwater robot's position / angle, speed / angular velocity, acceleration / angular acceleration and thruster control signal; the processor is a single-chip microcomputer, Raspberry Pi or FPGA.

Images