AUV (Autonomous Underwater Vehicle) optimal heading control method under digital-analog cooperative driving in three-dimensional unknown environment

Abstract

The invention discloses an optimal heading control method for a digital-analog cooperative driven AUV (Autonomous Underwater Vehicle) in a three-dimensional unknown environment, which comprises the following steps of: firstly, establishing a three-dimensional six-degree-of-freedom kinematics model and a dynamics model of an under-actuated AUV in an unknown time-varying ocean current environment, and designing a sliding mode controller; secondly, designing a compensation scheme of a PPO reinforcement learning algorithm according to interference force suffered by the sliding mode controller in a three-dimensional unknown ocean current environment, constructing a state space, an action space, a reward function, an actor network and an evaluator network, and training the AUV; and after training is completed, reinforcement learning outputs disturbance force compensation in sliding mode control in real time according to the position and speed of the under-actuated AUV under the influence of the current three-dimensional unknown time-varying ocean current, and a sliding mode controller outputs a final control quantity. According to the method, the precision and robustness of sliding mode control in a complex ocean current environment are improved, and environment optimal heading positioning control of the autonomous underwater vehicle in a three-dimensional space is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater autonomous vehicle motion control, and specifically refers to an optimal heading control method for an AUV with digital-analog collaborative drive in a three-dimensional unknown environment. Background Art

[0002] The motion control of autonomous underwater vehicles (AUVs) can be used to perform different tasks and perform well with minimal human intervention, so it plays an important role in ocean exploration and scientific missions. Some of these tasks, such as marine environmental monitoring and weather forecasting, require AUVs to remain in a certain area for a long time, that is, area keeping capability. At present, ships with fixed-point positioning or area keeping capabilities are basically fully maneuverable dynamic positioning (DP) ships or large marine operating platforms. It can achieve fixed-point positioning or area keeping control through anchorage positioning or DP systems. The fluid dynamics coefficients of AUVs vary according to time-varying environmental disturbances and the motion information of AUVs. These fluid dynamics coefficients are also affected by the motion of other degrees of freedom (DOF). In addition, most AUVs are designed as underactuated configurations, which require more DOFs to be controlled than the number of independent control inputs to maintain actuator efficiency at high speeds. Therefore, AUVs are usually underactuated, highly nonlinear, and highly coupled. At the same time, AUVs are inevitably affected by unknown time-varying disturbances, complex fluid mechanics, and uncertainties. Due to the characteristics of AUVs, the area keeping control methods of fully maneuverable powered vessels, such as anchor positioning and DP, cannot be directly used to achieve area keeping control of AUVs. Currently, there are few studies on area keeping control of AUVs. Therefore, effective and reliable area keeping control for AUVs is still a very challenging task.

[0003] The main structure of underwater autonomous vehicles is very complex, with highly coupled nonlinear characteristics and structured and unstructured uncertainties. Due to changes in the hydrodynamic environment, its motion characteristics will also change, which makes it difficult to obtain an accurate mathematical model, thus bringing great challenges to the design of the control system. In addition, in actual operation, underwater autonomous vehicles are easily disturbed by uncertain factors such as wind, waves, and currents in the marine environment, which will significantly affect their control performance. Traditional control methods such as PID control, sliding mode control, and fuzzy control are highly dependent on prior knowledge and are difficult to cope with changes in complex environments. In contrast, reinforcement learning can comprehensively consider multiple objectives and constraints and learn nonlinear control strategies that adapt to the dynamic and disturbance characteristics of complex systems. Through continuous interaction with the environment, reinforcement learning can continuously update strategies during operation, so that the controller can respond to changes and disturbances in the system in a timely manner, achieve better control performance, and have strong real-time learning and adaptation capabilities. However, the current research on combining reinforcement learning with sliding mode control for hovering positioning of underwater autonomous vehicles is more used to adjust fixed parameters that require manual debugging in sliding mode control, and the robustness and adaptability of the system need to be further improved. Summary of the invention

[0004] Aiming at the shortcomings of traditional model-based control methods in the three-dimensional spatial positioning of underwater autonomous vehicles, the present invention proposes an optimal heading position control method for AUVs with digital-analog collaborative drive in a three-dimensional unknown environment.

[0005] In order to solve the above technical problems, the present invention proposes an optimal heading control method for an AUV with digital-analog collaborative driving in a three-dimensional unknown environment, comprising:

[0006] Step 1: Establish a three-dimensional six-degree-of-freedom kinematic model and dynamic model of the underactuated AUV in an unknown time-varying ocean current environment;

[0007] Step 2: Based on the 3D kinematic model and dynamic model of the underactuated AUV, a sliding mode controller based on position error observation information is designed to achieve 3D optimal heading control.

[0008] Step 3: Design a compensation scheme based on the data-driven proximal policy optimization (PPO) reinforcement learning algorithm to address the unobservable disturbance force on the sliding mode controller in the three-dimensional unknown ocean current environment, and construct the state space, action space and reward function of PPO.

[0009] Step 4: Based on the established state space, action space and reward function, the actor network and evaluator network in the PPO algorithm are designed, and a simulation scenario considering three-dimensional unknown time-varying ocean currents is built to train the AUV, forming a complete digital-analog collaborative drive control method;

[0010] Step 5: After the training is completed, reinforcement learning outputs the corresponding disturbance force compensation in the sliding mode control in real time according to the position and speed of the under-actuated AUV under the influence of the current three-dimensional unknown time-varying ocean current, and the sliding mode controller outputs the final control amount to achieve the desired control effect.

[0011] Step 1 is as follows:

[0012] The kinematic model of a three-dimensional six-degree-of-freedom AUV in an unknown time-varying ocean current environment is as follows:

[0013]

[0014] The kinetic model is as follows:

[0015]

[0016] in is the transformation matrix from the self-motion coordinate system to the fixed coordinate system, η=η=[ξ,η,ζ,φ,θ,ψ] T is a vector consisting of the AUV's current position (ξ,η,ζ) and the roll angle φ, pitch angle θ, and heading angle ψ in a fixed coordinate system. is the speed of the AUV in its own motion coordinate system, is the velocity of the ocean current in the fixed coordinate system, M is the rigid body mass and the inertia matrix of the added mass; is the rigid body weight and the additional mass Coriolis centripetal force data matrix; is the damping force matrix; τ is the thrust generated by the AUV propeller; τ c It is the interference force exerted by the environment where the AUV is located on the AUV.

[0017] Step 2 is as follows:

[0018] According to the three-dimensional six-degree-of-freedom kinematic model and dynamic model, the three-dimensional underactuated AUV sliding mode control is divided into a forward controller and an attitude controller. The forward controller defines u according to the optimal heading principle. e and p e is the error of the sliding surface of the forward controller, where u e is the error between the current speed of the AUV and the desired speed, p e is the error between the current position of the AUV and the desired position, and is calculated by controlling the longitudinal thrust along the sliding surface s1 = u e +λ1p e Control the position error and speed error of the AUV to be close to 0, so that the AUV can stay on the virtual ball for a long time under the interference of unknown ocean currents, and select Exponential reaching law, where ε1, λ1 and k1 are parameters to be designed that are greater than 0. Combining the sliding surface function and the dynamic model, the forward control law can be obtained.

[0019] The attitude controller is divided into a pitch controller and a heading controller. To ensure the optimal heading control, it is necessary to ensure that the underactuated AUV always points to the center of the virtual sphere. According to the optimal heading principle, q e and θ e is the error of the sliding surface of the pitch controller, q e and θ e They are the error between the current pitch angular velocity of the AUV and the desired pitch angular velocity and the error between the current pitch angle of the AUV and the desired pitch angle. By controlling the pitch moment along the sliding surface s2 = q e +λ2θ e Control the pitch velocity and pitch angle error of the AUV to be close to 0, and select Exponential reaching law, where ε2, λ2 and k2 are parameters to be designed that are greater than 0. The pitch control law can be obtained by combining the sliding surface function and the dynamic model.

[0020] According to the optimal heading control principle, in order to ensure that the underactuated AUV always points to the center of the virtual sphere, it is necessary to ensure that the current pitch angle reaches the desired pitch angle and the current heading angle also reaches the desired heading angle. Define r e and ψ e is the error definition of the sliding surface of the heading controller, r e and ψ e are the error between the current heading angular velocity of the AUV and the desired heading angular velocity and the error between the current heading angle of the AUV and the desired heading angle. The heading controller moves along the sliding surface s3 = r e +λ3ψ e Control the errors of the heading angular velocity and heading angle of the AUV to be close to 0, and select Exponential reaching law, where ε3, λ3 and k3 are parameters to be designed that are greater than 0. The heading control law can be obtained by combining the sliding surface function and the dynamic model.

[0021] Step three is as follows:

[0022] Since the control law contains difficult-to-observe interference forces, the present invention designs a reinforcement learning network to compensate for the unknown interference forces. The designed reinforcement learning state space includes: the deviation e1 between the current AUV posture and the current expected posture, the deviation e2 between the current AUV position and the current expected position, and the speed and posture of the current AUV; the action space includes: three unknown interference forces f u 、f q and f r ; The reward function includes: the reward and penalty items related to the deviation e1 between the current AUV position and the expected position, and the reward and penalty items related to the deviation e2 between the current AUV attitude and the expected attitude.

[0023] Step 4 is as follows:

[0024] Based on the state space, action space and reward function of the reinforcement learning designed in step three, a suitable actor network and evaluator network are constructed. The model required for the PPO algorithm includes four networks, namely the actor network and the critic network, as well as the target actor network and the target critic network. The input layer of the actor network is the variable in the state space; the output layer outputs the interference force of the position in each sliding mode controller to compensate for the disturbance caused by the unknown environment to the AUV, ensuring that the AUV can work in the desired position and posture; the input layer of the critic network is the action and state of reinforcement learning, and outputs the evaluation Q value of the state-action; the structure of the actor network and the critic network is consistent with the corresponding target network; and the ocean current data in the unknown ocean current environment is used to train the AUV in the constructed reinforcement learning network.

[0025] Step 5 is as follows:

[0026] The disturbance force output by training is provided to the sliding mode controller as compensation to achieve the three-dimensional optimal heading control of the under-actuated AUV in a three-dimensional unknown ocean current environment.

[0027] Beneficial effects of the present invention:

[0028] The present invention designs a neural network through reinforcement learning, so that the underwater autonomous vehicle can continuously learn from the environment in a three-dimensional space with unknown ocean currents, without relying on prior knowledge or precise mathematical models. Reinforcement learning is used to compensate for the unknown part in the sliding mode control, which significantly improves the accuracy and robustness of traditional sliding mode control in complex ocean current environments, and realizes the environmental optimal heading positioning control of the underwater autonomous vehicle in three-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A control system framework diagram of the underwater autonomous vehicle of the method of the present invention;

[0030] Figure 2 A schematic diagram of the ocean current velocity in the simulated environment in the method of the present invention;

[0031] Figure 3 A schematic diagram of a three-dimensional coordinate system of an underwater autonomous vehicle in the method of the present invention;

[0032] Figure 4 A schematic diagram of the network structure of the reinforcement learning part in the method of the present invention;

[0033] Figure 5 The underwater autonomous vehicle operation trajectory diagram of the method of the present invention;

[0034] Figure 6 A schematic diagram of position error during the operation of the underwater autonomous vehicle according to the method of the present invention;

[0035] Figure 7 A schematic diagram of the pitch angle error during the operation of the underwater autonomous vehicle according to the method of the present invention;

[0036] Figure 8 This is a schematic diagram of the heading angle error during the operation of the underwater autonomous vehicle according to the method of the present invention. DETAILED DESCRIPTION

[0037] The implementation steps of the present invention will be further described below with reference to the accompanying drawings and specific examples:

[0038] The present invention proposes an optimal heading control method for AUV with digital-analog collaborative driving in a three-dimensional unknown environment. The flowchart is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0039] Step 1: According to the movement of the underwater autonomous vehicle in the unknown time-varying ocean current environment, such as Figure 3 The kinematic and dynamic models of a three-dimensional six-degree-of-freedom underwater autonomous vehicle are established as shown. Since the roll is balanced by the restoring torque of the AUV itself, the simplified model is as follows:

[0040]

[0041] in is the transformation matrix from the self-motion coordinate system to the fixed coordinate system, η=[ξ,η,ζ,θ,ψ] T is a vector consisting of the AUV’s current position (ξ,η,ζ) and pitch angle θ and heading angle ψ in a fixed coordinate system. is the speed of the AUV in its own motion coordinate system, such as Figure 2 Shown is the velocity of the ocean current in the fixed coordinate system, M is the rigid body mass and the inertia matrix of the added mass; is the rigid body weight and the additional mass Coriolis centripetal force data matrix; is the damping force matrix; τ is the thrust generated by the AUV propeller; τ c It is the interference force generated by the environment where the AUV is located on the AUV; the specific matrices are as follows:

[0042]

[0043] Where m x 、c xx d x All are scalars.

[0044] Step 2: Based on the established model, a three-dimensional underactuated AUV sliding mode controller is designed, which is divided into a forward controller and an attitude controller. The forward controller defines u according to the optimal heading principle. e and p eis the error definition of the sliding surface of the forward controller, where u e is the error between the current speed of the AUV and the desired speed, p e is the error between the current position of the AUV and the desired position, and is calculated by controlling the longitudinal thrust along the sliding surface s1 = u e +λ1p e Control the position error and speed error of the AUV to be close to 0, so that the AUV can stay on the virtual ball for a long time under the interference of unknown ocean currents, and select The exponential reaching law, combined with the sliding surface function and the dynamic model, can obtain the forward control law:

[0045]

[0046] where f u is the unknown forward disturbance force, λ1, k1, ε1 are the parameters to be designed.

[0047] The attitude controller is divided into a pitch controller and a heading controller. To ensure the optimal heading control, it is necessary to ensure that the underactuated AUV always points to the center of the virtual sphere. According to the optimal heading principle, q e and θ e is the error definition of the sliding surface of the pitch controller, q e and θ e They are the error between the current pitch angular velocity of the AUV and the desired pitch angular velocity and the error between the current pitch angle of the AUV and the desired pitch angle. By controlling the pitch moment along the sliding surface s2 = q e +λ2θ e Control the pitch velocity and pitch angle error of the AUV to be close to 0, and select The exponential reaching law, combined with the sliding surface function and the dynamic model, can obtain the pitch control law:

[0048]

[0049] where f q is the unknown pitch disturbance force, λ2, k2, ε2 are the parameters to be designed.

[0050] According to the optimal heading control principle, in order to ensure that the underactuated AUV always points to the center of the virtual sphere, it is necessary to ensure that the current pitch angle reaches the desired pitch angle and the current heading angle also reaches the desired heading angle. Define r e and ψ e is the error definition of the sliding surface of the heading controller, r e and ψ e are the error between the current heading angular velocity of the AUV and the desired heading angular velocity and the error between the current heading angle of the AUV and the desired heading angle. The heading controller moves along the sliding surface s3 = r e +λ3ψe Control the errors of the heading angular velocity and heading angle of the AUV to be close to 0, and select The exponential approach law, combined with the sliding surface function and the dynamic model, can obtain the heading control law:

[0051]

[0052] where f r is the unknown disturbance force at the bow, and λ3, k3, and ε3 are the parameters to be designed.

[0053] Step 3: Since the control law contains difficult-to-observe disturbance forces, the present invention designs a reinforcement learning compensation method based on the data-driven proximal policy priority algorithm PPO to compensate for unknown disturbances. The constructed reinforcement learning state space is defined as S:[p,u t ,v t ,w t ,q t ,r t ,θ e ,ψ e ] where p is the current distance between the AUV and the desired virtual sphere center, v t =[u t ,v t ,w t ,q t ,r t ] is the measured delayed average velocity, θ e ,ψ e They are the error between the current pitch angle of the AUV and the current expected pitch angle, and the error between the current pitch angle and the current expected pitch angle; the action space is: a:[f u ,f q ,f r ] are three unknown disturbance forces; the reward function consists of three parts:

[0054]

[0055] r = r1 + r2 + r3;

[0056] where p e It is the difference between the current distance of the AUV to the center of the virtual sphere and the expected working distance.

[0057] Step 4: Based on the state space, action space and reward function established in step 3, design the actor network and evaluator network in the PPO algorithm. The model required for the PPO algorithm includes four networks such as Figure 4As shown in the figure, they are the Actor and Critic networks and the target Actor network and the target Critic network. The Actor network has an input layer, two hidden layers and an output layer, and the input layer is the state space; the output layer outputs three unknown interference forces in the sliding mode controller to compensate for the disturbance caused by the unknown environment to the AUV, ensuring that the AUV can work in the desired position and posture; the Critic network has an input layer, three hidden layers and an output layer, and the input layer is the action space and state space of reinforcement learning, and the output is the evaluation Q value of ; the structure of the Actor network and the Critic network is consistent with the corresponding target network; and a simulation scene considering three-dimensional unknown time-varying currents is built to train the AUV using the time-varying current data in the unknown environment in the designed PPO network.

[0058] The trained neural network model is used as a strategy in reinforcement learning to output estimates of action values ​​(i.e., disturbance forces), and these estimates are provided to the sliding mode controller. The sliding mode controller generates a control law based on this information and eventually applies it to the underwater autonomous vehicle, thereby achieving three-dimensional optimal heading positioning control of the underwater autonomous vehicle in an unknown time-varying current environment.

[0059] Step 5: The output disturbance force obtained through training is provided as compensation to the sliding mode controller to achieve the optimal heading control of the underactuated AUV in a three-dimensional environment. The running trajectory is as follows: Figure 5 The control effect is shown in Figure 6 , Figure 7 and Figure 8 As shown, the solid line is the error of the autonomous underwater autonomous vehicle of the present invention, and the dotted line is the traditional sliding mode control error diagram. It can be seen from the figure that the traditional sliding mode does not process the interference force, and the error changes significantly. The method combining reinforcement learning and sliding mode control can make the underwater autonomous vehicle basically maintain on the predetermined virtual sphere, achieving the expected control effect and accuracy.

[0060] The preferred embodiments of the present invention disclosed above are only used to illustrate the basic concepts of the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the implementation methods of the present invention. Obviously, many modifications and changes can be made according to the content of this specification. The embodiments selected and specifically described herein are intended to better explain the principles of the present invention and its practical application, so that those skilled in the relevant art can better understand and apply the present invention. The scope of protection of the present invention is limited to the claims and their full scope and their equivalents.

Claims

1. An optimal heading control method for AUV with digital-analog collaborative driving in a three-dimensional unknown environment, characterized in that: The following steps are involved: Step 1: Establish a three-dimensional six-degree-of-freedom kinematic model and dynamic model of the underactuated AUV in an unknown time-varying ocean current environment; Step 2: Based on the 3D kinematic model and dynamic model of the underactuated AUV, a sliding mode controller based on position error observation information is designed to achieve 3D optimal heading control. Step 3: Design a compensation scheme based on the data-driven proximal policy optimization (PPO) reinforcement learning algorithm for the disturbance force of the sliding mode controller in the three-dimensional unknown ocean current environment, and construct the state space, action space and reward function of PPO; Step 4: Based on the state space, action space and reward function, design the actor network and evaluator network in the PPO algorithm, and build a simulation scenario considering three-dimensional unknown time-varying ocean currents to train the AUV; Step 5: After the training is completed, reinforcement learning outputs the corresponding disturbance force compensation in the sliding mode control in real time according to the position and speed of the under-actuated AUV under the influence of the current three-dimensional unknown time-varying ocean current. The sliding mode controller outputs the final control amount to achieve the optimal heading control of the under-actuated AUV in the three-dimensional environment.

2. The optimal heading control method of AUV with digital-analog collaborative driving in a three-dimensional unknown environment according to claim 1 is characterized in that: The three-dimensional six-degree-of-freedom kinematic model is as follows: The kinetic model is as follows: in is the transformation matrix from the self-motion coordinate system to the fixed coordinate system, η=[ξ,η,ζ,φ,θ,ψ] T is a vector consisting of the current position of the AUV (ξ,η,ζ) and the roll angle φ, pitch angle θ, and heading angle v in a fixed coordinate system. is the speed of the AUV in its own motion coordinate system, is the velocity of the ocean current in the fixed coordinate system, M is the rigid body mass and the inertia matrix of the added mass; is the rigid body weight and the additional mass Coriolis centripetal force data matrix; is the damping force matrix; τ is the thrust generated by the AUV propeller; τ c It is the interference force exerted by the environment where the AUV is located on the AUV.

3. The optimal heading control method of AUV with digital-analog collaborative driving in a three-dimensional unknown environment according to claim 2 is characterized in that: The specific implementation process of step 2 is as follows: According to the three-dimensional six-degree-of-freedom kinematic model and dynamic model, the three-dimensional underactuated AUV sliding mode control is divided into a forward controller and an attitude controller. The forward controller defines u according to the optimal heading principle. e and p e is the error of the sliding surface of the forward controller, where u e is the error between the current speed of the AUV and the desired speed, p e is the error between the current position of the AUV and the desired position, and is calculated by controlling the longitudinal thrust along the sliding surface s1 = u e +λ1p e Control the position error and speed error of the AUV to be close to 0, so that the AUV can stay on the virtual ball for a long time under the interference of unknown ocean currents, and select Exponential reaching law, where ε1, λ1 and k1 are parameters greater than 0, and the forward control law is obtained by combining the sliding surface function and the dynamic model; The attitude controller is divided into a pitch controller and a heading controller. According to the optimal heading principle, q e and θ e is the error of the sliding surface of the pitch controller, q e and θ e They are the error between the current pitch angular velocity of the AUV and the desired pitch angular velocity and the error between the current pitch angle of the AUV and the desired pitch angle. By controlling the pitch moment along the sliding surface s2 = q e +λ2θ e Control the pitch velocity and pitch angle error of the AUV to be close to 0, and select Exponential reaching law, where ε2, λ2 and k2 are parameters greater than 0, and the pitch control law is obtained by combining the sliding surface function and the dynamic model; According to the optimal heading control principle, r is defined as e and ψ e is the error definition of the sliding surface of the heading controller, r e and ψ e are the error between the current heading angular velocity of the AUV and the desired heading angular velocity and the error between the current heading angle of the AUV and the desired heading angle. The heading controller moves along the sliding surface s3 = r e +λ3ψ e Control the errors of the heading angular velocity and heading angle of the AUV to be close to 0, and select The exponential reaching law, where ε3, λ3 and k3 are parameters to be designed that are greater than 0, is used to obtain the heading control law by combining the sliding surface function and the dynamic model.

4. The optimal heading control method of AUV with digital-analog collaborative driving in a three-dimensional unknown environment according to claim 3 is characterized in that: The specific implementation process of step three is as follows: A reinforcement learning network is designed to compensate for unknown disturbances in the control law. The state space of the reinforcement learning includes: the deviation e1 between the current AUV posture and the current desired posture, the deviation e2 between the current AUV position and the current desired position, and the speed and posture of the current AUV; the action space includes: three unknown disturbance forces f u 、f q and f r ; The reward function includes: the reward and penalty items related to the deviation e1 between the current AUV position and the expected position, and the reward and penalty items related to the deviation e2 between the current AUV attitude and the expected attitude.

5. The optimal heading control method of AUV with digital-analog collaborative driving in a three-dimensional unknown environment according to claim 4 is characterized in that: The specific implementation process of step 4 is as follows: Based on the state space, action space and reward function established in step 3, the actor network and the evaluator network are constructed. The model of the PPO algorithm includes four networks, namely the actor network, the critic network, the target network and the target critic network. The input layer of the actor network is the variable in the state space; the output layer outputs the interference force of the position in each sliding mode controller to compensate for the disturbance caused by the unknown environment to the AUV, ensuring that the AUV can work in the desired position and posture; the input layer of the critic network is the action and state of reinforcement learning, and outputs the evaluation Q value of the state-action; the structure of the actor network and the critic network is consistent with the corresponding target network; The AUV is trained using ocean current data in unknown ocean current environments in the reinforcement learning network.

Images