Underwater robot attitude control method and device based on PPO algorithm

Abstract

The embodiment of the invention provides an underwater robot attitude control method and device based on a PPO algorithm, and belongs to the technical field of computers. The method comprises the following steps: constructing a first dynamic model, and determining a first training sample according to the first dynamic model; determining the posture of the underwater robot as a preset posture when the shooting target is in the center of the shooting picture, and determining a second training sample according to the first training sample and the preset posture; constructing a deep neural network based on an Actor-Critic mechanism, and inputting the second training sample into the deep neural network for training to obtain a trained target deep neural network; updating the first dynamic model by using the deep neural network to obtain a second dynamic model; and performing attitude control on the underwater robot based on the second kinetic model and the first interference value. According to the method, the dynamic model can be optimized, and the shooting stability and the control precision of the underwater robot in a complex underwater environment are ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for controlling the posture of an underwater robot based on a PPO algorithm. Background Art

[0002] With the development of marine tourism, more and more people use underwater robots for tracking and shooting. During the shooting process, the underwater robot shoots the target by motion tracking. In the process of following the target, the underwater robot needs to adjust the movement posture to ensure the shooting effect. The current underwater robot posture control methods generally include the following three methods:

[0003] 1. By designing the switching function, the system state can keep sliding on the sliding surface to achieve the desired attitude control effect;

[0004] Second, by adjusting the three parameters of proportion, integration and differentiation, the PID controller is used to adjust the system attitude in linear systems and environments with small static errors;

[0005] 3. Adjust the control strategy in real time according to the changes in system parameters, and adaptively control the system attitude under different operating conditions and environmental changes.

[0006] However, due to the complex ocean current environment, which affects the shooting stability and tracking accuracy of underwater robots, as the demand for underwater shooting continues to expand and the requirements for shooting effects continue to increase, the current tracking shooting technology of underwater robots cannot meet individual needs. Summary of the invention

[0007] In order to solve the above technical problems, the purpose of the present invention is to provide an underwater robot posture control method and device based on the PPO algorithm, which can achieve accurate tracking when the underwater robot is shooting the target, improve the stability and accuracy of the underwater robot's posture control in the underwater environment, and help the underwater robot cope with complex underwater environments.

[0008] To achieve the above purpose, one aspect of an embodiment of the present application proposes a posture control method of an underwater robot based on a PPO algorithm, the method comprising:

[0009] Constructing a first dynamics model, and determining a first training sample according to the first dynamics model, the first training sample being posture data when the underwater robot interacts with the underwater environment;

[0010] Determine the posture of the underwater robot when the shooting target is in the center of the shooting picture as a preset posture, determine a second training sample according to the first training sample and the preset posture, and the shooting target is the target tracked and shot by the underwater robot;

[0011] Building a deep neural network based on the Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network;

[0012] Acquire a first interference value, where the first interference value is an interference value generated by the underwater environment on the underwater robot when the underwater robot interacts with the underwater environment;

[0013] Using the trained deep neural network to update the first kinetic model to obtain a second kinetic model;

[0014] The underwater robot is posture controlled based on the second dynamics model and the first interference value.

[0015] Optionally, the first kinetic model is constructed, and the expression of the first kinetic model is:

[0016]

[0017] Wherein, M represents the matrix composed of the system inertia matrix and the additional mass matrix, C(v) represents the Coriolis centripetal force and centripetal force matrix composed of the rigid body term and the additional mass matrix, D(v) represents the damping matrix caused by the resistance, g(η) represents the restoring force matrix caused by gravity and buoyancy, B represents the control matrix depending on the configuration of the underwater robot thruster, u represents the force and torque vector provided by the underwater robot thruster, ζ represents the force and torque vector generated by the underwater environment, represents the position and attitude vector of the underwater robot in the earth coordinate system, represents the position and the derivative of the attitude vector of the underwater robot in the earth coordinate system, (x, y, z) represents the coordinates of the underwater robot in the earth coordinate system, represents the posture vector of the underwater robot in the earth coordinate system, ε=(u,v,w,p,q,r) T represents the linear velocity and angular velocity vector of the underwater robot in the moving coordinate system, represents the derivatives of the linear velocity and angular velocity vector of the underwater robot in the moving coordinate system, wherein (u, v, w) are the tangential component, radial component and normal component of the linear velocity of the underwater robot in the moving coordinate system, respectively.

[0018] Optionally, determining the posture of the underwater robot when the shooting target is in the center of the shooting screen as the preset posture, and determining the second training sample according to the first training sample and the preset posture specifically includes the following steps:

[0019] The posture of the underwater robot when the shooting target is in the center of the shooting screen is determined as the preset posture. The expression of the preset posture is (φ d ,θ d ,ψ d ) and d, where (φ d ,θ d ,ψ d ) is a preset coordinate, d is a preset distance between the underwater robot and the shooting target, and the shooting target is the target tracked and photographed by the underwater robot;

[0020] The expression of the first sample is (φ c ,θ c ,ψ c ) and d c , where (φ c ,θ c ,ψ c ) is the actual coordinate between the underwater robot and the shooting target, d c is the actual distance between the underwater robot and the shooting target;

[0021] The expression of the second training sample is:

[0022]

[0023] in, is the coordinate error, is the distance error between the underwater robot and the shooting target.

[0024] Optionally, constructing a deep neural network based on the Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network specifically comprises the following steps:

[0025] Configure the value network and policy network based on the Actor-Critic mechanism;

[0026] Inputting the second training sample into the value network and the policy network for training, and outputting a first training result;

[0027] The parameters of the value network and the strategy network are updated according to the first training result and the descent gradient parameters of the deep neural network to obtain a trained target deep neural network.

[0028] Optionally, the first interference value is obtained, and the expression of the first interference value is:

[0029]

[0030] in, is the first disturbance value, y is the output of the attitude control system of the underwater robot, G(s) is the transfer function of the observer, M represents the matrix composed of the system inertia matrix and the additional mass matrix, ε represents the linear velocity and angular velocity vector of the underwater robot in the moving system coordinate system, represents the derivatives of the linear velocity and angular velocity vectors of the underwater robot in the moving coordinate system, C(ε) represents the Coriolis centripetal force and centripetal force matrix composed of the rigid body term and the additional mass matrix, D(ε) represents the damping matrix caused by the resistance, g(η) represents the restoring force matrix caused by gravity and buoyancy, B represents the control matrix depending on the configuration of the underwater robot thruster, u represents the force and torque vector provided by the underwater robot thruster, and the observer is used to measure the interference caused by the underwater environment on the posture control of the underwater robot.

[0031] Optionally, the performing posture control on the underwater robot based on the second dynamic model and the first interference value specifically comprises the following steps:

[0032] Obtaining a first control value output by the second dynamic model;

[0033] The first interference value is compensated to the first control value to obtain a second control value. The calculation formula of the second control value is:

[0034]

[0035] Among them, a comp is the second control value, a is the first control value, is the first interference value.

[0036] Another part of the embodiment of the present invention further provides an underwater robot posture control device based on the PPO algorithm, including:

[0037] A first module is used to construct a first dynamics model and determine a first training sample according to the first dynamics model, where the first training sample is posture data when the underwater robot interacts with the underwater environment;

[0038] A second module is used to determine the posture of the underwater robot when the shooting target is in the center of the shooting picture as the preset posture, and determine a second training sample according to the first training sample and the preset posture, wherein the shooting target is the target tracked and shot by the underwater robot;

[0039] The third module is used to construct a deep neural network based on the Actor-Critic mechanism, input the second training sample into the deep neural network for training, and obtain a trained target deep neural network;

[0040] A fourth module is used to obtain a first interference value, where the first interference value is an interference value generated by the underwater environment to the underwater robot when the underwater robot interacts with the underwater environment;

[0041] A fifth module is used to update the first dynamics model using the trained deep neural network to obtain a second dynamics model;

[0042] The sixth module is used to perform posture control on the underwater robot based on the second dynamic model and the first interference value.

[0043] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0044] The memory is used to store programs;

[0045] The processor executes the program to implement the method described above.

[0046] The embodiment of the present invention further discloses a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0047] The embodiment of the present invention further discloses a computer program product, including a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0048] The embodiment of the present invention constructs a first dynamic model, and determines a first training sample according to the first dynamic model, the first training sample is the posture data when the underwater robot interacts with the underwater environment; the posture of the underwater robot when the shooting target is in the center of the shooting screen is determined as the preset posture, and the second training sample is determined according to the first training sample and the preset posture, the shooting target is the target tracked and photographed by the underwater robot; a deep neural network is constructed based on the Actor-Critic mechanism, and the second training sample is input into the deep neural network for training to obtain a trained target deep neural network; a first interference value is obtained, and the first interference value is the interference value generated by the underwater environment to the underwater robot when the underwater robot interacts with the underwater environment; the first dynamic model is updated using the trained deep neural network to obtain a second dynamic model; and the underwater robot is posture controlled based on the second dynamic model and the first interference value. The embodiment of the present invention can optimize the dynamic model, improve the intelligence of the dynamic model, and ensure the shooting stability and control accuracy of the underwater robot in a complex underwater environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0050] Figure 2 is a flow chart of an underwater robot posture control method based on a PPO algorithm provided by an embodiment of the present invention;

[0051] Figure 3 is a relative coordinate diagram between the underwater robot and the shooting target provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of an expansion process of constructing a deep neural network based on an Actor-Critic mechanism provided by an embodiment of the present invention, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network;

[0053] Figure 5 is a schematic diagram of a process of performing posture control on the underwater robot based on the second dynamic model and the first interference value provided by an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram of the algorithm core of a reinforcement learning PPO algorithm combined with interference compensation provided by the present invention;

[0055] Figure 7 It is another flow chart of underwater robot posture control based on PPO algorithm provided by the present invention;

[0056] Figure 8 It is a structural schematic diagram of an underwater robot posture control device based on a PPO algorithm provided in an embodiment of the present application;

[0057] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0059] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0060] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] With the booming development of the marine tourism industry, the number of diving enthusiasts has increased significantly, accompanied by a rapid increase in the demand for underwater photography. However, when divers currently take photos underwater, they usually can only rely on the help of their companions or use underwater robots to complete the shooting. Due to the good endurance and portability of underwater robots powered by cables, they are deeply favored by diving enthusiasts. However, when underwater robots track divers in complex ocean current environments, they are often disturbed by ocean currents, which often leads to unstable fixed-distance shooting or inability to accurately capture the target, which puts higher requirements on the underwater robot control system.

[0063] Therefore, in an uncertain ocean current environment, there is an urgent need for an underwater robot attitude control method that can timely perceive environmental changes and has strong adaptive adjustment to ensure the stability of the underwater robot's hovering shooting.

[0064] In the prior art, the posture control methods of underwater robots mainly include sliding mode control, PID control, adaptive control, etc. Sliding mode control has strong robustness. By designing the switching function, the system state can keep sliding on the sliding surface to achieve the desired posture control effect; PID control is widely used because it is simple and easy to implement. By adjusting the three parameters of proportion, integration and differentiation, the PID controller can accurately adjust the system posture in linear systems and environments with small static errors; adaptive control can adjust the control strategy in real time according to the changes in system parameters, and adaptively control the system posture under different operating conditions and environmental changes.

[0065] Compared with the shortcomings of traditional PID control, sliding mode control and adaptive control methods, such as fixed parameters are difficult to adapt to complex environments, limited control accuracy, insufficient real-time adjustment capabilities, etc., the embodiment of the present application combines reinforcement learning PPO control with interference compensation. Through its adaptive learning and strategy optimization capabilities, it can dynamically respond to complex nonlinear and uncertain environments, achieve higher control accuracy and robustness, and is particularly suitable for posture control of underwater robots under complex water flow conditions.

[0066] The underwater robot posture control method based on the PPO algorithm provided in the embodiment of the present invention relates to the field of computer technology. The underwater robot posture control method based on the PPO algorithm provided in the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a vehicle-mounted terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a conversion method for a roadbed slope model, etc., but is not limited to the above forms.

[0067] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] To facilitate the understanding of the technical solution of the present invention, the technical feature terms that may appear in the embodiments of the present invention are first explained:

[0069] 1) PPO algorithm: PPO (Proximal Policy Optimization) algorithm is a policy gradient method used to solve the policy update problem in reinforcement learning. The PPO algorithm ensures the stability of learning by limiting the difference between the new and old policies, while improving data utilization efficiency. The PPO algorithm is an on-policy method, but in practical applications it is often converted to an off-policy method to improve efficiency.

[0070] 2) Actor-Critic mechanism: The core idea of ​​the Actor-Critic mechanism is to separate policy learning (Actor) and value function estimation (Critic) into two cooperating components. The policy network is responsible for generating the action selection strategy of the agent in each state, and the value network is responsible for evaluating the quality of the strategy selected by the Actor, that is, estimating the state value function or the action value function. A closed-loop feedback mechanism is formed between the Actor and the Critic. This collaborative working mode effectively combines the stability of the value-based reinforcement learning method and the flexibility of the policy-based reinforcement learning method.

[0071] 3) Dynamic model: The dynamic model is the core of underwater robot research. Its modeling methods include traditional analytical models and simulation-based computational models. During the model construction process, it is necessary to determine the robot's constraint equations, dynamic equations, and the laws of action of various external and internal forces.

[0072] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0073] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0074] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0075] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention.

[0076] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides an underwater robot posture control method based on the PPO algorithm. The following is explained using the underwater robot posture control method based on the PPO algorithm applied in the terminal 102 as an example. It can be understood that the underwater robot posture control method based on the PPO algorithm can also be applied to the server 101.

[0077] On the one hand, referring to Figure 2 , Figure 2 The flowchart of the underwater robot posture control method based on the PPO algorithm provided by the embodiment of the present invention, the execution subject of the underwater robot posture control method based on the PPO algorithm can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method comprises the following steps:

[0078] S100: construct a first dynamics model, and determine a first training sample according to the first dynamics model, where the first training sample is posture data when the underwater robot interacts with the underwater environment.

[0079] Optionally, the expression of the first kinetic model is:

[0080]

[0081] Wherein, M represents the matrix composed of the system inertia matrix and the additional mass matrix, C(v) represents the Coriolis centripetal force and centripetal force matrix composed of the rigid body term and the additional mass matrix, D(v) represents the damping matrix caused by the resistance, g(η) represents the restoring force matrix caused by gravity and buoyancy, B represents the control matrix depending on the configuration of the underwater robot thruster, u represents the force and torque vector provided by the underwater robot thruster, ζ represents the force and torque vector generated by the underwater environment, η=(x, y, z, φ, θ, ψ) T represents the position and attitude vector of the underwater robot in the earth coordinate system, represents the derivative of the position and attitude vector of the underwater robot in the earth coordinate system, (x, y, z) represents the coordinates of the underwater robot in the earth coordinate system, (θ, ψ) represents the attitude vector of the underwater robot in the earth coordinate system, ε=(u, v, w, p, q, r) T represents the linear velocity and angular velocity vector of the underwater robot in the moving coordinate system, represents the derivatives of the linear velocity and angular velocity vector of the underwater robot in the moving coordinate system, wherein (u, v, w) are the tangential component, radial component and normal component of the linear velocity of the underwater robot in the moving coordinate system, respectively.

[0082] It should be noted that the first dynamic model is a dynamic model of the underwater robot in matrix form obtained based on the relevant derivation content in the first chapter of "Marine Control System" written by Fossen. When the underwater robot tracks and shoots the target in a complex ocean current environment, it is often disturbed by the ocean current, resulting in unstable fixed-distance shooting or inability to accurately capture the target.

[0083] Therefore, based on the original derivation content, this embodiment takes the dynamic interference of ocean currents into consideration in the posture control of the underwater robot. Specifically, the data ζ representing the force generated by the underwater environment and its torque vector is compensated into the posture control of the underwater robot, thereby significantly improving the stability and control accuracy of the underwater robot in a complex ocean current environment.

[0084] S200, determining the posture of the underwater robot when the shooting target is in the center of the shooting screen as a preset posture, and determining a second training sample according to the first training sample and the preset posture, wherein the shooting target is a target tracked and shot by the underwater robot.

[0085] It should be noted that in some embodiments, Figure 3 As shown, it is a relative coordinate diagram between the underwater robot and the shooting target. Based on the position of the underwater robot, the XYZ coordinate axes are set, and the underwater robot tracks and shoots the diver.

[0086] For example, in some specific implementations, 8 thrusters are set in the underwater robot, and the four thrusters distributed at both ends of the underwater robot body are responsible for driving the fuselage plane XOY, and the four thrusters distributed inside the fuselage are longitudinal thrusters, which are responsible for driving the movement of the fuselage on the Z axis. In the process of adjusting the posture of the underwater robot, the four thrusters distributed on both sides of the fuselage can be adjusted according to the distance information obtained based on the PPO algorithm to adjust the posture of the underwater robot, so that the fuselage moves forward or backward to meet the requirements of the distance information; in addition, the 8 thrusters can be driven according to the angle information obtained based on the PPO algorithm to adjust the posture of the underwater robot to meet the requirements of the angle information.

[0087] Optionally, the posture of the underwater robot when the shooting target is in the center of the shooting screen is determined as the preset posture, and the second training sample is determined according to the first training sample and the preset posture, and the shooting target is a target tracked and shot by the underwater robot, specifically:

[0088] The posture of the underwater robot when the shooting target is in the center of the shooting screen is determined as the preset posture, and the expression of the preset posture is: and d, where is a preset coordinate, d is a preset distance between the underwater robot and the shooting target, and the shooting target is a target tracked and photographed by the underwater robot;

[0089] Exemplarily, in some specific implementations, the first training sample and the second training sample are used to train an intelligent agent in a neural network based on a PPO algorithm, and the intelligent agent is used to enable the underwater robot to adjust the shooting target to the center of the shooting screen at a certain distance from the shooting target.

[0090] Specifically, the expression of the first sample is: and d c ,in, is the actual coordinate of the underwater robot, d c is the actual distance between the underwater robot and the shooting target;

[0091] The expression of the second training sample is:

[0092]

[0093] in, is the coordinate error, is the distance error between the underwater robot and the shooting target.

[0094] For example, in some specific implementations, the process of adjusting the posture of the underwater robot using an agent in a neural network based on the PPO algorithm can be as follows:

[0095] 1. Project the target onto the XOY plane and use the fixed-range sonar to measure the actual distance d between the underwater robot and the target on the XOY plane. c and the error of the distance d set on the xoy plane and any posture To any posture The posture error is defined as the state space s as the input of the reinforcement learning state;

[0096] 2. Defined as the posture angle and distance adjustment as a continuous action space a, in some specific embodiments, it can be The range is set within [-5°,5°];

[0097] 3. After taking state s as the input of the reinforcement learning state, the agent outputs the posture adjustment information of the underwater robot. In some specific implementations, the posture adjustment information may include: angle information for adjusting the three Euler angles of the underwater robot and distance information for adjusting the position of the underwater robot.

[0098] S300: construct a deep neural network based on the Actor-Critic mechanism, input the second training sample into the deep neural network for training, and obtain a trained target deep neural network.

[0099] It should be noted that in some embodiments, Figure 4 As shown, constructing a deep neural network based on the Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network may include the following steps:

[0100] S401. Configure the value network and strategy network based on the Actor-Critic mechanism;

[0101] It should be noted that the Actor-Critic mechanism is a neural network mechanism in the PPO algorithm, wherein the Actor network is a policy network, the Critic network is a value network, the Actor network and the Critic network include an input layer, a hidden layer and an output layer, the activation function of the hidden layer can use the Relu function, and the output layer can use the tanh function.

[0102] Exemplarily, in some specific implementations, configuring the value network and the policy network based on the Actor-Critic mechanism may include initializing the parameters of the Actor network and the Critic network. The parameters used for initialization may include the learning rate of the two networks, the network update frequency, the experience pool size and other parameters.

[0103] It should be noted that the Actor network can be used to output posture adjustment information according to the actual posture and distance error of the input underwater robot. The posture adjustment information may include: angle information for adjusting the three Euler angles of the underwater robot and distance information for adjusting the position of the underwater robot.

[0104] It should be noted that the Critic network can be used to output the state value function V(s) of the underwater robot performing the action under the actual state according to the input state quantity s and action quantity a, and the advantage value at time t is obtained through V(s):

[0105]

[0106] Among them, λ is the tAn additional hyperparameter is introduced when performing exponential weighted average calculation, γ is the discount factor, V(s t+ι ) is the state value function at time t+ι, r t+ι is the reward function at time t+ι, and T is the maximum number of time steps.

[0107] For example, in some specific implementations, the reward function can be defined as C1, C2 are weight coefficients, r e (t), and X is defined as follows:

[0108]

[0109] where k i ,i=1,2.3,4,5,6 are the set reward values, μ1, μ2, μ3 are the tolerance errors.

[0110] S402, inputting the second training sample into the value network and the strategy network for training, and outputting a first training result;

[0111] For example, in some specific implementations, before training the value network and the strategy network, the number of training rounds is set to TN, the number of training cycles is set to T, and the position of the underwater robot is initialized to obtain the initial state vector s, and the initial time step number i=0; based on the current state s, the action a=π(s|θ) output by the Actor network is adopted, θ is a strategy network parameter, and represents the target strategy π θ parameterization.

[0112] S403: Update the parameters of the value network and the policy network according to the first training result and the descent gradient parameters of the deep neural network to obtain a trained target deep neural network.

[0113] Exemplarily, in some specific implementations, the underwater robot interacts with the environment based on the current posture to obtain environmental feedback.

[0114] The posture data and environmental information obtained by the sensor can calculate the current relative distance and posture error between the target position and the target position, generate the corresponding action instruction a through the reinforcement learning algorithm, and store the information obtained by the sensor in the experience pool.

[0115] At the same time, the compensation control observer is used to estimate the intensity and direction of ocean current disturbance in real time. The interference estimate Compensate to the action output by the deep neural network to obtain the actual control value Execute action a compGet the new state s′ and action a′, set s=s′, and then calculate the immediate reward r, the action probability strategy logarithm and the round end flag done, ending the training of the deep neural network for this round.

[0116] After completing a time step, the state, action, immediate reward, logarithmic value of the action probability strategy and the round end flag done obtained by the interaction in the current time step are recorded and stored in the experience pool in sequence to form a track. The data in the experience pool will be used for subsequent strategy updates.

[0117] Let i increment and determine whether the time step number i has reached the maximum time step T or has reached the end of the round.

[0118] If not, continue the above cycle until the maximum time step T is reached or the round end mark is reached, forming a complete trajectory. If yes, take out the complete trajectory currently trained in the experience pool, calculate the reward value r(t), advantage value A t and TD error γV(s t+1 )+r t+1 -V(s t ).

[0119] Using the mean square error MSE loss function Update the parameters of the value network Cirtic through the gradient descent direction propagation of the deep neural network.

[0120] Using the loss function L clip (θγ)=E t [min(r t (θ)A t ,clip(r t (θ),1-∈,1+ε)A t )], the parameters of the value network Critic are updated through the gradient descent direction propagation of the deep neural network, where r t (θ) is the action probability ratio , π old Representing the old policy ∈ is a hyperparameter in PPO, which is used to control the magnitude of the policy update.

[0121] S400, obtaining a first interference value, where the first interference value is an interference value generated by the underwater environment on the underwater robot when the underwater robot interacts with the underwater environment;

[0122] Since the influence of ocean currents on the movement of underwater robots in underwater environments cannot be ignored, illustratively, in some specific implementations, an observer that quantifies ocean current interference can be set to calculate the first interference value. In a specific implementation, the observer can be an ocean current interference observer.

[0123] Optionally, the expression of the first interference value is:

[0124]

[0125] in, is the first disturbance value, y is the output of the attitude control system of the underwater robot, G(s) is the transfer function of the observer, and the observer is used to measure the disturbance caused by the underwater environment to the attitude control of the underwater robot, is the rate of change of the three Euler angles in the first sample and the second training sample of the underwater robot, and M represents the matrix composed of the system inertia matrix and the additional mass matrix.

[0126] S500: Use the trained deep neural network to update the first dynamics model to obtain a second dynamics model.

[0127] When the number of training rounds of the deep neural network reaches a preset value, the trained deep neural network is saved, and when the posture of the underwater robot is controlled, the first dynamics model is updated to obtain a new second dynamics model, and the new second dynamics model is used to control the posture of the underwater robot.

[0128] S600: Performing posture control on the underwater robot based on the second dynamics model and the first interference value.

[0129] It should be noted that, in some embodiments, Figure 5 As shown, performing posture control on the underwater robot based on the second dynamic model and the first interference value may include the following steps:

[0130] S501, obtaining a first control value output by the second dynamic model;

[0131] S502: Compensate the first interference value to the first control value to obtain a second control value. The calculation formula of the second control value is:

[0132]

[0133] Among them, a comp is the second control value, a is the first control value, is the first interference value.

[0134] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0135] In the embodiment of the present application, a reinforcement learning PPO algorithm combined with interference compensation is provided, referring to Figure 6 , Figure 6 The core of the algorithm of a reinforcement learning PPO algorithm combined with interference compensation is shown:

[0136] Will represent the current posture and distance error of the underwater robot The loss function Actor loss is input to the Actor network, and the Actor network outputs the angle and thrust adjustment vector a in this state. t .

[0137] The output of the ocean current interference observer Supplement the output of the Actor network to obtain the actual control value of the underwater robot's posture control The actual control value is executed through the dynamic model of the underwater robot to obtain the new state S of the underwater robot. t ′ and action a′, let S t =S t ′, and then calculate the immediate reward r, the action probability strategy logarithm and the round end flag done.

[0138] Get the new state S of the underwater robot t ′, the error calculation is performed in combination with the preset posture set for the underwater robot. Specifically, the preset posture set for the underwater robot may include the preset coordinates (φ d ,θ d ,ψ d ) and the preset distance d between the underwater robot and the shooting target d .

[0139] After obtaining the result of the error calculation, the immediate reward r is calculated based on the result of the error calculation. Specifically, the result of the error calculation may include the posture error Error change rate of Euler angles The distance difference between the underwater robot in the current posture and the underwater robot in the preset posture

[0140] After obtaining the instant reward r and the state function V of the output of the Critic network, the advantage value A at time t is obtained through V t The loss function calculation model CLIP is used, based on the advantage value A t Calculate and update the Actor Loss parameter of the Actor network. Use the mean square error MSE loss function to update the Critic Loss parameter of the Critic network.

[0141] In some specific application scenarios, such as Figure 7 As shown, another underwater robot posture control method based on the PPO algorithm proposed in this application is specifically implemented as follows:

[0142] S701. Establish a dynamic model of the underwater robot.

[0143] S702, set the state space and action space of the underwater robot, establish the corresponding reward function and target weight, initialize the deep neural network Actor-Critic network, and set the number of training rounds and training cycle.

[0144] S703. Interact with the environment.

[0145] S704: Get the output action, the state at the next moment and the immediate reward.

[0146] S705: Save the status, action and immediate reward obtained from the interaction into the experience pool.

[0147] S706: Use the interference value output by the observer to compensate for the action output by the deep neural network, and use it as the actual action to adjust the current posture of the underwater robot.

[0148] It should be noted that, in some embodiments, when the trajectory is not ended, that is, when the maximum time step T is not reached or the end mark done is not reached, step S703 is performed; when the trajectory is ended, step S707 is performed.

[0149] S707. Use the data from the experience pool to update the value network and strategy network.

[0150] It should be noted that, in some embodiments, when the number of training rounds has not been reached, step S702 is performed; when the number of training rounds has been reached, step S708 is performed.

[0151] S708. Use the data from the experience pool to update the value network and strategy network.

[0152] In summary, the present invention proposes an underwater robot posture control method based on the PPO model-free learning algorithm, which takes the interference of ocean currents on the underwater robot into account in the posture control of the underwater robot, can observe and estimate the ocean current interference, and compensate it in the posture control of the underwater robot, thereby ensuring the shooting stability and control accuracy of the underwater robot in a complex underwater environment. Compared with the prior art, the present invention has at least the following beneficial effects:

[0153] 1. The target distance error was introduced, and a targeted reward function was designed. Through training, real-time iteration and adjustment of network parameters were achieved, so that the control strategy could be adaptively adjusted in the dynamically changing underwater ocean current environment, and accurate tracking of the shooting target was achieved, ensuring that the shooting target was always in the center of the shooting screen, and the underwater robot could be stabilized at a certain distance from the shooting target, effectively improving the posture control stability of the underwater robot in complex water flow environments.

[0154] 2. The dynamic disturbance of ocean currents is taken into account in the control adjustment, thereby significantly improving the stability and control accuracy of the underwater robot under complex water flow conditions.

[0155] 3. The modeling of multi-dimensional state and action space is added, and the robot posture error and target distance error are simultaneously incorporated into the state space. The action space is adaptively adjusted through a deep neural network. Specifically, the adaptive adjustment of the action space includes posture angle adjustment and horizontal thrust adjustment, thereby achieving precise control of the underwater robot's posture and stable hovering, making the control effect easier to observe and the control method more precise.

[0156] See also Figure 8 The embodiment of the present application further provides an underwater robot posture control device 800 based on the PPO algorithm, which can implement the above-mentioned underwater robot posture method based on the PPO algorithm, and the device includes:

[0157] The first module 801 is used to construct a first dynamics model and determine a first training sample according to the first dynamics model, where the first training sample is posture data when the underwater robot interacts with the underwater environment;

[0158] The second module 802 is used to determine the posture of the underwater robot when the shooting target is in the center of the shooting picture as the preset posture, and determine a second training sample according to the first training sample and the preset posture, and the shooting target is the target tracked and shot by the underwater robot;

[0159] The third module 803 is used to build a deep neural network based on the Actor-Critic mechanism, and input the second training sample into the deep neural network for training until the number of training rounds reaches a preset value;

[0160] The fourth module 804 is used to obtain a first interference value, where the first interference value is an interference value generated by the underwater environment to the underwater robot when the underwater robot interacts with the underwater environment;

[0161] The fifth module 805 is used to update the first dynamics model using the trained deep neural network to obtain a second dynamics model;

[0162] The sixth module 806 is used to perform posture control on the underwater robot based on the second dynamic model and the first interference value.

[0163] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned underwater robot posture method based on the PPO algorithm when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0165] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0166] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0167] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0168] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the underwater robot posture method based on the PPO algorithm in the embodiment of this application;

[0169] Input / output interface 903, used to implement information input and output;

[0170] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0171] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0172] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0173] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned underwater robot posture method based on the PPO algorithm is implemented.

[0174] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0175] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The underwater robot posture control method and device based on the PPO algorithm provided in the embodiment of the present application take the interference of ocean currents on the underwater robot into consideration in the posture control of the underwater robot, can observe and estimate the ocean current interference, and compensate it in the posture control of the underwater robot, thereby ensuring the shooting stability and control accuracy of the underwater robot in a complex underwater environment.

[0177] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0178] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0179] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0181] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0182] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0183] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0186] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0187] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for underwater robot posture control based on PPO algorithm, characterized in that: The method comprises: Constructing a first dynamics model, and determining a first training sample according to the first dynamics model, the first training sample being posture data when the underwater robot interacts with the underwater environment; Determine the posture of the underwater robot when the shooting target is in the center of the shooting picture as a preset posture, determine a second training sample according to the first training sample and the preset posture, and the shooting target is the target tracked and shot by the underwater robot; Building a deep neural network based on the Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network; Acquire a first interference value, where the first interference value is an interference value generated by the underwater environment on the underwater robot when the underwater robot interacts with the underwater environment; Using the trained deep neural network to update the first dynamics model to obtain a second dynamics model; The underwater robot is posture controlled based on the second dynamics model and the first interference value.

2. The method according to claim 1, characterized in that The first kinetic model is constructed, and the expression of the first kinetic model is: Wherein, M represents the matrix composed of the system inertia matrix and the additional mass matrix, C(ε) represents the Coriolis centripetal force and centripetal force matrix composed of the rigid body term and the additional mass matrix, D(ε) represents the damping matrix caused by the resistance, g(η) represents the restoring force matrix caused by gravity and buoyancy, B represents the control matrix depending on the configuration of the underwater robot thruster, u represents the force and torque vector provided by the underwater robot thruster, ζ represents the force and torque vector generated by the underwater environment, represents the position and attitude vector of the underwater robot in the earth coordinate system, represents the position and the derivative of the attitude vector of the underwater robot in the earth coordinate system, (x, y, z) represents the coordinates of the underwater robot in the earth coordinate system, represents the posture vector of the underwater robot in the earth coordinate system, ε=(u,v,w,p,q,r) T represents the linear velocity and angular velocity vector of the underwater robot in the moving coordinate system, represents the derivatives of the linear velocity and angular velocity vector of the underwater robot in the moving coordinate system, and (u, v, w) are the tangential component, radial component and normal component of the linear velocity of the underwater robot in the moving coordinate system, respectively.

3. The method according to claim 1, characterized in that The step of determining the posture of the underwater robot when the shooting target is in the center of the shooting screen as the preset posture, and determining the second training sample according to the first training sample and the preset posture specifically includes the following steps: The posture of the underwater robot when the shooting target is in the center of the shooting screen is determined as the preset posture, and the expression of the preset posture is: and d, where is a preset coordinate, d is a preset distance between the underwater robot and the shooting target, and the shooting target is the target tracked and photographed by the underwater robot; The expression of the first sample is (φ c ,θ c ,ψ c ) and d c , where (φ c ,θ c ,ψ c ) is the actual coordinate of the underwater robot, d c is the actual distance between the underwater robot and the shooting target; The expression of the second training sample is: in, is the coordinate error, is the distance error between the underwater robot and the shooting target.

4. The method according to claim 1, characterized in that: The method of constructing a deep neural network based on the Actor-Critic mechanism, inputting the second training sample into the deep neural network for training, and obtaining a trained target deep neural network specifically includes the following steps: Configure the value network and policy network based on the Actor-Critic mechanism; Inputting the second training sample into the value network and the policy network for training, and outputting a first training result; The parameters of the value network and the strategy network are updated according to the first training result and the descent gradient parameters of the deep neural network to obtain a trained target deep neural network.

5. The method according to claim 1, characterized in that The first interference value is obtained, and the expression of the first interference value is: in, is the first disturbance value, y is the output of the attitude control system of the underwater robot, G(s) is the transfer function of the observer, M represents the matrix composed of the system inertia matrix and the additional mass matrix, ε represents the linear velocity and angular velocity vector of the underwater robot in the moving system coordinate system, represents the derivatives of the linear velocity and angular velocity vectors of the underwater robot in the moving coordinate system, C(ε) represents the Coriolis centripetal force and centripetal force matrix composed of the rigid body term and the additional mass matrix, D(ε) represents the damping matrix caused by the resistance, g(η) represents the restoring force matrix caused by gravity and buoyancy, B represents the control matrix depending on the configuration of the underwater robot thruster, u represents the force and torque vector provided by the underwater robot thruster, and the observer is used to measure the interference caused by the underwater environment on the posture control of the underwater robot.

6. The method according to claim 1, characterized in that The performing posture control on the underwater robot based on the second dynamic model and the first interference value specifically comprises the following steps: Obtaining a first control value output by the second dynamic model; The first interference value is compensated to the first control value to obtain a second control value. The calculation formula of the second control value is: Among them, a comp is the second control value, a is the first control value, is the first interference value.

7. An underwater robot posture control device based on PPO algorithm, characterized in that: include: A first module is used to construct a first dynamics model and determine a first training sample according to the first dynamics model, where the first training sample is posture data when the underwater robot interacts with the underwater environment; A second module is used to determine the posture of the underwater robot when the shooting target is in the center of the shooting picture as the preset posture, and determine a second training sample according to the first training sample and the preset posture, wherein the shooting target is the target tracked and shot by the underwater robot; A third module is used to build a deep neural network based on the Actor-Critic mechanism, and input the second training sample into the deep neural network for training until the number of training rounds reaches a preset value; A fourth module is used to obtain a first interference value, where the first interference value is an interference value generated by the underwater environment to the underwater robot when the underwater robot interacts with the underwater environment; A fifth module is used to update the first dynamics model using the trained deep neural network to obtain a second dynamics model; The sixth module is used to perform posture control on the underwater robot based on the second dynamic model and the first interference value.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images